A method for extracting faces from surveillance videos
By generating reflection component maps through logarithmic transformation and multi-scale Gaussian kernel processing, and combining them with feature weight fusion of a face recognition neural network, the problem of low face recognition accuracy in surveillance videos under varying lighting conditions and complex backgrounds is solved, achieving more stable and accurate face feature extraction and recognition.
Patent Information
- Application Number
- CN202611176644.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-05
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-08-05
AI Technical Summary
Existing facial recognition technology in surveillance videos has low accuracy and difficulty in reliably extracting facial features under conditions of drastic changes in lighting and complex backgrounds.
The reflection component map is obtained by logarithmic transformation and multi-scale Gaussian kernel processing, and a global structural information enhancement map and a face skin color response map are generated. The feature weights generated by the face recognition neural network are then weighted and fused to improve the stability of face feature extraction and recognition accuracy.
It effectively compresses the dynamic range of illumination, enhances facial structural features, reduces background interference, and improves the accuracy of facial region localization and recognition results.
Smart Images

Figure CN122695676B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more specifically to a method for facial recognition extraction from surveillance videos. Background Technology
[0002] With the rapid development of intelligent security and urban monitoring systems, facial recognition technology based on surveillance videos has become an important means of identity verification, trajectory analysis, and early warning of abnormal behavior. Because surveillance scenes typically involve drastic changes in lighting, variable shooting distances, large differences in target scale, and complex background interference, how to stably extract facial features from video images has become a persistent concern in this field.
[0003] The existing technical solution involves preprocessing the surveillance video frames, then directly extracting facial region features using a face detection model based on a deep convolutional neural network, and finally completing face recognition through a classification network. This method typically relies on large-scale training data to enable the network to automatically learn texture and semantic information, thereby achieving the localization and recognition of facial targets.
[0004] However, in real-world monitoring environments, strong light, backlighting, or low light conditions at night can easily lead to decreased image contrast, obscuring facial structural information. Simultaneously, highly textured areas in complex backgrounds may be confused with local facial features, causing the network to pay insufficient attention to key structures. Furthermore, when skin color information is similar to the ambient color, the model's ability to distinguish facial regions further decreases, thus affecting the stability and accuracy of facial feature extraction. Therefore, existing technologies suffer from low facial recognition accuracy. Summary of the Invention
[0005] In view of the above-mentioned shortcomings in the prior art, the present invention provides a method for extracting faces from surveillance videos, which solves the problem of low face recognition accuracy in the prior art.
[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is: a method for extracting faces from surveillance videos, comprising the following steps: S1. Perform a logarithmic transformation on the images in the surveillance video and process them using Gaussian kernels of three scales to obtain three reflection component images; S2. Obtain the global structural information enhancement map based on the three reflection component maps; S3. Obtain the prior response value of the pixel from the global structural information enhancement map, and generate the face structure response map by combining its ratio with the background prior response value. S4. Perform YCrCb color space conversion on the global structural information enhancement map, obtain the face skin color response value, and generate a face skin color response map. S5. A face recognition neural network is used to generate feature weights based on the face skin color response map and the face structure response map. The gray-scale features of the global structure information enhancement map are weighted using the feature weights and then fused and classified to obtain the face recognition result.
[0007] Furthermore, S1 includes the following sub-steps: S11. Perform a logarithmic transformation on the images in the surveillance video to obtain a logarithmic image; S12. Using Gaussian kernels of large, medium and small scales, Gaussian filtering operations are performed on the logarithmic image in sequence to obtain three logarithmic results of illumination components. S13. Subtract the logarithmic results of the three illumination components from the logarithmic image to obtain the logarithmic results of the three reflection components; S14. Restore the logarithmic result of each reflection component to obtain the reflection component map.
[0008] Furthermore, S2 includes the following sub-steps: S21. Based on the horizontal and vertical gradients of each pixel in each reflection component image, obtain the gradient magnitude of each pixel. S22. Add the gradient magnitudes of the same pixel in the three reflection component images to obtain the total gradient magnitude; S23. Take the ratio of the gradient magnitude of the same pixel in the three reflection component images to the total gradient magnitude to obtain three gradient ratios; S24. Based on the three gradient ratios, the three reflection component maps are weighted to obtain the global structural information enhancement map.
[0009] Furthermore, S3 includes the following sub-steps: S31. For each pixel in the global structural information enhancement image, take a 3×3 neighborhood and obtain the gradient values of each pixel in four directions. S32. Take the absolute value of the gradient values in the four directions to obtain the absolute values of the gradients in the four directions; S33. Select the maximum and minimum values among the absolute values of the gradients in the four directions to obtain the maximum absolute gradient value and the minimum absolute gradient value; S34. The ratio of the absolute value of the minimum gradient to the absolute value of the maximum gradient is used as the directional balance of the corresponding pixel. S35. Sum the absolute values of the gradients in the four directions and take the average to obtain the structure density of the corresponding pixel. S36. Multiply the directional uniformity of the same pixel by the structural density to obtain the prior response value of the pixel; S37. Take the ratio of the prior response value of the pixel to the prior response value of the background as the face structure response value of the corresponding pixel to obtain the face structure response map.
[0010] Furthermore, S4 includes the following sub-steps: S41. Convert the global structural information enhancement map to the YCrCb color space and extract the red component Cr and the blue component Cb. S42. If the red component Cr is within the range of the red component of the skin tone and the blue component Cb is within the range of the blue component of the skin tone, calculate the skin tone matching degree of the corresponding pixel; otherwise, assign the skin tone matching degree of the corresponding pixel to 0. S43. Calculate the variance of the red component Cr in the neighborhood range of each pixel to obtain the red component non-uniformity of the corresponding pixel. S44. Calculate the variance of the blue component Cb in the neighborhood of each pixel to obtain the blue component non-uniformity of the corresponding pixel. S45. Add the red component non-uniformity and blue component non-uniformity of the same pixel and map them to the color uniformity of the corresponding pixel. S46. Multiply the skin color matching degree and color uniformity of the same pixel as the face skin color response value to obtain the face skin color response map.
[0011] Furthermore, the formula for calculating skin tone matching accuracy in S42 is as follows: , in, For the first Skin tone matching accuracy per pixel For the first The red component Cr of each pixel, For the first The blue component Cb of each pixel, For standard skin tone red content, The blue component of standard skin tone, To take the absolute value, The pixel number; The formula for calculating color uniformity in S45 is: , in, For the first Color uniformity of each pixel For the first Unevenness of red component in individual pixels For the first Unevenness of the blue component in each pixel.
[0012] Furthermore, the face recognition neural network recognition process in S5 includes: S51. Multi-layer feature extraction is performed on the grayscale image corresponding to the global structural information enhancement image, the face structure response image, and the face skin color response image through the multi-layer feature extraction module to obtain global structural features, face structure features, and face skin color features. S52. Use the Sigmoid layer to generate weights for facial structure features and facial skin color features respectively, to obtain the weights of facial structure features and facial skin color features. S53. Multiply the face structure feature weight and face skin color feature weight with the global structure feature element by element to obtain the face structure guiding feature and face skin color guiding feature. S54. The first convolutional layer is used to map the facial structure guidance features to obtain facial structure discrimination features; S55. The second convolutional layer is used to map the skin color guidance features of the face to obtain the skin color discrimination features of the face. S56. Multiply the facial structure discrimination features and facial skin color discrimination features element-wise to obtain the collaboratively enhanced facial features; S57. Perform deep feature extraction on the collaboratively enhanced facial features and classify them to obtain the facial recognition results.
[0013] Furthermore, the multi-layer feature extraction module includes: a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a seventh convolutional layer, and a first concat layer; The input of the third convolutional layer serves as the input of the multi-layer feature extraction module, and its output is connected to the input of the fourth convolutional layer. The output of the fourth convolutional layer is connected to the input of the fifth convolutional layer and the input of the first concat layer, respectively. The output of the fifth convolutional layer is connected to the input of the sixth convolutional layer and the input of the first Concat layer, respectively. The output of the sixth convolutional layer is connected to the input of the seventh convolutional layer and the input of the first Concat layer, respectively. The output of the seventh convolutional layer is connected to the input of the first concat layer; The output of the first Concat layer serves as the output of the multi-layer feature extraction module.
[0014] Furthermore, S57 includes the following sub-steps: S571. A multi-receptive field fusion module is used to process the collaboratively enhanced facial features to obtain global discriminative features; S572. Residual modules are used to perform residual connections on the global discriminant features to obtain enhanced global discriminant features; S573. A fully connected layer is used to classify the enhanced global discriminative features to obtain the face recognition results.
[0015] Furthermore, the multi-receptive field fusion module includes: a small receptive field feature branch, a medium receptive field feature branch, a large receptive field feature branch, a second Concat layer, and an eighth convolutional layer; The input of the small receptive field feature branch is connected to the input of the medium receptive field feature branch and the input of the large receptive field feature branch, respectively, and serves as the input of the multi-receptive field fusion module. The input of the second Concat layer is connected to the output of the small receptive field feature branch, the output of the medium receptive field feature branch and the output of the large receptive field feature branch, respectively, and its output is connected to the input of the eighth convolutional layer. The output of the eighth convolutional layer serves as the output of the multi-receptive field fusion module.
[0016] The beneficial effects of this invention are as follows: 1. This invention performs logarithmic transformation on the image and extracts the reflection component using a multi-scale Gaussian kernel, which helps to compress the dynamic range of illumination, reduce the impact of strong and weak light on the image, make facial structural details more prominent, and thus improve the stability of subsequent feature extraction.
[0017] 2. This invention constructs a global structural information enhancement map using three reflection component maps, making the structural information more prominent. It further generates a facial structure response map, which strengthens key areas related to facial contours and features, reduces interference from background textures on structural discrimination, and improves the accuracy of facial region localization. Furthermore, by obtaining skin tone response information through color space conversion, the structural and color information become complementary, helping to more effectively distinguish between facial and non-facial regions in scenes with complex or similar background colors.
[0018] 3. This invention utilizes a face recognition neural network to fuse the face structure response and the face skin color response to generate feature weights, and adaptively weights the global structural features, enabling the network to pay more attention to effective features related to the face, reduce interference from irrelevant information, and thus improve the accuracy of the recognition results. Attached Figure Description
[0019] Figure 1 A flowchart of a method for extracting faces from surveillance videos; Figure 2 This is a schematic diagram of the structure of a multi-layer feature extraction module; Figure 3 This is a schematic diagram of the structure of the multi-receptor field fusion module. Detailed Implementation
[0020] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.
[0021] like Figure 1 As shown, a method for extracting faces from surveillance videos includes the following steps: S1. Perform a logarithmic transformation on the images in the surveillance video and process them using Gaussian kernels of three scales to obtain three reflection component images; S2. Obtain the global structural information enhancement map based on the three reflection component maps; S3. Obtain the prior response value of the pixel from the global structural information enhancement map, and generate the face structure response map by combining its ratio with the background prior response value. S4. Perform YCrCb color space conversion on the global structural information enhancement map, obtain the face skin color response value, and generate a face skin color response map. S5. A face recognition neural network is used to generate feature weights based on the face skin color response map and the face structure response map. The gray-scale features of the global structure information enhancement map are weighted using the feature weights and then fused and classified to obtain the face recognition result.
[0022] In this embodiment, S1 includes the following sub-steps: S11. Perform a logarithmic transformation on the images in the surveillance video to obtain a logarithmic image; S12. Using Gaussian kernels of large, medium and small scales, Gaussian filtering operations are performed on the logarithmic image in sequence to obtain three logarithmic results of illumination components. S13. Subtract the logarithmic results of the three illumination components from the logarithmic image to obtain the logarithmic results of the three reflection components; S14. Restore the logarithmic result of each reflection component to obtain the reflection component map.
[0023] The logarithmically transformed image is: , Images in surveillance video; the formula for Gaussian filtering in S12 is: ,in, Take 1, 2, 3 in sequence. For small-scale Gaussian kernel functions, The Gaussian kernel function is a mesoscale function. For large-scale Gaussian kernel functions, For the first Zhang Guangzhao's logarithmic transformation results This is a convolution operation.
[0024] The formula for obtaining the logarithmic results of the three reflection components in S13 is: ,in, For the first Logarithmic transformation of the Zhang reflection component, for Performing an inverse logarithmic transform yields the corresponding reflection component map, i.e. .
[0025] This invention obtains the result by performing a logarithmic transformation on the original surveillance image. By compressing the non-linear dynamic range of image brightness, overexposure of highlights in strong light and backlight scenes can be effectively reduced, while image contrast in low light scenes at night can be improved, preventing facial structural information from being obscured in extreme lighting conditions and making basic structural features such as facial edges and contours clearer. The difference operation directly extracts the illumination components at various scales from the logarithmically transformed image, retaining only the logarithmically transformed reflection components that reflect the essential structure of the face. This completely eliminates the interference of factors such as strong light, weak light, and uneven illumination in surveillance scenes, allowing the extracted... It only contains core structural information such as the texture, outline, and arrangement of facial features of the face, which solves the problem of confusion between lighting noise and facial features in the preprocessing of existing technologies.
[0026] In this embodiment, the small-scale Gaussian kernel is set to σ=1.5 to separate the small-scale lighting noise corresponding to the fine local textures of the face (such as eye wrinkles and nasolabial folds) and retain the microstructure information; the medium-scale Gaussian kernel is set to σ=3 to separate the medium-scale lighting unevenness corresponding to the facial features (eyes, nose, mouth) and retain the core structure information; the large-scale Gaussian kernel is set to σ=5 to separate the large-scale global lighting deviation corresponding to the overall contour of the face and retain the macrostructure information.
[0027] In this embodiment, S2 includes the following sub-steps: S21. Based on the horizontal and vertical gradients of each pixel in each reflection component image, obtain the gradient magnitude of each pixel. S22. Add the gradient magnitudes of the same pixel in the three reflection component images to obtain the total gradient magnitude; S23. Take the ratio of the gradient magnitude of the same pixel in the three reflection component images to the total gradient magnitude to obtain three gradient ratios; S24. Based on the three gradient ratios, the three reflection component maps are weighted to obtain the global structural information enhancement map.
[0028] In this embodiment, the formula for calculating the gradient ratio in S23 is: in, For the first The first reflection component map Gradient ratio of each pixel For the first The first reflection component map The gradient magnitude of each pixel; in this embodiment, the expression for obtaining the global structural information enhancement map in S24 is: ,in, The first part of the global structural information enhancement graph The pixel value of each pixel. For the first Zhang reflection component diagram The The pixel value of each pixel.
[0029] This invention achieves adaptive weighted fusion of three reflection component maps through gradient proportions, generating a global structural information enhancement map. It effectively aggregates multi-scale structural features of the face, including small-scale fine texture, medium-scale facial structure, and large-scale overall contour, objectively reflecting the actual contribution proportion of structural features at each scale. This avoids the feature loss problem of fixed-weight fusion, weakens noise interference at each scale, and improves the clarity and overall recognizability of facial structural features.
[0030] In this embodiment, S3 includes the following sub-steps: S31. For each pixel in the global structural information enhancement image, take a 3×3 neighborhood and obtain the gradient values of each pixel in four directions. S32. Take the absolute value of the gradient values in the four directions to obtain the absolute values of the gradients in the four directions; S33. Select the maximum and minimum values among the absolute values of the gradients in the four directions to obtain the maximum absolute gradient value and the minimum absolute gradient value; S34. The ratio of the absolute value of the minimum gradient to the absolute value of the maximum gradient is used as the directional balance of the corresponding pixel. S35. Sum the absolute values of the gradients in the four directions and take the average to obtain the structure density of the corresponding pixel. S36. Multiply the directional uniformity of the same pixel by the structural density to obtain the prior response value of the pixel; S37. Take the ratio of the prior response value of the pixel to the prior response value of the background as the face structure response value of the corresponding pixel to obtain the face structure response map.
[0031] In this embodiment, the four directions include: a 0° horizontal direction, a 90° vertical direction, a 45° diagonal direction, and a 135° diagonal direction. The structure density is the average of the absolute values of the four gradients.
[0032] The human face and body regions have rich structural features such as facial features and contours. The gradient changes in the four directions within the 3×3 neighborhood are obvious, the structure density is high and the directional balance is moderate. The prior response value obtained by multiplying the two is higher. In contrast, the background region is mostly a single texture, smooth region or irregular low texture region. The gradient changes within the neighborhood are weak, the structure density is low, and the directional balance is not supported by effective values due to the overall small gradient value. The prior response value is significantly lower.
[0033] This invention extracts multi-directional gradient information within the pixel neighborhood and constructs a priori responses by combining directional balance and structural density. This allows regions with multi-directional variations and concentrated structures to achieve higher responses, helping to highlight key structures such as facial contours and features. Simultaneously, by measuring consistency through the ratio of minimum to maximum gradients, false responses caused by single strong edges or noise can be reduced. Furthermore, by comparing the ratio of pixel prior responses to background prior responses, interference from complex background textures can be effectively reduced, improving the discriminability and stability of facial structural regions.
[0034] The background prior response value is the arithmetic mean of the prior response values of all pixels in the image frame where no people appear in the surveillance video, calculated according to steps S31 to S36. It serves as a benchmark reference value to characterize the structural features of the pure background area of the surveillance scene.
[0035] In this embodiment, the grayscale value corresponding to the image is used to calculate the gradient.
[0036] In this embodiment, S4 includes the following sub-steps: S41. Convert the global structural information enhancement map to the YCrCb color space and extract the red component Cr and the blue component Cb. S42. If the red component Cr is within the range of the red component of the skin tone and the blue component Cb is within the range of the blue component of the skin tone, calculate the skin tone matching degree of the corresponding pixel; otherwise, assign the skin tone matching degree of the corresponding pixel to 0. S43. Calculate the variance of the red component Cr in the neighborhood range (5×5) of each pixel to obtain the red component non-uniformity of the corresponding pixel. S44. Calculate the variance of the blue component Cb in the neighborhood range (5×5) of each pixel to obtain the blue component non-uniformity of the corresponding pixel. S45. Add the red component non-uniformity and blue component non-uniformity of the same pixel and map them to the color uniformity of the corresponding pixel. S46. Multiply the skin color matching degree and color uniformity of the same pixel as the face skin color response value to obtain the face skin color response map.
[0037] In this embodiment, the red component of skin tone ranges from [133 to 173], and the blue component ranges from [77 to 127]. The global structural information enhancement image is an RGB image.
[0038] This invention transforms the global structural information enhancement map into the YCrCb color space. Based on a fixed skin tone range of Cr and Cb components, it initially screens effective skin tone pixels. It then calculates color uniformity by combining the variance of the neighborhood components. By multiplying the skin tone matching degree with the color uniformity, it generates a facial skin tone response map. Utilizing the stability of the YCrCb color space in representing skin tone, it effectively separates brightness and chromaticity information, avoiding interference from changes in illumination on skin tone features. Through the initial screening using dual component ranges, it accurately filters non-skin tone pixels. Combined with color uniformity, it further enhances the continuity of skin tone features in the facial region, weakens interference from ambient colors similar to skin tone, and achieves precise localization of the facial skin tone region.
[0039] In this embodiment, the formula for calculating skin color matching degree in S42 is: , in, For the first Skin tone matching accuracy per pixel For the first The red component Cr of each pixel, For the first The blue part of each pixel Quantity Cb, For standard skin tone red content, The blue component of standard skin tone, To take the absolute value, The pixel number.
[0040] In this embodiment, the standard skin tone red component is set to 153, and the standard skin tone blue component is set to 102.
[0041] The formula for calculating color uniformity in S45 is: ,in, For the first Color uniformity of each pixel For the first Unevenness of red component in individual pixels For the first Unevenness of the blue component in each pixel.
[0042] The expression for obtaining the facial skin color response value in S46 is: ,in, The first of the facial skin tone response maps The facial skin tone response value per pixel.
[0043] In this embodiment, the face recognition neural network includes: a first multi-layer feature extraction module, a second multi-layer feature extraction module, a third multi-layer feature extraction module, a first Sigmoid layer, a second Sigmoid layer, multipliers M1, M2, and M3, a first convolutional layer, a second convolutional layer, a multi-receptive field fusion module, a residual module, and a fully connected layer.
[0044] In this embodiment, S5 includes the following sub-steps: S51. Multi-layer feature extraction is performed on the grayscale image corresponding to the global structural information enhancement image, the face structure response image, and the face skin color response image through the multi-layer feature extraction module to obtain global structural features, face structure features, and face skin color features. S52. Use the Sigmoid layer to generate weights for facial structure features and facial skin color features respectively, to obtain the weights of facial structure features and facial skin color features. S53. Multiply the face structure feature weight and face skin color feature weight with the global structure feature element by element to obtain the face structure guiding feature and face skin color guiding feature. S54. The first convolutional layer is used to map the facial structure guidance features to obtain facial structure discrimination features; S55. The second convolutional layer is used to map the skin color guidance features of the face to obtain the skin color discrimination features of the face. S56. Multiply the facial structure discrimination features and facial skin color discrimination features element-wise to obtain the collaboratively enhanced facial features; S57. Perform deep feature extraction on the collaboratively enhanced facial features and classify them to obtain the facial recognition results.
[0045] In this embodiment, the global structural information enhancement map is first processed into a grayscale image to obtain a grayscale image, which makes the grayscale image, the face structure response map, and the face skin color response... Figure 3 All images are of the same size.
[0046] This invention extracts global structure, facial structure, and skin color features through multi-layer feature extraction. An adaptive feature weight is generated by a Sigmoid layer to guide the weighting of global structure features. Then, a convolutional layer maps the discriminative features, and element-wise multiplication achieves synergistic enhancement. Finally, deep extraction and classification yield the recognition result. This invention achieves adaptive weighting and deep synergistic fusion of structural and skin color features, allowing the model to focus on the core effective features of the face while weakening ineffective features caused by background and noise interference. The synergistic enhancement through convolutional mapping and element-wise multiplication strengthens the complementarity of structural and skin color features, improving the completeness of facial feature representation.
[0047] In this embodiment, S51 includes: using a first multi-layer feature extraction module to perform multi-layer feature extraction on the face structure response map to obtain face structure features; using a second multi-layer feature extraction module to perform multi-layer feature extraction on the grayscale image corresponding to the global structure information enhancement map to obtain global structure features; and using a third multi-layer feature extraction module to perform multi-layer feature extraction on the face skin color response map to obtain face skin color features.
[0048] In this embodiment, S52 includes: generating facial structure feature weights by using a first Sigmoid layer to process facial structure features; and generating facial skin color feature weights by using a second Sigmoid layer to process facial skin color features.
[0049] In this embodiment, S53 includes: using multiplier M1 to multiply the face structure feature weights with the global structure features element-wise to obtain face structure guidance features; and using multiplier M2 to multiply the face skin color feature weights with the global structure features element-wise to obtain face skin color guidance features.
[0050] In this embodiment, S56 includes: using multiplier M3 to multiply the face structure discrimination features and the face skin color discrimination features element by element to obtain the collaboratively enhanced face features.
[0051] In this embodiment, the kernel size of both the first and second convolutional layers is 1×1.
[0052] like Figure 2 As shown, the first multi-layer feature extraction module, the second multi-layer feature extraction module, and the third multi-layer feature extraction module have the same structure, each including: a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a seventh convolutional layer, and a first concat layer; The input of the third convolutional layer serves as the input of the multi-layer feature extraction module, and its output is connected to the input of the fourth convolutional layer. The output of the fourth convolutional layer is connected to the input of the fifth convolutional layer and the input of the first concat layer, respectively. The output of the fifth convolutional layer is connected to the input of the sixth convolutional layer and the input of the first Concat layer, respectively. The output of the sixth convolutional layer is connected to the input of the seventh convolutional layer and the input of the first Concat layer, respectively. The output of the seventh convolutional layer is connected to the input of the first concat layer; The output of the first Concat layer serves as the output of the multi-layer feature extraction module.
[0053] In this embodiment, the kernel size of the third convolutional layer is 3×3, the stride is 1, and the padding is 1; the kernel size of the fourth convolutional layer is 1×1, the stride is 1, and the padding is 0; the kernel size of the fifth convolutional layer is 3×3, the stride is 1, and the padding is 1; the kernel size of the sixth convolutional layer is 5×5, the stride is 1, and the padding is 2; and the kernel size of the seventh convolutional layer is 7×7, the stride is 1, and the padding is 3.
[0054] In this embodiment, the multi-receptive field fusion module, the residual module, and the fully connected layer are connected sequentially. S57 includes the following sub-steps: S571. A multi-receptive field fusion module is used to process the collaboratively enhanced facial features to obtain global discriminative features; S572. Residual modules are used to perform residual connections on the global discriminant features to obtain enhanced global discriminant features; S573. A fully connected layer is used to classify the enhanced global discriminative features to obtain the face recognition results.
[0055] like Figure 3 As shown, the multi-receptive field fusion module includes: a small receptive field feature branch, a medium receptive field feature branch, a large receptive field feature branch, a second Concat layer, and an eighth convolutional layer; The input of the small receptive field feature branch is connected to the input of the medium receptive field feature branch and the input of the large receptive field feature branch, respectively, and serves as the input of the multi-receptive field fusion module. The input of the second Concat layer is connected to the output of the small receptive field feature branch, the output of the medium receptive field feature branch and the output of the large receptive field feature branch, respectively, and its output is connected to the input of the eighth convolutional layer. The output of the eighth convolutional layer serves as the output of the multi-receptive field fusion module.
[0056] In this embodiment, the small receptive field feature branch includes, in sequence, the ninth and tenth convolutional layers, both with a kernel size of 3×3, a stride of 1, and padding of 1. The medium receptive field feature branch includes, in sequence, the eleventh convolutional layer with a kernel size of 5×5, a stride of 1, and padding of 2; and the twelfth convolutional layer with a kernel size of 3×3, a stride of 1, and padding of 1. The large receptive field feature branch includes, in sequence, the thirteenth and fourteenth convolutional layers, both with a kernel size of 7×7, a stride of 1, and padding of 3; and the fourteenth convolutional layer with a kernel size of 3×3, a stride of 1, and padding of 1. The eighth convolutional layer has a kernel size of 1×1.
[0057] The residual module uses the basic residual block of the original ResNet, which includes two convolutional layers and an adder. The kernel size of the two convolutional layers is 3×3.
[0058] The multi-receptive field fusion module can capture facial feature details at different scales by setting up small, medium and large receptive field feature branches in parallel: the small receptive field branch extracts fine textures such as eye wrinkles and lip lines through continuous 3×3 convolutions; the medium receptive field branch extracts the core structure such as facial feature layout through a combination of 5×5 and 3×3 convolutions; and the large receptive field branch extracts the overall contour and shape of the face through a combination of 7×7 and 3×3 convolutions.
[0059] This invention constructs a global structural information enhancement map by performing a logarithmic transformation on the monitoring image and extracting the reflection component using a multi-scale Gaussian kernel to highlight facial structural details. Simultaneously, it generates a structural response based on the ratio of structural prior to background prior and integrates skin color response information to form a collaborative discrimination of facial regions. On this basis, it uses a neural network to generate feature weights to adaptively enhance key features, thereby reducing the interference of lighting changes and complex backgrounds on feature extraction and improving the stability and recognition accuracy of facial feature extraction.
[0060] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for extracting faces from surveillance videos, characterized in that, Includes the following steps: S1. Perform a logarithmic transformation on the images in the surveillance video and process them using Gaussian kernels of three scales to obtain three reflection component images; S2. Obtain the global structural information enhancement map based on the three reflection component maps; S3. Obtain the prior response value of the pixel from the global structural information enhancement map, and generate the face structure response map by combining its ratio with the background prior response value. S4. Perform YCrCb color space conversion on the global structural information enhancement map, obtain the face skin color response value, and generate a face skin color response map; S5. A face recognition neural network is adopted. Feature weights are generated based on the face skin color response map and the face structure response map. The gray-scale features of the global structure information enhancement map are weighted using the feature weights and fused for classification to obtain the face recognition result. S3 includes the following steps: S31. For each pixel in the global structural information enhancement image, take a 3×3 neighborhood and obtain the gradient values of each pixel in four directions. S32. Take the absolute value of the gradient values in the four directions to obtain the absolute values of the gradients in the four directions; S33. Select the maximum and minimum values among the absolute values of the gradients in the four directions to obtain the maximum absolute gradient value and the minimum absolute gradient value; S34. The ratio of the absolute value of the minimum gradient to the absolute value of the maximum gradient is used as the directional balance of the corresponding pixel. S35. Add the absolute values of the gradients in the four directions and take the average to obtain the structure density of the corresponding pixel. S36. Multiply the directional uniformity of the same pixel by the structural density to obtain the prior response value of the pixel; S37. Take the ratio of the prior response value of the pixel to the prior response value of the background as the face structure response value of the corresponding pixel to obtain the face structure response map. The face recognition neural network recognition process in S5 includes: S51. Multi-layer feature extraction is performed on the grayscale image corresponding to the global structural information enhancement image, the face structure response image, and the face skin color response image through the multi-layer feature extraction module to obtain global structural features, face structure features, and face skin color features. S52. Use the Sigmoid layer to generate weights for facial structure features and facial skin color features respectively, to obtain the weights of facial structure features and facial skin color features. S53. Multiply the face structure feature weight and face skin color feature weight with the global structure feature element by element to obtain the face structure guiding feature and face skin color guiding feature. S54. The first convolutional layer is used to map the facial structure guidance features to obtain facial structure discrimination features; S55. The second convolutional layer is used to map the skin color guidance features of the face to obtain the skin color discrimination features of the face. S56. Multiply the facial structure discrimination features and facial skin color discrimination features element-wise to obtain the collaboratively enhanced facial features; S57. Perform deep feature extraction on the collaboratively enhanced facial features and classify them to obtain the facial recognition results.
2. The method for extracting faces from surveillance videos according to claim 1, characterized in that, S1 includes the following steps: S11. Perform a logarithmic transformation on the images in the surveillance video to obtain a logarithmic image; S12. Using Gaussian kernels of large, medium and small scales, Gaussian filtering operations are performed on the logarithmic image in sequence to obtain three logarithmic results of illumination components. S13. Subtract the logarithmic results of the three illumination components from the logarithmic image to obtain the logarithmic results of the three reflection components; S14. Restore the logarithmic result of each reflection component to obtain the reflection component map.
3. The method for extracting faces from surveillance videos according to claim 1, characterized in that, S2 includes the following steps: S21. Based on the horizontal and vertical gradients of each pixel in each reflection component image, obtain the gradient magnitude of each pixel. S22. Add the gradient magnitudes of the same pixel in the three reflection component images to obtain the total gradient magnitude; S23. Take the ratio of the gradient magnitude of the same pixel in the three reflection component images to the total gradient magnitude to obtain three gradient ratios; S24. Based on the three gradient ratios, the three reflection component maps are weighted to obtain the global structural information enhancement map.
4. The method for extracting faces from surveillance videos according to claim 1, characterized in that, S4 includes the following steps: S41. Convert the global structural information enhancement map to the YCrCb color space and extract the red component Cr and the blue component Cb. S42. If the red component Cr is within the range of the red component of the skin tone and the blue component Cb is within the range of the blue component of the skin tone, calculate the skin tone matching degree of the corresponding pixel; otherwise, assign the skin tone matching degree of the corresponding pixel to 0. S43. Calculate the variance of the red component Cr in the neighborhood range of each pixel to obtain the red component non-uniformity of the corresponding pixel. S44. Calculate the variance of the blue component Cb in the neighborhood of each pixel to obtain the blue component non-uniformity of the corresponding pixel. S45. Add the red component non-uniformity and blue component non-uniformity of the same pixel and map them to the color uniformity of the corresponding pixel. S46. Multiply the skin color matching degree and color uniformity of the same pixel as the face skin color response value to obtain the face skin color response map. The formula for calculating skin tone matching accuracy in S42 is: ,in, For the first Skin tone matching accuracy per pixel For the first The red component Cr of each pixel, For the first The blue component Cb of each pixel, For standard skin tone red content, The blue component of standard skin tone, To take the absolute value, The pixel number; The formula for calculating color uniformity in S45 is: , in, For the first Color uniformity of each pixel For the first Unevenness of red component in individual pixels For the first Unevenness of the blue component in each pixel.
5. The method for extracting faces from surveillance videos according to claim 1, characterized in that, The multi-layer feature extraction module includes: a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a seventh convolutional layer, and a first concat layer; The input of the third convolutional layer serves as the input of the multi-layer feature extraction module, and its output is connected to the input of the fourth convolutional layer. The output of the fourth convolutional layer is connected to the input of the fifth convolutional layer and the input of the first Concat layer, respectively. The output of the fifth convolutional layer is connected to the input of the sixth convolutional layer and the input of the first Concat layer, respectively. The output of the sixth convolutional layer is connected to the input of the seventh convolutional layer and the input of the first Concat layer, respectively. The output of the seventh convolutional layer is connected to the input of the first concat layer. The output of the first Concat layer serves as the output of the multi-layer feature extraction module.
6. The method for extracting faces from surveillance videos according to claim 1, characterized in that, S57 includes the following steps: S571. A multi-receptive field fusion module is used to process the collaboratively enhanced facial features to obtain global discriminative features; S572. Residual modules are used to perform residual connections on the global discriminant features to obtain enhanced global discriminant features; S573. A fully connected layer is used to classify the enhanced global discriminative features to obtain the face recognition results.
7. The method for extracting faces from surveillance videos according to claim 6, characterized in that, The multi-receptive field fusion module includes: a small receptive field feature branch, a medium receptive field feature branch, a large receptive field feature branch, a second Concat layer, and an eighth convolutional layer; The input of the small receptive field feature branch is connected to the input of the medium receptive field feature branch and the input of the large receptive field feature branch, respectively, and serves as the input of the multi-receptive field fusion module. The input of the second Concat layer is connected to the output of the small receptive field feature branch, the output of the medium receptive field feature branch and the output of the large receptive field feature branch, respectively, and its output is connected to the input of the eighth convolutional layer. The output of the eighth convolutional layer serves as the output of the multi-receptive field fusion module.
Citation Information
Patent Citations
Face detection and recognition method based on skin color segmentation and template matching
CN103632132A
Extracting method for face algorithm standard face image
CN104036278A
