A face recognition method based on binocular local region

Abstract

The application discloses a face recognition method based on binocular local regions, and belongs to the technical field of image processing. The application firstly collects a left image and a right image of a target face through a binocular camera, and generates a depth map by using parallax calculation; then, the left image and the depth map are finely meshed to extract grid mean values, so that corresponding low-resolution images are obtained, and high points and low points are marked based on the depth values through coarse meshing; based on the marking results, local pixel features and local depth features are extracted from the coarse meshes of the left image and the depth map respectively, and a three-channel feature matrix is constructed; finally, a binocular multi-modal local region fusion face recognition network is used to fuse and process the low-resolution images and the two types of feature matrices, and a face recognition result is output. The application strengthens the expression of local region features by fusing pixel information and depth information of binocular vision, and effectively improves the precision of face recognition.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a face recognition method based on binocular local regions. Background Technology

[0002] With the development of stereo vision technology, the application of binocular vision in face recognition has gradually attracted attention. Binocular cameras simultaneously acquire facial images from both left and right perspectives, and can calculate a depth map based on the parallax information between the two images, thereby obtaining the three-dimensional geometric structure information of the face. Compared with traditional monocular images, binocular images not only contain brightness and texture information, but also provide spatial depth features of the facial surface, which theoretically can significantly improve the accuracy and robustness of recognition.

[0003] However, existing binocular face recognition technologies only focus on the overall depth feature fusion or 3D reconstruction process of the face, emphasizing global-scale feature extraction while neglecting to capture fine-grained depth changes and pixel differences in key local facial regions (such as the eye sockets, bridge of the nose, and corners of the mouth). Since these local regions often possess strong geometric discriminative power and individual variability, the lack of targeted feature extraction for them leads to the weakening of local features within the global information, resulting in low face recognition accuracy. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a face recognition method based on binocular local regions, which solves the problem of low face recognition accuracy in existing technologies.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows: a face recognition method based on binocular local regions, comprising the following steps:

[0006] A stereo camera is used to capture images of the target's facial area, resulting in a left and a right image. The depth map is then obtained based on the parallax between the left and right images.

[0007] Both the left image and the depth image are divided into fine grids, and the grid mean is extracted to obtain a low-resolution depth image and a low-resolution left image.

[0008] Both the left image and the depth map are divided into coarse grids. In each coarse grid, high points and low points are marked based on the depth value.

[0009] Based on the high and low points in the depth map, local depth features are extracted from each coarse grid in the depth map to construct a three-channel local depth feature matrix;

[0010] Based on the high and low points in the left image, local pixel features are extracted from each coarse grid in the left image to construct a three-channel local pixel feature matrix.

[0011] A binocular multimodal local region fusion face recognition network is used to process the low-resolution depth map, the left low-resolution map, the three-channel local depth feature matrix, and the three-channel local pixel feature matrix to obtain the face recognition result.

[0012] Furthermore, the process of obtaining the depth low-resolution map and the left low-resolution map includes:

[0013] The depth map and the left image are divided using a fine mesh.

[0014] Extract the mean depth value from each fine grid of the depth map, and use the mean depth value as the depth value at the corresponding fine grid to obtain a low-resolution depth map.

[0015] Extract the pixel mean from each fine grid in the left image, and use the pixel mean as the pixel value at the corresponding fine grid to obtain the left low-resolution image.

[0016] Furthermore, the process of marking highs and lows includes:

[0017] Both the left image and the depth map are divided into coarse grids;

[0018] In each coarse grid of the depth map, calculate the average grid depth;

[0019] In the left image and the depth image, pixels with depth values ​​greater than the average depth of the corresponding grid in the coarse grid are marked as high points, and other points are marked as low points.

[0020] Furthermore, local depth features include: local depth contrast value, local area contrast value, and local center offset value;

[0021] The local depth contrast value, local area contrast value, and local center offset value of each coarse grid are concatenated according to the channel dimension to form the local depth feature vector of the coarse grid. The local depth feature vectors of each coarse grid are arranged according to their spatial position to obtain a three-channel local depth feature matrix.

[0022] Furthermore, the process of obtaining the local depth contrast value includes: in the depth map, taking the average depth value of each high point in the coarse grid to obtain the average depth value of the high points, taking the average depth value of each low point in the coarse grid to obtain the average depth value of the low points, and using the ratio of the average depth value of the high points to the average depth value of the low points as the local depth contrast value.

[0023] The process of obtaining the local area contrast value includes: in the depth map, the ratio of the number of each high point in the coarse grid to the number of each low point in the coarse grid is used as the local area contrast value;

[0024] The process of obtaining the local center offset value includes: in the depth map, calculating the geometric center of each high point for each coarse grid to obtain the high point center; calculating the geometric center of each low point for each coarse grid to obtain the low point center; calculating the distance between the high point center and the low point center and normalizing the distance to obtain the local center offset value.

[0025] Furthermore, local pixel features include: local pixel contrast value, local energy contrast value, and local complexity contrast value;

[0026] The local pixel contrast value, local energy contrast value, and local complexity contrast value of each coarse grid are concatenated along the channel dimension to form the local pixel feature vector of that coarse grid. The local pixel feature vectors of each coarse grid are arranged according to their spatial positions to obtain a three-channel local pixel feature matrix.

[0027] Furthermore, the process of obtaining the local pixel contrast value includes: in the left image, calculating the pixel value of each high point in each coarse grid to obtain the average pixel value of the high points, calculating the pixel value of each low point in each coarse grid to obtain the average pixel value of the low points, and taking the ratio of the average pixel value of the high points to the average pixel value of the low points in the coarse grid as the local pixel contrast value.

[0028] The process of obtaining the local energy contrast value includes: in the left figure, calculating the pixel gradient magnitude of each pixel in each coarse grid, and squaring the pixel gradient magnitude to obtain the texture energy; summing the texture energy of each high point to obtain the high point texture energy sum; summing the texture energy of each low point to obtain the low point texture energy sum; and taking the ratio of the high point texture energy sum to the low point texture energy sum of the coarse grid as the local energy contrast value.

[0029] The process of obtaining the local complexity comparison value includes: in the left figure, calculating the variance of the pixel values ​​of each high point in each coarse grid to obtain the high point texture complexity, calculating the variance of the pixel values ​​of each low point in each coarse grid to obtain the low point texture complexity, and taking the ratio of the high point texture complexity to the low point texture complexity as the local complexity comparison value.

[0030] Furthermore, the processing steps of the binocular multimodal local region fusion face recognition network include:

[0031] Two downsampling encoding modules are used to extract features from the depth low-resolution map and the left low-resolution map respectively, resulting in a three-channel depth feature map and a three-channel pixel feature map.

[0032] Two pointwise convolutional layers are used to process the three-channel local depth feature matrix and the three-channel local pixel feature matrix respectively, to obtain the three-channel depth mapping feature and the three-channel pixel mapping feature;

[0033] Multiply the three-channel depth feature map and the three-channel depth mapping feature by elements to obtain the three-channel depth feature enhancement map. Multiply the three-channel pixel feature map and the three-channel pixel mapping feature by elements to obtain the three-channel pixel feature enhancement map.

[0034] Channel aggregation and multi-scale feature extraction are performed on the three-channel depth feature enhancement map and the three-channel pixel feature enhancement map, respectively. Then, the same-scale feature is fused. A fully connected layer is used to classify the fused maps of multiple scales to obtain the face recognition results.

[0035] Furthermore, the two downsampling encoding modules have the same structure, both including: a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first max pooling layer, a second max pooling layer, a first average pooling layer, a second average pooling layer, and a splicing layer;

[0036] The input of the third convolutional layer serves as the input of two downsampling coding modules, and its output is connected to the input of the first max pooling layer.

[0037] The output of the first max pooling layer is connected to the input of the fourth convolutional layer; the output of the fourth convolutional layer is connected to the inputs of the second max pooling layer, the first average pooling layer, and the fifth convolutional layer; the output of the fifth convolutional layer is connected to the input of the second average pooling layer; the input of the splicing layer is connected to the outputs of the second max pooling layer, the first average pooling layer, and the second average pooling layer, and its output serves as the output of two downsampling encoding modules.

[0038] Furthermore, the process of obtaining facial recognition results includes:

[0039] Two convolutional layers are used to perform channel aggregation on the three-channel depth feature enhancement map and the three-channel pixel feature enhancement map, respectively, to obtain a single-channel depth feature enhancement map and a single-channel pixel feature enhancement map.

[0040] Two multi-scale feature extraction units are used to process the single-channel depth feature enhancement map and the single-channel pixel feature enhancement map respectively, to obtain the first and second scale depth feature enhancement maps and the first and second scale pixel feature enhancement maps;

[0041] The depth feature enhancement map and the pixel feature enhancement map of the same scale are added together to obtain the first scale fusion map and the second scale fusion map;

[0042] A fully connected layer is used to classify the first-scale fusion image and the first-scale fusion image to obtain the face recognition result.

[0043] The beneficial effects of this invention are as follows:

[0044] 1. This invention, by dividing the depth map and the left image into coarse grids and marking high and low points in each grid, can accurately extract the depth and pixel features of key local areas of the face (such as the eye sockets, bridge of the nose, and corners of the mouth). This method of extracting local features significantly improves the sensitivity to subtle geometric and texture differences, enhancing local discrimination capabilities.

[0045] 2. This invention employs a binocular multimodal local region fusion network, jointly processing the low-resolution depth map, the left low-resolution map, the local depth feature matrix, and the local pixel feature matrix to achieve complementary fusion of depth and pixel information. This multimodal information integration avoids the weakening of local features within global features, thereby improving overall recognition performance and increasing face recognition accuracy. Attached Figure Description

[0046] Figure 1 A flowchart of a face recognition method based on binocular local regions;

[0047] Figure 2 A schematic diagram of the structure of a binocular multimodal local region fusion face recognition network;

[0048] Figure 3 This is a schematic diagram of the structure of two downsampling encoding modules;

[0049] Figure 4 This is a schematic diagram of the structure of two multi-scale feature extraction units. Detailed Implementation

[0050] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0051] like Figure 1 As shown, a face recognition method based on binocular local regions includes the following steps:

[0052] A stereo camera is used to capture images of the target's facial area, resulting in a left and a right image. The depth map is then obtained based on the parallax between the left and right images.

[0053] Both the left image and the depth image are divided into fine grids, and the grid mean is extracted to obtain a low-resolution depth image and a low-resolution left image.

[0054] Both the left image and the depth map are divided into coarse grids. In each coarse grid, high points and low points are marked based on the depth value.

[0055] Based on the high and low points in the depth map, local depth features are extracted from each coarse grid in the depth map to construct a three-channel local depth feature matrix;

[0056] Based on the high and low points in the left image, local pixel features are extracted from each coarse grid in the left image to construct a three-channel local pixel feature matrix.

[0057] A binocular multimodal local region fusion face recognition network is used to process the low-resolution depth map, the left low-resolution map, the three-channel local depth feature matrix, and the three-channel local pixel feature matrix to obtain the face recognition result.

[0058] In this embodiment, the disparity map d(x,y) is the horizontal position difference d(x,y) = x between corresponding pixels in the left and right images in binocular vision. L -x R x L Let x be the x-coordinate of the pixel in the left image. R The x-coordinate of the pixel in the right image is given. The depth map Z(x,y) is derived from the disparity map, Z(x,y)=f•B / [d(x,y)+ε], where f is the camera focal length, B is the optical center distance (baseline) of the binocular cameras, and ε is used to prevent division by zero.

[0059] In this embodiment, the process of obtaining the depth low-resolution map and the left low-resolution map includes:

[0060] The depth map and the left image are divided using a fine mesh.

[0061] Extract the mean depth value from each fine grid of the depth map, and use the mean depth value as the depth value at the corresponding fine grid to obtain a low-resolution depth map.

[0062] Extract the pixel mean from each fine grid in the left image, and use the pixel mean as the pixel value at the corresponding fine grid to obtain the left low-resolution image.

[0063] In this embodiment, the fine grid size is 4×4 pixels, and the coarse grid size is 16×16 pixels. If the dimensions of the left image and the depth image are H×W, then the dimensions of the low-resolution depth image and the left low-resolution image are (H / 4)×(W / 4), where H is the height and W is the width.

[0064] This invention reduces image resolution by dividing the image into fine grids and extracting the mean, while preserving the global depth distribution trend of the depth map (such as the overall concave-convex structure of a face) and the global texture / pixel distribution features of the left image (such as the overall skin color and contour of the face). Compared to the original high-resolution image, the low-resolution image reduces pixel-level redundant information, lowers the computational complexity of subsequent networks, and improves processing efficiency.

[0065] In this embodiment, the process of marking high points and low points includes:

[0066] Both the left image and the depth map are divided into coarse grids;

[0067] In each coarse grid of the depth map, calculate the average grid depth;

[0068] In the left image and the depth image, pixels with depth values ​​greater than the average depth of the corresponding grid in the coarse grid are marked as high points, and other points are marked as low points.

[0069] In the depth map, when a pixel is marked as a high point, the same pixel position in the left image is also marked as a high point, meaning that the same pixel position is marked the same.

[0070] This invention divides the human face into several local sub-regions using a coarse grid. Within each sub-region, high and low points are marked based on the average grid depth, enabling precise identification of local structures with strong geometric discriminative power. High points naturally correspond to prominent structures such as the bridge of the nose and brow bones, while low points correspond to recessed structures such as the eye sockets and nostrils. These local three-dimensional differences are core features that distinguish different faces (such as high and low nose bridges, deep and shallow eye sockets), facilitating the extraction of features from local areas and improving facial differentiation.

[0071] In this embodiment, the local depth features include: local depth contrast value, local area contrast value, and local center offset value;

[0072] The local depth contrast value, local area contrast value, and local center offset value of each coarse grid are concatenated according to the channel dimension to form the local depth feature vector of the coarse grid. The local depth feature vectors of each coarse grid are arranged according to their spatial position to obtain a three-channel local depth feature matrix.

[0073] In this embodiment, the process of obtaining the local depth comparison value includes: in the depth map, taking the average depth value of each high point in the coarse grid to obtain the average depth value of the high points, taking the average depth value of each low point in the coarse grid to obtain the average depth value of the low points, and using the ratio of the average depth value of the high points to the average depth value of the low points as the local depth comparison value.

[0074] The process of obtaining the local area contrast value includes: in the depth map, the ratio of the number of each high point in the coarse grid to the number of each low point in the coarse grid is used as the local area contrast value;

[0075] The process of obtaining the local center offset value includes: in the depth map, calculating the geometric center of each high point for each coarse grid to obtain the high point center; calculating the geometric center of each low point for each coarse grid to obtain the low point center; calculating the distance between the high point center and the low point center and normalizing the distance to obtain the local center offset value.

[0076] The depth local contrast value, through the ratio of the average depths of high and low points, directly reflects the "intensity" of three-dimensional undulations within a region (such as the depth difference between the brow bone and the eye socket); the area local contrast value, through the ratio of the number of high and low points, depicts the "distribution ratio" of high and low structures within a region (such as a high proportion of high points in the bridge of the nose and a higher proportion of low points in the cheek area); the local center offset value, through the distance between the centers of high and low points, captures the "spatial relationship" of high and low structures (such as the distance between the center of the eye socket and the center of the brow bone). These three values ​​complement each other, comprehensively covering the "intensity-proportion-position" characteristics of local three-dimensional structures, avoiding the one-sidedness of a single-dimensional description, and providing richer features for distinguishing the three-dimensional differences of different faces (such as high and low nasal bridges, deep and shallow eye sockets).

[0077] In this embodiment, the process of obtaining the local center offset value includes: taking the spatial size of the current coarse grid as a reference, first calculating the diagonal length of the coarse grid (i.e., the straight-line distance from the top left vertex to the bottom right vertex of the grid, in pixels), and dividing the distance between the center of the high point and the center of the low point by the diagonal length to obtain the local center offset value.

[0078] In this embodiment, local pixel features include: local pixel contrast value, local energy contrast value, and local complexity contrast value;

[0079] The local pixel contrast value, local energy contrast value, and local complexity contrast value of each coarse grid are concatenated along the channel dimension to form the local pixel feature vector of that coarse grid. The local pixel feature vectors of each coarse grid are arranged according to their spatial positions to obtain a three-channel local pixel feature matrix.

[0080] In this embodiment, the process of obtaining the local pixel contrast value includes: in the left image, calculating the pixel value of each high point in each coarse grid to obtain the average pixel value of the high points, calculating the pixel value of each low point in each coarse grid to obtain the average pixel value of the low points, and taking the ratio of the average pixel value of the high points to the average pixel value of the low points in the coarse grid as the local pixel contrast value.

[0081] The process of obtaining the local energy contrast value includes: in the left figure, calculating the pixel gradient magnitude of each pixel in each coarse grid, and squaring the pixel gradient magnitude to obtain the texture energy; summing the texture energy of each high point to obtain the high point texture energy sum; summing the texture energy of each low point to obtain the low point texture energy sum; and taking the ratio of the high point texture energy sum to the low point texture energy sum of the coarse grid as the local energy contrast value.

[0082] The process of obtaining the local complexity comparison value includes: in the left figure, calculating the variance of the pixel values ​​of each high point in each coarse grid to obtain the high point texture complexity, calculating the variance of the pixel values ​​of each low point in each coarse grid to obtain the low point texture complexity, and taking the ratio of the high point texture complexity to the low point texture complexity as the local complexity comparison value.

[0083] Local pixel contrast values ​​capture the brightness / color differences between high and low points within a region by comparing the ratio of the average pixel values ​​of high and low points (e.g., the contrast in skin tone between the high point of the brow bone and the low point of the eye socket); local energy contrast values ​​reflect the difference in texture intensity between high and low points by comparing the ratio of texture energy (sum of squared gradients) between high and low points (e.g., the energy contrast between the smooth texture at the high point of the bridge of the nose and the rough texture at the low point of the wing of the nose); local complexity contrast values ​​characterize the difference in texture complexity between high and low points by comparing the ratio of the pixel variances of high and low points (e.g., the complexity contrast between the uniform texture at the high point of the forehead and the fine lines around the eyes). These three values ​​work together to comprehensively cover the "pixel contrast-intensity difference-complexity" features of local facial texture, accurately capturing fine-grained texture details such as moles, wrinkles, and lip lines, providing key discriminative criteria for distinguishing similar faces.

[0084] In this embodiment, the processing procedure of the binocular multimodal local region fusion face recognition network includes:

[0085] Two downsampling encoding modules are used to extract features from the depth low-resolution map and the left low-resolution map respectively, resulting in a three-channel depth feature map and a three-channel pixel feature map.

[0086] Two pointwise convolutional layers are used to process the three-channel local depth feature matrix and the three-channel local pixel feature matrix respectively, to obtain the three-channel depth mapping feature and the three-channel pixel mapping feature;

[0087] Multiply the three-channel depth feature map and the three-channel depth mapping feature by elements to obtain the three-channel depth feature enhancement map. Multiply the three-channel pixel feature map and the three-channel pixel mapping feature by elements to obtain the three-channel pixel feature enhancement map.

[0088] Channel aggregation and multi-scale feature extraction are performed on the three-channel depth feature enhancement map and the three-channel pixel feature enhancement map, respectively. Then, the same-scale feature is fused. A fully connected layer is used to classify the fused maps of multiple scales to obtain the face recognition results.

[0089] In this embodiment, the process of obtaining the face recognition result includes:

[0090] Two convolutional layers are used to perform channel aggregation on the three-channel depth feature enhancement map and the three-channel pixel feature enhancement map, respectively, to obtain a single-channel depth feature enhancement map and a single-channel pixel feature enhancement map.

[0091] Two multi-scale feature extraction units are used to process the single-channel depth feature enhancement map and the single-channel pixel feature enhancement map respectively, to obtain the first and second scale depth feature enhancement maps and the first and second scale pixel feature enhancement maps;

[0092] The depth feature enhancement map and the pixel feature enhancement map of the same scale are added together to obtain the first scale fusion map and the second scale fusion map;

[0093] A fully connected layer is used to classify the first-scale fusion image and the first-scale fusion image to obtain the face recognition result.

[0094] like Figure 2 As shown, the binocular multimodal local region fusion face recognition network includes: a first downsampling coding module, a second downsampling coding module, a first pointwise convolutional layer, a second pointwise convolutional layer, a first multiplication module, a second multiplication module, a first convolutional layer, a second convolutional layer, a first multi-scale feature extraction unit, a second multi-scale feature extraction unit, an adder A1, an adder A1, and a fully connected layer;

[0095] The first downsampling encoding module is used to extract three-channel depth feature maps from the low-resolution depth map;

[0096] The second downsampling encoding module is used to extract three-channel pixel feature maps from the left low-resolution image;

[0097] The three-channel local depth feature matrix is ​​processed by the first pointwise convolutional layer to obtain the three-channel depth mapping features;

[0098] The three-channel local pixel feature matrix is ​​processed by a second pointwise convolutional layer to obtain the three-channel pixel mapping features;

[0099] The first multiplication module is used to multiply the three-channel depth feature map and the three-channel depth mapping feature element-wise to obtain the three-channel depth feature enhancement map;

[0100] The second multiplication module is used to multiply the three-channel pixel feature map and the three-channel pixel mapping feature element-wise to obtain the three-channel pixel feature enhancement map;

[0101] The first convolutional layer is used to perform channel aggregation on the three-channel depth feature enhancement map to obtain a single-channel depth feature enhancement map;

[0102] A second convolutional layer is used to perform channel aggregation on the three-channel pixel feature enhancement map to obtain a single-channel pixel feature enhancement map.

[0103] The first multi-scale feature extraction unit is used to process the single-channel depth feature enhancement map to obtain the first-scale depth feature enhancement map and the second-scale depth feature enhancement map.

[0104] The single-channel pixel feature enhancement map is processed by the second multi-scale feature extraction unit to obtain the first-scale pixel feature enhancement map and the second-scale pixel feature enhancement map.

[0105] Adder A1 is used to add the first-scale depth feature enhancement map and the first-scale pixel feature enhancement map to obtain the first-scale fused map;

[0106] Adder A2 is used to add the second-scale depth feature enhancement map and the second-scale pixel feature enhancement map to obtain the second-scale fused map;

[0107] A fully connected layer is used to classify the first-scale fused map and the first-scale fused map.

[0108] The kernel size of the first and second convolutional layers is 1×1, which maps the three channels to a single channel.

[0109] The binocular multimodal local region fusion face recognition network constructs a comprehensive feature system that takes into account both 3D structure and texture details through cross-modal collaboration of depth and pixel features, significantly improving the accuracy and scene adaptability of face recognition. Depth features focus on the 3D undulations of the face (such as the height of the bridge of the nose and the protrusion of the brow bone), effectively distinguishing real faces from flat forgeries (such as photos and screen displays), resisting attacks from the physical structure level; pixel features capture texture details (such as moles, wrinkles, and lip lines), accurately identifying subtle differences between similar faces (such as twins). The two modalities achieve spatial alignment based on the same coarse grid division, forming complementary features of "3D skeleton + textured skin and flesh," which not only solves the problem of one-sided information from a single modality, but also fills the recognition blind spots of "similar 3D but different textures" and "flat forgeries but realistic textures," allowing the network to depict faces more comprehensively.

[0110] The network achieves precise enhancement of local discriminative features through a combination of pointwise convolution and multiplication modules. The first and second pointwise convolutional layers perform element-wise convolution on the local depth and pixel feature matrices, respectively, extracting the features of each element. The multiplication module then multiplies the features with the downsampled encoded base features element-wise, further amplifying the contribution of effective features and suppressing redundant information. This mechanism allows the network to "focus" on the most critical local regions for identity recognition (such as the area around the eyes, bridge of the nose, and lip line), avoiding interference from irrelevant information (such as background noise and lighting interference), and significantly improving the discriminative power of local features.

[0111] The first and second multi-scale feature extraction units process the single-channel enhancement maps of depth and pixels respectively to generate feature maps of different scales. Adders A1 and A2 fuse depth and pixel features of the same scale, integrating complementary information of "multimodal + multi-scale", so that features of different scales have both texture and depth features, thereby improving the accuracy of classification.

[0112] like Figure 3As shown, the two downsampling encoding modules have the same structure, both including: a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first max pooling layer, a second max pooling layer, a first average pooling layer, a second average pooling layer, and a splicing layer.

[0113] The input of the third convolutional layer serves as the input of two downsampling coding modules, and its output is connected to the input of the first max pooling layer.

[0114] The output of the first max pooling layer is connected to the input of the fourth convolutional layer; the output of the fourth convolutional layer is connected to the inputs of the second max pooling layer, the first average pooling layer, and the fifth convolutional layer; the output of the fifth convolutional layer is connected to the input of the second average pooling layer; the input of the splicing layer is connected to the outputs of the second max pooling layer, the first average pooling layer, and the second average pooling layer, and its output serves as the output of two downsampling encoding modules.

[0115] This invention achieves feature extraction from shallow texture to deep semantics through layer-by-layer processing of the third, fourth, and fifth convolutional layers, effectively capturing the spatial distribution features and texture details of face images. At the same time, it adopts a parallel branch of "max pooling + average pooling + convolution + average pooling" to extract complementary features from three dimensions. The second max pooling layer retains local peak features, the first average pooling layer reflects local mean features, and the fifth convolutional layer + the second average pooling layer captures detailed texture features. A concatenation layer is used to concatenate the three outputs to obtain a three-channel depth feature map and a three-channel pixel feature map.

[0116] In this embodiment, the third convolutional layer uses a 3×3 kernel with a stride of 1 and padding of 1; the first max pooling layer uses a 2×2 kernel with a stride of 2; the fourth convolutional layer uses a 3×3 kernel with a stride of 1 and padding of 1; the second max pooling layer uses a 2×2 kernel with a stride of 2; the first average pooling layer uses a 2×2 kernel with a stride of 2; the fifth convolutional layer uses a 3×3 kernel with a stride of 1 and padding of 1; and the second average pooling layer uses a 2×2 kernel with a stride of 2, so that the dimensions of the three-channel depth feature map and the three-channel pixel feature map are consistent with the dimensions of the three-channel depth mapping feature and the three-channel pixel mapping feature.

[0117] like Figure 4 As shown, the two multi-scale feature extraction units have the same structure, both including an upsampling layer, a sixth convolutional layer, and a seventh convolutional layer;

[0118] The input of the upsampling layer is connected to the input of the seventh convolutional layer and serves as the input of two multi-scale feature extraction units; the output of the upsampling layer is connected to the input of the sixth convolutional layer.

[0119] The output of the sixth convolutional layer serves as the first output of the two multi-scale feature extraction units; the output of the seventh convolutional layer serves as the second output of the two multi-scale feature extraction units.

[0120] Interpolation method for upsampling layers: bilinear interpolation, sampling factor ×2, the kernels of the sixth and seventh convolutional layers are 3×3, stride is 1, padding=1.

[0121] In this embodiment, the binocular multimodal local region fusion face recognition network is trained using the existing gradient descent method.

[0122] This invention, by dividing the depth map and the left image into a coarse grid and marking high and low points in each grid, can accurately extract the depth and pixel features of key local areas of the face (such as the eye sockets, bridge of the nose, and corners of the mouth). This method of extracting local features significantly improves the sensitivity to subtle geometric and texture differences, enhancing local discrimination capabilities.

[0123] This invention employs a binocular multimodal local region fusion network, jointly processing the low-resolution depth map, the left low-resolution map, the local depth feature matrix, and the local pixel feature matrix to achieve complementary fusion of depth and pixel information. This multimodal information integration avoids the weakening of local features within global features, thereby improving overall recognition performance and increasing face recognition accuracy.

[0124] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A face recognition method based on binocular local region recognition, characterized in that, Includes the following steps: A stereo camera is used to capture images of the target's facial area, resulting in a left and a right image. The depth map is then obtained based on the parallax between the left and right images. Both the left image and the depth image are divided into fine grids, and the grid mean is extracted to obtain a low-resolution depth image and a low-resolution left image. Both the left image and the depth map are divided into coarse grids. In each coarse grid, high points and low points are marked based on the depth value. Based on the high and low points in the depth map, local depth features are extracted from each coarse grid in the depth map to construct a three-channel local depth feature matrix; Based on the high and low points in the left image, local pixel features are extracted from each coarse grid in the left image to construct a three-channel local pixel feature matrix. A binocular multimodal local region fusion face recognition network is used to process the low-resolution depth map, the left low-resolution map, the three-channel local depth feature matrix, and the three-channel local pixel feature matrix to obtain the face recognition result. The processing steps of a binocular multimodal local region fusion face recognition network include: Two downsampling encoding modules are used to extract features from the depth low-resolution map and the left low-resolution map respectively, resulting in a three-channel depth feature map and a three-channel pixel feature map. Two pointwise convolutional layers are used to process the three-channel local depth feature matrix and the three-channel local pixel feature matrix respectively, to obtain the three-channel depth mapping feature and the three-channel pixel mapping feature; Multiply the three-channel depth feature map and the three-channel depth mapping feature by elements to obtain the three-channel depth feature enhancement map. Multiply the three-channel pixel feature map and the three-channel pixel mapping feature by elements to obtain the three-channel pixel feature enhancement map. Channel aggregation and multi-scale feature extraction are performed on the three-channel depth feature enhancement map and the three-channel pixel feature enhancement map respectively, and the same-scale feature fusion is performed. A fully connected layer is used to classify the fused maps of multiple scales to obtain the face recognition results. The process of obtaining facial recognition results includes: Two convolutional layers are used to perform channel aggregation on the three-channel depth feature enhancement map and the three-channel pixel feature enhancement map, respectively, to obtain a single-channel depth feature enhancement map and a single-channel pixel feature enhancement map. Two multi-scale feature extraction units are used to process the single-channel depth feature enhancement map and the single-channel pixel feature enhancement map respectively, to obtain the first and second scale depth feature enhancement maps and the first and second scale pixel feature enhancement maps; The depth feature enhancement map and the pixel feature enhancement map of the same scale are added together to obtain the first scale fusion map and the second scale fusion map; A fully connected layer is used to classify the first-scale fusion image and the first-scale fusion image to obtain the face recognition result.

2. The face recognition method based on binocular local region as described in claim 1, characterized in that, The process of obtaining the depth low-resolution map and the left low-resolution map includes: The depth map and the left image are divided using a fine mesh. Extract the mean depth value from each fine grid of the depth map, and use the mean depth value as the depth value at the corresponding fine grid to obtain a low-resolution depth map. Extract the pixel mean from each fine grid in the left image, and use the pixel mean as the pixel value at the corresponding fine grid to obtain the left low-resolution image.

3. The face recognition method based on binocular local region as described in claim 1, characterized in that, The process of marking highs and lows includes: Both the left image and the depth map are divided into coarse grids; In each coarse grid of the depth map, calculate the average grid depth; In the left image and the depth image, pixels with depth values ​​greater than the average depth of the corresponding grid in the coarse grid are marked as high points, and other points are marked as low points.

4. The face recognition method based on binocular local region as described in claim 1, characterized in that, Local depth features include: local depth contrast value, local area contrast value, and local center offset value; The local depth contrast value, local area contrast value, and local center offset value of each coarse grid are concatenated according to the channel dimension to form the local depth feature vector of the coarse grid. The local depth feature vectors of each coarse grid are arranged according to their spatial position to obtain a three-channel local depth feature matrix.

5. The face recognition method based on binocular local region as described in claim 4, characterized in that, The process of obtaining the local depth contrast value includes: in the depth map, taking the average depth value of each high point in the coarse grid to obtain the average depth value of the high points, taking the average depth value of each low point in the coarse grid to obtain the average depth value of the low points, and taking the ratio of the average depth value of the high points to the average depth value of the low points as the local depth contrast value. The process of obtaining the local area contrast value includes: in the depth map, the ratio of the number of each high point in the coarse grid to the number of each low point in the coarse grid is used as the local area contrast value; The process of obtaining the local center offset value includes: in the depth map, calculating the geometric center of each high point for each coarse grid to obtain the high point center; calculating the geometric center of each low point for each coarse grid to obtain the low point center; calculating the distance between the high point center and the low point center and normalizing the distance to obtain the local center offset value.

6. The face recognition method based on binocular local region as described in claim 1, characterized in that, Local pixel features include: local pixel contrast value, local energy contrast value, and local complexity contrast value; The local pixel contrast value, local energy contrast value, and local complexity contrast value of each coarse grid are concatenated along the channel dimension to form the local pixel feature vector of that coarse grid. The local pixel feature vectors of each coarse grid are arranged according to their spatial positions to obtain a three-channel local pixel feature matrix.

7. The face recognition method based on binocular local region as described in claim 6, characterized in that, The process of obtaining local pixel contrast values ​​includes: in the left image, calculating the pixel values ​​of each high point in each coarse grid to obtain the average pixel value of the high points, calculating the pixel values ​​of each low point in each coarse grid to obtain the average pixel value of the low points, and taking the ratio of the average pixel value of the high points to the average pixel value of the low points in the coarse grid as the local pixel contrast value. The process of obtaining the local energy contrast value includes: in the left figure, calculating the pixel gradient magnitude of each pixel in each coarse grid, and squaring the pixel gradient magnitude to obtain the texture energy; summing the texture energy of each high point to obtain the high point texture energy sum; summing the texture energy of each low point to obtain the low point texture energy sum; and taking the ratio of the high point texture energy sum to the low point texture energy sum of the coarse grid as the local energy contrast value. The process of obtaining the local complexity comparison value includes: in the left figure, calculating the variance of the pixel values ​​of each high point in each coarse grid to obtain the high point texture complexity, calculating the variance of the pixel values ​​of each low point in each coarse grid to obtain the low point texture complexity, and taking the ratio of the high point texture complexity to the low point texture complexity as the local complexity comparison value.

8. The face recognition method based on binocular local region as described in claim 1, characterized in that, The two downsampling encoding modules have the same structure, both including: a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first max pooling layer, a second max pooling layer, a first average pooling layer, a second average pooling layer, and a splicing layer; The input of the third convolutional layer serves as the input of two downsampling coding modules, and its output is connected to the input of the first max pooling layer. The output of the first max pooling layer is connected to the input of the fourth convolutional layer; the output of the fourth convolutional layer is connected to the inputs of the second max pooling layer, the first average pooling layer, and the fifth convolutional layer; the output of the fifth convolutional layer is connected to the input of the second average pooling layer; the input of the splicing layer is connected to the outputs of the second max pooling layer, the first average pooling layer, and the second average pooling layer, and its output serves as the output of two downsampling encoding modules.

Images