A face image recognition method, device, equipment and medium

Abstract

A face image recognition method, device, equipment and medium are disclosed. The method comprises: obtaining a face standardized image of a recognition object according to an original collection image of the recognition object; generating a first single-scale feature map according to the face standardized image, and obtaining a depth map and an ROI mask map of the face of the recognition object according to the first single-scale feature map; generating a second single-scale feature map according to the face standardized image, and generating a third single-scale feature map according to the depth map; obtaining a face feature mask according to the ROI mask map, and obtaining a target feature vector of the recognition object according to the second single-scale feature map, the third single-scale feature map and the face feature mask; matching the target feature vector with template feature vectors of each reference object in a template library, and determining a target reference object matched with the recognition object according to a matching result. The above technical solution can improve the face matching accuracy and realize high-precision face recognition.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a facial image recognition method, apparatus, device, and medium. Background Technology

[0002] Facial recognition technology can confirm or retrieve an individual's identity based on biometrics. With the development of facial recognition technology, it can not only be used for the analysis and detection of human faces, but also for scenarios such as pet health monitoring and pet activity analysis.

[0003] In existing technical solutions, two approaches are generally selected: one is to collect single-modal data based on an RGB camera, and the other is to collect data collaboratively through multiple devices such as an RGB camera and a depth sensor. After acquiring the data of the object to be identified, a deep neural network model is often used to extract facial data features, and image recognition is performed based on the extracted facial features.

[0004] However, the hardware cost of depth sensors is significantly higher than that of ordinary RGB cameras, and the alignment operation between RGB images and depth maps introduces additional errors. Combined with the errors in depth measurement itself, this directly affects the recognition accuracy. If recognition is based solely on single-modal data acquired by RGB cameras, the recognition effect is easily affected by environmental factors such as lighting and background, resulting in poor recognition accuracy in complex scenes. Furthermore, when using deep neural network models for facial data feature extraction, the background information introduced during facial data cropping can easily lead to a decrease in the accuracy of similarity calculation between the acquired data and the target data, thus affecting the target matching accuracy. Summary of the Invention

[0005] This invention provides a facial image recognition method, apparatus, device, and medium that can improve facial matching accuracy and achieve high-precision facial recognition.

[0006] According to one aspect of the present invention, a facial image recognition method is provided, comprising:

[0007] Based on the original captured image of the object being identified, obtain a standardized facial image of the object being identified;

[0008] Based on the standardized facial image, a first single-scale feature map is generated, and based on the first single-scale feature map, a depth map and a ROI mask map of the face of the object to be identified are obtained.

[0009] A second single-scale feature map is generated based on the facial normalization image, and a third single-scale feature map is generated based on the depth map.

[0010] Based on the ROI mask map, obtain the facial feature mask, and based on the second single-scale feature map, the third single-scale feature map, and the facial feature mask, obtain the target feature vector of the identified object.

[0011] The target feature vector is matched with the template feature vectors of each reference object in the template library, and the target reference object that matches the identified object is determined based on the matching results.

[0012] According to another aspect of the present invention, a facial image recognition device is provided, comprising:

[0013] The facial standardization image generation module is used to obtain a facial standardization image of the object to be identified based on the original captured image of the object.

[0014] The depth estimation module is used to generate a first single-scale feature map based on the normalized facial image, and to obtain the depth map and ROI mask map of the face of the object to be identified based on the first single-scale feature map.

[0015] The single-scale feature map generation module is used to generate a second single-scale feature map based on the face normalization image, and to generate a third single-scale feature map based on the depth map.

[0016] The target feature vector acquisition module is used to obtain the facial feature mask based on the ROI mask map, and to obtain the target feature vector of the object to be identified based on the second single-scale feature map, the third single-scale feature map and the facial feature mask.

[0017] The object matching module is used to match the target feature vector with the template feature vectors of each reference object in the template library, and determine the target reference object that matches the identified object based on the matching results.

[0018] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0019] At least one processor; and

[0020] A memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the facial image recognition method according to any embodiment of the present invention.

[0022] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the facial image recognition method according to any embodiment of the present invention.

[0023] The technical solution of this invention involves obtaining a standardized facial image of the object to be identified based on the original captured image of the object. A first single-scale feature map is generated based on the standardized facial image. A depth map and a ROI mask map of the object's face are then obtained based on the first single-scale feature map. A second single-scale feature map is generated based on the standardized facial image. A third single-scale feature map is generated based on the depth map. A facial feature mask is obtained based on the ROI mask map. A target feature vector of the object to be identified is obtained based on the second single-scale feature map, the third single-scale feature map, and the facial feature mask. The target feature vector is matched with the template feature vectors of various reference objects in a template library. Based on the matching results, a target reference object matching the object to be identified is determined. This method can acquire images using only a regular RGB camera, generate depth maps and extract foreground masks using a deep neural network, and generate more robust feature vectors through dual-branch feature extraction and ROI-guided multimodal fusion. Finally, high-precision facial recognition is achieved through similarity comparison, significantly shielding the interference of background information and effectively improving the accuracy of facial recognition.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a facial image recognition method provided in Embodiment 1 of the present invention;

[0027] Figure 2 This is a schematic diagram of a facial standardization image generation process provided by an embodiment of the present invention;

[0028] Figure 3 This is a flowchart of a depth estimation model provided according to an embodiment of the present invention;

[0029] Figure 4 This is a flowchart illustrating a multimodal feature extraction and fusion process according to an embodiment of the present invention;

[0030] Figure 5 This is a flowchart of another facial image recognition method provided in Embodiment 2 of the present invention;

[0031] Figure 6 This is a schematic diagram of the structure of a facial image recognition device according to Embodiment 3 of the present invention;

[0032] Figure 7 This is a schematic diagram of the structure of an electronic device that implements the facial image recognition method of this invention. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] Example 1

[0036] Figure 1 This is a flowchart of a facial image recognition method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where a reference object matching the acquired image of the object is identified in a template library. This method can be executed by a facial image recognition device, which can be implemented in hardware and / or software, and is generally configured in a computer or processor with image processing capabilities. Figure 1 As shown, the method includes:

[0037] S110. Obtain a standardized facial image of the object to be identified based on the original captured image of the object.

[0038] Optionally, the original acquired image can refer to a three-channel color image containing the object to be identified, which is directly acquired by an RGB camera. The data dimension of the original acquired image can be represented as H×W×3, where H is the height of the original image, W is the width of the original image, and 3 represents the red, green, and blue channels of RGB.

[0039] Optionally, a standardized facial image can refer to a standardized image obtained by detecting, cropping, correcting pose, and unifying the size of the facial region in the original acquired image. The data dimension is fixed and can be represented as A_H×A_W×3, where A_H and A_W are the preset standard facial image height and standard facial image width.

[0040] Obtaining a standardized facial image of the object to be identified, based on the original captured image of the object, may include:

[0041] Face detection is performed on the original captured image of the object to be identified, thereby obtaining an image of the facial region of the object; wherein, the original captured image is an RGB image;

[0042] The facial region image is scaled up and down, and an affine transformation is performed on the face based on the actual coordinates and standard coordinates of each facial feature point in the facial region image to generate a standardized facial image of the object to be recognized.

[0043] Optionally, the object to be identified can be captured by an RGB camera to obtain the original image of the object. The original image contains the complete scene background. The facial region is located using a target detection model based on a deep neural network, and the coordinates of a rectangle [X1,Y1,X2,Y2] are output. Where [X1,Y1] is the pixel coordinate of the upper left corner of the rectangle, and [X2,Y2] is the pixel coordinate of the lower right corner of the rectangle. The image within the rectangle is cropped to serve as the facial region image of the object to be identified.

[0044] Optionally, an interpolation algorithm can be used to uniformly scale the facial region image to a preset standard facial image height and width, thereby eliminating the facial size difference caused by different acquisition distances. At this time, the image dimension becomes A_H×A_W×3, but there may be pose offset.

[0045] Optionally, facial feature points can refer to key points that are distinctive on the face, such as the center of the left eye, the center of the right eye, the tip of the nose, and the corners of the mouth. The actual coordinates of the facial feature points can refer to the pixel positions of the feature points in the facial region image, while the standard coordinates can refer to the pixel positions of the feature points under a preset ideal frontal face pose.

[0046] Optionally, facial affine transformation can refer to rotating, translating, and scaling misaligned facial region images by calculating the mapping relationship between actual feature point coordinates and standard feature point coordinates, thereby ensuring the uniformity of facial feature point positions across different acquisition postures.

[0047] Figure 2 This is a schematic diagram of an optional facial normalization image generation process. (Example) Figure 2 As shown, the RGB Image is the original image of the object to be identified, with a height of H and a width of W. The original image is an RGB three-channel image. The original image is input into the facial target detection model. Based on the facial detection results, the pixel coordinates [X1,Y1,X2,Y2] of the rectangular box of the facial region are obtained, where [X1,Y1] is the pixel coordinate of the upper left corner of the rectangle, and [X2,Y2] is the pixel coordinate of the lower right corner of the rectangle. Then, the image within the rectangular box of the facial region is input into the facial alignment module. The facial alignment module performs cropping and alignment on the content of the facial region image to obtain a facial alignment result image with a data dimension of A_H×A_W×3, which is the facial standardization image described in this embodiment of the invention.

[0048] S120. Generate a first single-scale feature map based on the standardized facial image, and obtain the depth map and ROI mask map of the face of the object to be identified based on the first single-scale feature map.

[0049] Optionally, by performing depth estimation on the standardized facial image, a depth map of the identified object's face and a ROI (Region of Interest) mask map can be generated. The depth map is a single-channel grayscale image, with a data dimension of A_H×A_W×1. The pixel values ​​of the depth map are floating-point numbers in the range [0,1]. The larger the value, the closer the corresponding region is to the camera. The ROI mask map is a single-channel binary image, with a data dimension of A_H×A_W×1. The pixel values ​​are only 0 or 1, where 1 represents the region of interest, such as a pet's face, and 0 represents the background region, such as the floor or wall in the background. Here, A_H and A_W are the preset standard facial image height and standard facial image width.

[0050] Optionally, the height and width of the standardized facial image, depth map, and ROI mask image are the same, both being preset standard facial image height and standard facial image width.

[0051] Optionally, a depth map can be generated by depth estimation, which can fill in the missing spatial depth information of the RGB modality and improve the recognition robustness. By generating an ROI mask map, background interference can be accurately blocked and key facial features can be focused.

[0052] Specifically, based on the standardized facial image, a first single-scale feature map is generated, and based on the first single-scale feature map, a depth map of the face of the object to be identified and a ROI mask map are obtained, which may include:

[0053] Based on the first feature map dimension, the facial standardized image is downsampled multiple times to obtain the first set of multi-scale feature maps of the facial standardized image, and the first set of multi-scale feature maps are fused to generate the first single-scale feature map.

[0054] The first single-scale feature map is input into the decoding module to obtain the depth map and ROI mask map of the face of the object to be identified; wherein, the depth map is a single-channel grayscale image and the ROI mask map is a single-channel binary image.

[0055] Optionally, the first feature map dimension can be preset. The first feature map dimension can be used to extract the first set of multi-scale feature maps of the face normalization image. For example, the first feature map dimension can be set to F_H×F_W×F_C, where F_H is the preset first feature map height, F_W is the preset first feature map width, and F_C is the preset number of first feature map channels.

[0056] Optionally, downsampling can gradually reduce the spatial size of the feature map. After each downsampling process, the height and width of the resulting image are halved compared to the image before downsampling, while the number of channels is increased.

[0057] Optionally, the multi-scale feature map may include three feature maps of different scales. When the dimension of the first feature map is set to F_H×F_W×F_C, the first set of multi-scale feature maps after downsampling may include feature maps with scales of F_H×F_W×F_C, (1 / 2×F_H)×(1 / 2×F_W)×(2×F_C), and (1 / 4×F_H)×(1 / 4×F_W)×(4×F_C).

[0058] Optionally, a normalized facial image can be input into a backbone network that uses a convolutional neural network as a depth estimation model, and the first set of multi-scale feature maps output by the backbone network can be obtained.

[0059] Optionally, the first set of multi-scale feature maps can be input into the feature fusion module of the depth estimation model. The feature fusion module will then output a single-scale, high-dimensional first single-scale feature map. The feature dimension of the first single-scale feature map is the same as the preset first feature map dimension. The feature dimension of the first single-scale feature map can be F_H×F_W×F_C. The first single-scale feature map obtained after downsampling and feature fusion can take into account both detailed and global information.

[0060] Optionally, the first single-scale feature map can be input into the decoding module of the depth estimation model, and the decoding module can output the depth map and the RIO mask map respectively.

[0061] Optional, Figure 3 This is a flowchart illustrating the processing of an optional depth estimation model. (e.g.) Figure 3 As shown, a convolutional neural network is used as the backbone network of the depth estimation model. The normalized facial image, i.e., the facial alignment result image, is input into the backbone network to extract the corresponding multi-scale feature maps. The dimensions of the multi-scale feature maps are F_H×F_W×F_C, (1 / 2×F_H)×(1 / 2×F_W)×(2×F_C), and (1 / 4×F_H)×(1 / 4×F_W)×(4×F_C). Then, the multi-scale feature maps are processed by the feature fusion module and the decoding module to obtain the corresponding depth map and ROI mask. The depth map data dimension is A_H×A_W×1, and the ROI mask data dimension is A_H×A_W×1.

[0062] S130. Generate a second single-scale feature map based on the standardized facial image, and generate a third single-scale feature map based on the depth map.

[0063] Optionally, the second single-scale feature map can be used to represent RGB modal features, and the third single-scale feature map can be used to represent depth modal features.

[0064] Specifically, a second single-scale feature map is generated based on the standardized facial image, and a third single-scale feature map is generated based on the depth map, including:

[0065] Based on the second feature map dimension, the face normalization image and the depth map are downsampled multiple times to obtain the second set of multi-scale feature maps of the face normalization image and the multi-scale feature maps of the depth map.

[0066] The second set of multi-scale feature maps of the face-normalized image are fused to generate the second single-scale feature map, and the multi-scale feature maps of the depth map are fused to generate the third single-scale feature map.

[0067] Optionally, the second feature map dimension can be preset. The second feature map dimension can be used to extract the second set of multi-scale feature maps of the face normalization image and the multi-scale feature maps of the depth map. For example, the second feature map dimension can be set to fea_H×fea_W×fea_C, where fea_H is the preset second feature map height, fea_W is the preset second feature map width, and fea_C is the preset number of second feature map channels.

[0068] Optionally, when the dimension of the second feature map is set to fea_H×fea_W×fea_C, the second set of multi-scale feature maps after downsampling and the multi-scale feature maps of the depth map may include feature maps with scales of fea_H×fea_W×fea_C, (1 / 2×fea_H)×(1 / 2×fea_W)×(2×fea_C), and (1 / 4×fea_H)×(1 / 4×fea_W)×(4×fea_C), respectively.

[0069] Optionally, the normalized facial image and the depth image can be input into the backbone network using a convolutional neural network as the depth estimation model. The backbone network downsamples based on the second feature map dimension and outputs the second set of multi-scale feature maps of the normalized facial image and the multi-scale feature maps of the depth image, respectively. The second set of multi-scale feature maps can focus on the color and texture features of the RGB modality and are the core feature representations of the RGB modality. The multi-scale feature maps of the depth image can focus on the spatial depth features of the depth modality and are the core feature representations of the depth modality.

[0070] Optionally, the second set of multi-scale feature maps can be input into the RGB feature map fusion module, and the feature fusion module can finally output a single-scale, high-dimensional second single-scale feature map. The multi-scale feature maps of the depth map can be input into the depth feature map fusion module to obtain the third single-scale feature map finally output by the depth feature map fusion module.

[0071] Optionally, the feature dimensions of the second and third single-scale feature maps are the same as those of the second feature map, which are both fea_H×fea_W×fea_C.

[0072] S140. Based on the ROI mask map, obtain the facial feature mask, and based on the second single-scale feature map, the third single-scale feature map, and the facial feature mask, obtain the target feature vector of the object to be identified.

[0073] Optionally, a facial feature mask can be obtained by downsampling the ROI mask image. The facial feature mask can be adapted to the feature map size of the second and third single-scale feature maps. The data dimension of the facial feature mask can be represented as fea_H×fea_W×1. The facial feature mask can support the filtering of effective features and the removal of background features during multimodal feature fusion.

[0074] Optionally, the second single-scale feature map, the third single-scale feature map, and the facial feature mask can be input into the multimodal feature fusion module to obtain the target feature vector output by the multimodal feature fusion module.

[0075] Optionally, in the multimodal feature fusion module, facial feature masks can be used to filter the second single-scale feature map and the third single-scale feature map, retaining only facial features and removing background features. After feature filtering, the retained features are fused to obtain the fused target feature vector.

[0076] Optionally, the target feature vector is a one-dimensional feature representation, and the vector dimension can be represented as 1×fea_dim, where fea_dim is the feature dimension, such as 512, 1024, etc.

[0077] Specifically, based on the ROI mask map, a facial feature mask is obtained, and based on the second single-scale feature map, the third single-scale feature map, and the facial feature mask, the target feature vector of the identified object is obtained, including:

[0078] Based on the image sizes of the second and third single-scale feature maps, the ROI mask image is downsampled to obtain the facial feature mask.

[0079] The second single-scale feature map, the third single-scale feature map, and the facial feature mask are subjected to feature fusion processing to generate the target feature vector of the identified object; wherein, the target feature vector is a one-dimensional feature vector.

[0080] Optionally, the average pixel value of each target region can be calculated based on the image size of the second single-scale feature map and the third single-scale feature map, that is, according to the height fea_H and width fea_W of the second feature map, and the facial feature mask with size fea_H×fea_W×1 can be output. This can avoid boundary distortion under a fixed step size and ensure the spatial correspondence between the mask and the feature map.

[0081] Optionally, the second single-scale feature map, the third single-scale feature map, and the facial feature mask can be input into the multimodal feature fusion module. The multimodal feature fusion module combines RGB and depth modal features to obtain the final one-dimensional feature vector as the target feature vector, thereby eliminating spatial dimensional redundancy and retaining only global feature information. The target feature vector can be directly used for similarity calculation, improving computational efficiency.

[0082] Figure 4 This is a flowchart illustrating an optional multimodal feature extraction and fusion process. Figure 4As shown, the normalized facial image and depth map can be input into the multimodal feature extraction and fusion model, respectively. The model adopts a dual-branch structure, with each branch using an independent backbone network to generate multi-scale feature maps for the normalized facial image and depth map, respectively. The data dimensions of the second set of multi-scale feature maps for the normalized facial image are fea_H×fea_W×fea_C, (1 / 2×fea_H)×(1 / 2×fea_W)×(2×fea_C), and (1 / 4×fea_H)×(1 / 4×fea_W)×(4×fea_C), respectively. The data dimensions of the multi-scale feature maps for the depth map are fea_H×fea_W×fea_C and (1 / 2×fea_H)×(1 / 2×fea_W), respectively. ×(2×fea_C) and (1 / 4×fea_H)×(1 / 4×fea_W)×(4×fea_C), and then the second set of multi-scale feature maps of the face normalization image and the multi-scale feature maps of the depth map are respectively processed by the feature map fusion module to obtain their respective fused single-scale feature maps. The data dimensions are the same, both being fea_H×fea_W×fea_C. Then, the ROI mask map is downsampled through an adaptive average pooling layer to obtain the feature map mask. The data dimension of the feature map mask is fea_H×fea_W×1. Through the multi-modal feature fusion module, the second single-scale feature map, the third single-scale feature map, and the feature map mask are fused to obtain the feature vector of the identified object, with a data dimension of 1×fea_dim.

[0083] S150. Match the target feature vector with the template feature vectors of each reference object in the template library, and determine the target reference object that matches the identified object based on the matching results.

[0084] Optionally, the template library stores the facial image of the reference object, the identity identifier of the reference object, and the template feature vector. The generation process of the template feature vector is the same as the process of generating the target feature vector. If the target reference object is successfully matched for the recognition object, the identity identifier of the reference object can be used as the facial recognition result of the recognition object.

[0085] This process involves matching the target feature vector with the template feature vectors of each reference object in the template library, and determining the target reference object that matches the identified object based on the matching results. This may include:

[0086] Calculate the similarity between the target feature vector and the template feature vectors of each reference object in the template library;

[0087] If the similarity between the target feature vector and the target template feature vector is the highest and the similarity is greater than the preset similarity threshold, then the target reference object is determined based on the target template feature vector, and the identification object is determined to match the target reference object.

[0088] Optionally, template feature vectors and corresponding object information of all reference objects can be read from the template library, such as pet identifier, pet name, and breed. Then, based on the batch cosine similarity algorithm, the cosine similarity between the target feature vector and all template feature vectors in the template library can be calculated at once to obtain a set of similarity values. All similarity values ​​are sorted from largest to smallest, and the highest similarity value and the corresponding target template feature vector are selected. If the highest similarity value is greater than a preset threshold, the reference object corresponding to the target template feature vector is determined to be the target reference object, and the information of the target reference object is output. For example, the output can be: Match successful, Pet ID: 123, Name: Milk Candy, Breed: Bichon Frise; If the highest similarity value is less than or equal to the preset threshold, it is determined that no valid reference object was matched, and a recognition failure message is output.

[0089] The technical solution of this invention involves obtaining a standardized facial image of the object to be identified based on the original captured image of the object. A first single-scale feature map is generated based on the standardized facial image. A depth map and a ROI mask map of the object's face are then obtained based on the first single-scale feature map. A second single-scale feature map is generated based on the standardized facial image. A third single-scale feature map is generated based on the depth map. A facial feature mask is obtained based on the ROI mask map. A target feature vector of the object to be identified is obtained based on the second single-scale feature map, the third single-scale feature map, and the facial feature mask. The target feature vector is matched with the template feature vectors of various reference objects in a template library. Based on the matching results, a target reference object matching the object to be identified is determined. This method can acquire images using only a regular RGB camera, generate depth maps and extract foreground masks using a deep neural network, and generate more robust feature vectors through dual-branch feature extraction and ROI-guided multimodal fusion. Finally, high-precision facial recognition is achieved through similarity comparison, significantly shielding the interference of background information and effectively improving the accuracy of facial recognition.

[0090] Example 2

[0091] Figure 5 This is a flowchart of a facial image recognition method provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment specifically describes the facial image recognition method. Figure 5 As shown, the method includes:

[0092] S210. Perform face detection on the original acquired image of the object to obtain the facial region image of the object.

[0093] The original acquired image is an RGB image.

[0094] S220. The facial region image is scaled up and down, and facial affine transformation is performed based on the actual coordinates of each facial feature point in the facial region image and the standard coordinates of each facial feature point to generate a standardized facial image of the object to be recognized.

[0095] S230. Based on the dimension of the first feature map, the standardized facial image is downsampled multiple times to obtain the first set of multi-scale feature maps of the standardized facial image, and the first set of multi-scale feature maps is fused to generate the first single-scale feature map.

[0096] S240. Input the first single-scale feature map into the decoding module to obtain the depth map of the face of the object to be identified and the ROI mask map.

[0097] The depth map is a single-channel grayscale image, and the ROI mask image is a single-channel binary image.

[0098] S250. Based on the second feature map dimension, perform multiple downsampling processes on the standardized facial image and the depth map respectively to obtain the second set of multi-scale feature maps of the standardized facial image and the multi-scale feature maps of the depth map.

[0099] S260. The second set of multi-scale feature maps of the face normalization image are fused to generate the second single-scale feature map, and the multi-scale feature maps of the depth map are fused to generate the third single-scale feature map.

[0100] S270. Based on the image sizes of the second and third single-scale feature maps, downsample the ROI mask map to obtain the facial feature mask.

[0101] S280. Perform feature fusion processing on the second single-scale feature map, the third single-scale feature map, and the facial feature mask to generate the target feature vector of the identified object.

[0102] The target feature vector is a one-dimensional feature vector.

[0103] S290. Calculate the similarity between the target feature vector and the template feature vectors of each reference object in the template library.

[0104] S2100. If the similarity between the target feature vector and the target template feature vector is the highest and the similarity is greater than the preset similarity threshold, then the target reference object is determined based on the target template feature vector, and the identification object is determined to match the target reference object.

[0105] Furthermore, before obtaining the standardized facial image of the object to be identified based on the original captured image of the object, the process may further include:

[0106] Acquire RGB images of each reference object, and based on the RGB images, obtain the standardized facial images of each reference object respectively;

[0107] Multimodal information fusion is performed on the standardized facial images of each reference object to obtain the template feature vector of each reference object, and a template library is established based on the template feature vector of each reference object.

[0108] Optionally, based on the RGB acquired images, standardized facial images of each reference object can be obtained, which may include:

[0109] Face detection is performed on the RGB acquired image of the reference object to obtain the facial region image of the reference object; the facial region image of the reference object is scaled up, and an affine transformation of the face is performed based on the actual coordinates and standard coordinates of each facial feature point in the facial region image of the reference object to generate a standardized facial image of the reference object.

[0110] Optionally, multimodal information fusion can be performed on the standardized facial images of each reference object to obtain template feature vectors for each reference object, which may include:

[0111] Based on the standardized facial image of the reference object, a first single-scale feature map is generated, and based on the first single-scale feature map, a depth map of the reference object's face and a mask map of the region of interest (ROI) are obtained.

[0112] A second single-scale feature map is generated based on the facial normalization image, and a third single-scale feature map is generated based on the depth map.

[0113] Based on the ROI mask map, obtain the facial feature mask, and based on the second single-scale feature map, the third single-scale feature map, and the facial feature mask, obtain the template feature vector of the reference object.

[0114] Optionally, the template feature vector generation method for each reference object in the template library is exactly the same as the target feature vector generation method for the identified object, which will not be elaborated here.

[0115] The technical solution of this invention involves obtaining a standardized facial image of the object to be identified based on the original captured image of the object. A first single-scale feature map is generated based on the standardized facial image. A depth map and a ROI mask map of the object's face are then obtained based on the first single-scale feature map. A second single-scale feature map is generated based on the standardized facial image. A third single-scale feature map is generated based on the depth map. A facial feature mask is obtained based on the ROI mask map. A target feature vector of the object to be identified is obtained based on the second single-scale feature map, the third single-scale feature map, and the facial feature mask. The target feature vector is matched with the template feature vectors of various reference objects in a template library. Based on the matching results, a target reference object matching the object to be identified is determined. This method can acquire images using only a regular RGB camera, generate depth maps and extract foreground masks using a deep neural network, and generate more robust feature vectors through dual-branch feature extraction and ROI-guided multimodal fusion. Finally, high-precision facial recognition is achieved through similarity comparison, significantly shielding the interference of background information and effectively improving the accuracy of facial recognition.

[0116] Example 3

[0117] Figure 6 This is a schematic diagram of the structure of a facial image recognition device provided in Embodiment 3 of the present invention. Figure 6 As shown, the device includes: a face normalization image generation module 310, a depth estimation module 320, a single-scale feature map generation module 330, a target feature vector acquisition module 340, and an object matching module 350.

[0118] The facial standardization image generation module 310 is used to obtain a facial standardization image of the object to be identified based on the original acquired image of the object.

[0119] The depth estimation module 320 is used to generate a first single-scale feature map based on the facial normalization image, and to obtain the depth map and ROI mask map of the face of the object to be identified based on the first single-scale feature map.

[0120] The single-scale feature map generation module 330 is used to generate a second single-scale feature map based on the face-normalized image and a third single-scale feature map based on the depth map.

[0121] The target feature vector acquisition module 340 is used to obtain a facial feature mask based on the ROI mask map, and to obtain the target feature vector of the object to be identified based on the second single-scale feature map, the third single-scale feature map and the facial feature mask.

[0122] The object matching module 350 is used to match the target feature vector with the template feature vector of each reference object in the template library, and determine the target reference object that matches the identified object based on the matching result.

[0123] The technical solution of this invention involves obtaining a standardized facial image of the object to be identified based on the original captured image of the object. A first single-scale feature map is generated based on the standardized facial image. A depth map and a ROI mask map of the object's face are then obtained based on the first single-scale feature map. A second single-scale feature map is generated based on the standardized facial image. A third single-scale feature map is generated based on the depth map. A facial feature mask is obtained based on the ROI mask map. A target feature vector of the object to be identified is obtained based on the second single-scale feature map, the third single-scale feature map, and the facial feature mask. The target feature vector is matched with the template feature vectors of various reference objects in a template library. Based on the matching results, a target reference object matching the object to be identified is determined. This method can acquire images using only a regular RGB camera, generate depth maps and extract foreground masks using a deep neural network, and generate more robust feature vectors through dual-branch feature extraction and ROI-guided multimodal fusion. Finally, high-precision facial recognition is achieved through similarity comparison, significantly shielding the interference of background information and effectively improving the accuracy of facial recognition.

[0124] Based on the above embodiments, the facial standardization image generation module 310 can be specifically used for:

[0125] Face detection is performed on the original captured image of the object to be identified, thereby obtaining an image of the facial region of the object; wherein, the original captured image is an RGB image;

[0126] The facial region image is scaled up and down, and an affine transformation is performed on the face based on the actual coordinates and standard coordinates of each facial feature point in the facial region image to generate a standardized facial image of the object to be recognized.

[0127] Based on the above embodiments, the depth estimation module 320 can be specifically used for:

[0128] Based on the first feature map dimension, the facial standardized image is downsampled multiple times to obtain the first set of multi-scale feature maps of the facial standardized image, and the first set of multi-scale feature maps are fused to generate the first single-scale feature map.

[0129] The first single-scale feature map is input into the decoding module to obtain the depth map and ROI mask map of the face of the object to be identified; wherein, the depth map is a single-channel grayscale image and the ROI mask map is a single-channel binary image.

[0130] Based on the above embodiments, the single-scale feature map generation module 330 can be specifically used for:

[0131] Based on the second feature map dimension, the face normalization image and the depth map are downsampled multiple times to obtain the second set of multi-scale feature maps of the face normalization image and the multi-scale feature maps of the depth map.

[0132] The second set of multi-scale feature maps of the face-normalized image are fused to generate the second single-scale feature map, and the multi-scale feature maps of the depth map are fused to generate the third single-scale feature map.

[0133] Based on the above embodiments, the target feature vector acquisition module 340 can be specifically used for:

[0134] Based on the image sizes of the second and third single-scale feature maps, the ROI mask image is downsampled to obtain the facial feature mask.

[0135] The second single-scale feature map, the third single-scale feature map, and the facial feature mask are subjected to feature fusion processing to generate the target feature vector of the identified object; wherein, the target feature vector is a one-dimensional feature vector.

[0136] Based on the above embodiments, the object matching module 350 can be specifically used for:

[0137] Calculate the similarity between the target feature vector and the template feature vectors of each reference object in the template library;

[0138] If the similarity between the target feature vector and the target template feature vector is the highest and the similarity is greater than the preset similarity threshold, then the target reference object is determined based on the target template feature vector, and the identification object is determined to match the target reference object.

[0139] Based on the above embodiments, a template library creation module may also be included, used for:

[0140] Acquire RGB images of each reference object, and based on the RGB images, obtain the standardized facial images of each reference object respectively;

[0141] Multimodal information fusion is performed on the standardized facial images of each reference object to obtain the template feature vector of each reference object, and a template library is established based on the template feature vector of each reference object.

[0142] The facial image recognition device provided in the embodiments of the present invention can execute the facial image recognition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0143] Example 4

[0144] Figure 7 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0145] like Figure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0146] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0147] Processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing unit (CPU), graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the facial image recognition method described in the embodiments of the present invention. That is:

[0148] Based on the original captured image of the object being identified, obtain a standardized facial image of the object being identified;

[0149] Based on the standardized facial image, a first single-scale feature map is generated, and based on the first single-scale feature map, a depth map and a ROI mask map of the face of the object to be identified are obtained.

[0150] A second single-scale feature map is generated based on the facial normalization image, and a third single-scale feature map is generated based on the depth map.

[0151] Based on the ROI mask map, obtain the facial feature mask, and based on the second single-scale feature map, the third single-scale feature map, and the facial feature mask, obtain the target feature vector of the identified object.

[0152] The target feature vector is matched with the template feature vectors of each reference object in the template library, and the target reference object that matches the identified object is determined based on the matching results.

[0153] In some embodiments, the facial image recognition method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the facial image recognition method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the facial image recognition method by any other suitable means (e.g., by means of firmware).

[0154] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0155] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0156] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0157] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0158] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0159] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0160] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0161] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A facial image recognition method, characterized in that, include: Based on the original captured image of the object being identified, obtain a standardized facial image of the object being identified; Based on the standardized facial image, a first single-scale feature map is generated, and based on the first single-scale feature map, a depth map of the face of the object to be identified and a mask map of the region of interest (ROI) are obtained. A second single-scale feature map is generated based on the facial normalization image, and a third single-scale feature map is generated based on the depth map. Based on the ROI mask map, obtain the facial feature mask, and based on the second single-scale feature map, the third single-scale feature map, and the facial feature mask, obtain the target feature vector of the identified object. The target feature vector is matched with the template feature vectors of each reference object in the template library, and the target reference object that matches the identified object is determined based on the matching results.

2. The method according to claim 1, characterized in that, Based on the original captured image of the object to be identified, obtain a standardized facial image of the object, including: Face detection is performed on the original captured image of the object to be identified, thereby obtaining an image of the facial region of the object; wherein, the original captured image is an RGB image; The facial region image is scaled up and down, and an affine transformation is performed on the face based on the actual coordinates and standard coordinates of each facial feature point in the facial region image to generate a standardized facial image of the object to be recognized.

3. The method according to claim 1, characterized in that, Based on the standardized facial image, a first single-scale feature map is generated. Then, based on this first single-scale feature map, a depth map and a Region of Interest (ROI) mask map of the identified object's face are obtained, including: Based on the first feature map dimension, the facial standardized image is downsampled multiple times to obtain the first set of multi-scale feature maps of the facial standardized image, and the first set of multi-scale feature maps are fused to generate the first single-scale feature map. The first single-scale feature map is input into the decoding module to obtain the depth map and ROI mask map of the face of the object to be identified; wherein, the depth map is a single-channel grayscale image and the ROI mask map is a single-channel binary image.

4. The method according to claim 1, characterized in that, Based on the normalized facial image, a second single-scale feature map is generated, and based on the depth map, a third single-scale feature map is generated, including: Based on the second feature map dimension, the face normalization image and the depth map are downsampled multiple times to obtain the second set of multi-scale feature maps of the face normalization image and the multi-scale feature maps of the depth map. The second set of multi-scale feature maps of the face-normalized image are fused to generate the second single-scale feature map, and the multi-scale feature maps of the depth map are fused to generate the third single-scale feature map.

5. The method according to claim 1, characterized in that, Based on the ROI mask map, obtain the facial feature mask, and based on the second single-scale feature map, the third single-scale feature map, and the facial feature mask, obtain the target feature vector of the identified object, including: Based on the image sizes of the second and third single-scale feature maps, the ROI mask image is downsampled to obtain the facial feature mask. The second single-scale feature map, the third single-scale feature map, and the facial feature mask are subjected to feature fusion processing to generate the target feature vector of the identified object; wherein, the target feature vector is a one-dimensional feature vector.

6. The method according to claim 1, characterized in that, The target feature vector is matched with the template feature vectors of each reference object in the template library. Based on the matching results, the target reference objects that match the identified object are determined, including: Calculate the similarity between the target feature vector and the template feature vectors of each reference object in the template library; If the similarity between the target feature vector and the target template feature vector is the highest and the similarity is greater than the preset similarity threshold, then the target reference object is determined based on the target template feature vector, and the identification object is determined to match the target reference object.

7. The method according to claim 1, characterized in that, Before obtaining the standardized facial image of the object to be identified based on the original captured image of the object, the process also includes: Acquire RGB images of each reference object, and based on the RGB images, obtain the standardized facial images of each reference object respectively; Multimodal information fusion is performed on the standardized facial images of each reference object to obtain the template feature vector of each reference object, and a template library is established based on the template feature vector of each reference object.

8. A facial image recognition device, characterized in that, include: The facial standardization image generation module is used to obtain a facial standardization image of the object to be identified based on the original captured image of the object. The depth estimation module is used to generate a first single-scale feature map based on the normalized facial image, and to obtain the depth map of the face of the object to be identified and the mask map of the region of interest (ROI) based on the first single-scale feature map. The single-scale feature map generation module is used to generate a second single-scale feature map based on the face normalization image, and to generate a third single-scale feature map based on the depth map. The target feature vector acquisition module is used to obtain the facial feature mask based on the ROI mask map, and to obtain the target feature vector of the object to be identified based on the second single-scale feature map, the third single-scale feature map and the facial feature mask. The object matching module is used to match the target feature vector with the template feature vectors of each reference object in the template library, and determine the target reference object that matches the identified object based on the matching results.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the facial image recognition method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the facial image recognition method according to any one of claims 1-7.

Images