Face recognition method and system and storage medium
Through the combination of cross-modal candidate-guided feature mining and adversarial generation network, the problem of poor recognition effect of dual-light facial recognition in extreme environments is solved, effective feature extraction and fusion of unregistered images is achieved, and the Identification accuracy and robustness.
Patent Information
- Application Number
- CN202411032658.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-08
AI Technical Summary
The existing dual-light facial recognition technology has poor recognition effect in extreme lighting and harsh environments, and it is difficult to effectively extract and fusion of unregistered dual-light images, resulting in low recognition accuracy.
The cross-modal candidate-guided feature mining mechanism and the adversarial generation network are used to fusion of dual-light image features. The features of visible and thermal infrared images are extracted through an adversarial game architecture, combined with complementary information and retained face details, and used mask guidance. The face detail information discriminator improves recognition accuracy.
It improves the robustness and accuracy of face recognition in extreme lighting and harsh environments, can effectively process unregistered dual-light images, extract discriminative features, and enhances the robustness and recognition performance of the model.
Smart Images

Figure CN118918625A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a face recognition method, system and storage medium. Background Art
[0002] Face recognition is an important research direction in computer vision and is widely used in video surveillance, human-computer interaction, equipment security and other fields. The task of face recognition is to identify the identity of a person after obtaining a given face image. However, general face recognition technology usually uses visible light images for face recognition. In harsh environments (such as extreme lighting environments, occlusion, interference from similar objects, etc.), general face recognition technology cannot provide reliable recognition capabilities. Therefore, some studies use dual-light face recognition to provide more reliable and accurate face recognition capabilities by combining images of different modalities.
[0003] Cross-modal face recognition problems can be divided into two categories: feature-based cross-modal face recognition and image-generation-based cross-modal face recognition. In feature-based cross-modal face recognition, traditional methods are committed to mapping image features of different modalities into subspaces to achieve cross-modal feature extraction and perform face recognition. Some deep learning methods extract features from two modalities separately and then perform face recognition based on the features. The essence of the above-mentioned traditional feature-based methods and deep learning methods is to map features of different modalities into subspaces.
[0004] The difference between the image generation-based cross-modal face recognition method and the feature-based cross-modal face recognition algorithm is that the image generation method refers to the fusion of the two modal images by building a generative network to generate a new image based on a given visible light-thermal infrared image pair. After that, further recognition is performed based on the fused image. In the image generation-based cross-modal face recognition method, the model extracts unique and discriminative features from the cross-modal image through a generative method to improve the accuracy of face recognition.
[0005] Defects of background technology:
[0006] Data requirements: In existing dual-light face recognition algorithms, the input image is usually required to be a registered dual-light image, and the feature points of the face need to be aligned. However, in real life, it is difficult to obtain naturally aligned image pairs, and directly using unaligned image pairs will result in poor performance of the dual-light face recognition algorithm.
[0007] Feature fusion methods: In classic cross-modal face recognition algorithms, the feature extraction process focuses on extracting consistent features from cross-modal information, while ignoring complementary features and failing to fully utilize the unique information contained in different modalities.
[0008] In terms of facial identification information: the facial detail information contained in the fused features extracted by the existing cross-modal face recognition algorithm is often not sufficient, which will result in the fused image features being unable to provide sufficient facial identification information for face recognition needs. Summary of the invention
[0009] The present invention provides a face recognition method, comprising the following steps:
[0010] Image fusion steps: the bi-optical image is input into the data mining module. The data mining module extracts the corresponding features of different modalities from the unregistered bi-optical image through the feature mining mechanism guided by cross-modal candidates to form visible light features and thermal infrared features. The visible light features and thermal infrared features are fused through the feature fusion module. The bi-optical image includes the visible light image and the thermal infrared image.
[0011] Face recognition step: The fused image is input into the face recognition module for face recognition.
[0012] As a further improvement of the present invention, the face recognition module includes a face feature extraction module and a feature matching module, the face feature extraction module is used for extracting face features, and the feature matching module is used for matching face features.
[0013] As a further improvement of the present invention, the feature fusion module is a generative adversarial network.
[0014] As a further improvement of the present invention, the generative adversarial network is an adversarial game architecture consisting of a generator and a discriminator.
[0015] The present invention also provides a face recognition system, comprising: a memory, a processor, and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the face recognition method of the present invention when called by the processor.
[0016] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the face recognition method of the present invention when called by a processor.
[0017] The beneficial effects of the present invention are: 1. By using a generative adversarial network to fuse bi-light features and perform face recognition based on the fused features, the robustness of the face recognition algorithm under challenges such as extreme lighting and bad weather is improved. 2. Through a feature mining mechanism guided by cross-modal candidates, the corresponding features of the two modalities are extracted from unregistered bi-light images, and further feature fusion is performed, so that the model can still perform accurate face recognition when an unregistered image is input, thereby improving the robustness of the model. 3. By introducing a dual generative adversarial network, not only the consistency information in the bi-light image is paid attention to, but also the complementary information therein is extracted, and the detail information in the visible light image and the structural information in the thermal infrared image are combined to improve the face recognition performance of the model in harsh environments. 4. By proposing a mask-guided facial detail information discriminator, the detail information in the fused face is identified, and the model is guided to retain more facial detail information, thereby improving the model accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 It is the algorithm architecture diagram of the present invention;
[0019] Figure 2 This is the architecture diagram of the generator in the adversarial generation network;
[0020] Figure 3 This is the architecture diagram of the feature fusion module. DETAILED DESCRIPTION
[0021] In order to solve the problems in the prior art, the present invention provides an RGBT dual-light face recognition method to improve the performance of the face recognition algorithm under extreme lighting, bad weather and other environments. Specifically, the following steps are included:
[0022] Image fusion steps: the bi-optical image is input into the data mining module. The data mining module extracts the corresponding features of different modalities from the unregistered bi-optical image through the feature mining mechanism guided by cross-modal candidates to form visible light features and thermal infrared features. The visible light features and thermal infrared features are fused through the feature fusion module. The bi-optical image includes the visible light image and the thermal infrared image.
[0023] Face recognition step: The fused image is input into the face recognition module for face recognition. The face recognition module includes a face feature extraction module and a feature matching module. The face feature extraction module is used for extracting face features, and the feature matching module is used for matching face features.
[0024] 1. The framework of the present invention:
[0025] The main framework of the algorithm proposed in this invention is Figure 1As shown in the figure, the main modules include: image fusion module and face recognition module. During the training process, the bi-optical image is first used as the input of the image fusion module for cross-modal feature fusion. In order to improve the robustness of the model, the image fusion module will extract the corresponding features of different modalities from the unregistered bi-optical image through the feature mining mechanism guided by cross-modal candidates, and then fuse the features through the feature fusion module based on the adversarial generative network. The fused image will be input into the face recognition module for further face recognition: the main process is the extraction of facial features and facial feature matching.
[0026] 2. Feature fusion module based on adversarial generation network:
[0027] In order to solve the problem that single-modal face recognition is highly sensitive to the environment, easily affected by light intensity, and has poor performance in bad weather and environment, the present invention adopts a generative adversarial network as a feature fusion module to fuse visible light image features and thermal infrared image features. In the thermal infrared image and visible light image fusion module, the present invention constructs an adversarial game architecture composed of a generator and a discriminator to extract unique features from thermal infrared images and visible light images for feature fusion. The training purpose of the generator is to generate a fused image close to reality to deceive the two discriminators. In contrast, the purpose of the discriminator is to evaluate whether the generated image is consistent with the true face image.
[0028] The generator of the adversarial generative network proposed in the present invention is as follows Figure 2 As shown in the figure, it is built based on the encoder-decoder structure. The encoder consists of a feature extraction module and a feature enhancement module. The feature extraction module consists of a convolutional neural network module and a module based on Transformer. The dual-light feature will be directly input into the convolutional neural network module. At the same time, it will be divided into multiple image blocks and then used as the input of the Transformer module. The feature enhancement module merges the outputs of the convolutional neural network module and the Transformer module, and further enhances the features through the feature enhancement module based on the convolutional neural network. Finally, the output image will pass through the decoder composed of the convolutional neural network network to obtain the fused image.
[0029] 3. Feature mining mechanism guided by cross-modal candidate items:
[0030] In order to solve the problem that bi-light fusion requires alignment of the face, has high data requirements and is difficult to obtain data, the present invention proposes a cross-modal candidate-guided feature mining mechanism to extract corresponding features of the two modalities from unaligned bi-light images for further feature fusion.
[0031] The present invention proposes a cross-modal feature extraction method for unregistered bi-optical images, thereby extracting cross-modal features from unregistered visible light images and thermal infrared images. The present invention proposes a cross-modal candidate-guided feature mining mechanism to extract two precise fusion features, which can extract corresponding facial feature information even when the input bi-optical images are not registered, which enables the algorithm to effectively utilize the complementarity between the two modalities.
[0032] The cross-modal candidate-guided data mining module first detects faces in different modalities based on the corresponding information, and then performs adaptive feature mining on the features in the candidates based on the detected faces as candidates to obtain facial features in a specific modality. In the cross-modal candidate extraction process, this module obtains face candidates in each modality through a regression method based on a convolutional neural network. After that, the data mining module obtains the discriminative information contained in each candidate through a local attention mechanism, and enhances the facial information contained therein through a feature enhancement module based on a convolutional neural network.
[0033] 4. Dual Adversarial Generation Network:
[0034] In order to more fully extract discriminative and unique facial features from the two modal images, the present invention proposes a dual adversarial generative network, which not only extracts consistency information from the visible light-thermal infrared image pair, but also pays attention to the complementary information therein, retaining more features in the dual light image. In the image, the size of the pixel value represents the intensity, and the difference between adjacent pixel values represents the gradient. The smaller the pixel value, the more facial structure information the image contains, and the larger the gradient, the richer the background detail information represented by the image. Visible light images contain rich detail information, while thermal infrared images contain rich structural information. Therefore, the present invention introduces an intensity extraction module and a gradient extraction module to generate true value images of face discriminators and detail discriminators, so that the fused image can contain more facial structure information and background detail information. Figure 3 The architecture of the dual adversarial generative network proposed in the present invention is shown.
[0035] The purpose of the intensity extraction module and the gradient extraction module is to generate a true value image for image identification. The role of the intensity extraction module is to extract pixels from thermal infrared images and visible light images to highlight the face while reducing the loss of facial structure information caused by occlusions such as eyes. Since white pixels have the highest value in the image, most faces in thermal infrared images are white. In order to obtain more facial structure information, a minimum filter is used in the intensity extraction module to extract image structure information. The formula for this process is expressed as Formula 1, where IEB represents the intensity extraction module, Ivi represents the visible light image, I ir Represents a thermal infrared image, and min() represents the minimum value operation.
[0036] IEB(·)=min(I vi , I ir ) Formula 1
[0037] The introduction of the gradient extraction module can make the background contain richer detail information. The greater the difference between adjacent pixels in the image, the greater the gradient, and the richer the detail information contained. In the gradient extraction module, this paper first uses the gradient operation ▽ to calculate the gradients of the thermal infrared image and the visible light image respectively. Since the larger the gradient, the richer the detail information, the gradient extraction module uses the maximum filter to extract the information. The formula of this process is defined as Formula 2, where GEB represents the gradient extraction module and max() represents the maximum value operation.
[0038]
[0039] 5. Mask-guided face detail information discriminator
[0040] In order to preserve more facial detail information in the fused image, a mask-guided facial detail information discriminator is used in the generative adversarial network to identify the richness of facial detail information, guide the model to retain more facial detail information, and improve the accuracy of face recognition.
[0041] The role of the face discriminator in the dual adversarial generative network proposed in the present invention is to make the face in the fused image closer to the face effect in the thermal infrared image. The role of the detail discriminator is to make the background details in the fused image closer to the background in the visible light image. First, in order to distinguish between the face and the background, this paper segments the target and the background through a pre-trained model to determine the scope of the face discriminator and the detail discriminator. In the calculation of the loss function, the algorithm proposed in the present invention distinguishes the foreground and the background through the obtained mask, and calculates the detail loss therein respectively. In this way, more facial detail information is retained in the image fusion, thereby improving the accuracy of the final result in the subsequent face recognition.
[0042] The specific innovations of the present invention are as follows:
[0043] 1. In order to solve the problem that bi-light fusion needs to align the face, has high data requirements and is difficult to obtain data, the present invention proposes a cross-modal candidate-guided feature mining mechanism to extract the corresponding features of the two modalities from the unaligned bi-light images for further feature fusion.
[0044] 2. In order to more fully extract discriminative and unique facial features from images of two modalities, the present invention proposes a dual adversarial generative network, which not only extracts consistency information from visible light-thermal infrared image pairs, but also focuses on the complementary information therein, thereby retaining more features in the dual-light image.
[0045] 3. In order to preserve more facial detail information in the fused image, a mask-guided facial detail information discriminator is used in the generative adversarial network to identify the richness of facial detail information, guide the model to retain more facial detail information, and improve the accuracy of face recognition.
[0046] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.
Claims
1. A face recognition method, characterized in that: The steps include: Image fusion steps: the bi-optical image is input into the data mining module. The data mining module extracts the corresponding features of different modalities from the unregistered bi-optical image through the feature mining mechanism guided by cross-modal candidates to form visible light features and thermal infrared features. The visible light features and thermal infrared features are fused through the feature fusion module. The bi-optical image includes the visible light image and the thermal infrared image. Face recognition step: The fused image is input into the face recognition module for face recognition.
2. The face recognition method according to claim 1, characterized in that: The face recognition module includes a face feature extraction module and a feature matching module. The face feature extraction module is used for extracting face features, and the feature matching module is used for matching face features.
3. The face recognition method according to claim 1, characterized in that: The feature fusion module is a generative adversarial network.
4. The face recognition method according to claim 1, characterized in that: The generative adversarial network is an adversarial game architecture consisting of a generator and a discriminator.
5. The face recognition method according to claim 4, characterized in that: The generator includes an encoder and a decoder. The encoder is composed of a feature extraction module and a feature enhancement module. The feature extraction module is composed of a convolutional neural network module and a Transformer module. The visible light features and thermal infrared features are input into the convolutional neural network module. At the same time, the visible light features and thermal infrared features are divided into multiple image blocks, and the multiple image blocks are input into the Transformer module. The feature enhancement module merges the outputs of the convolutional neural network module and the Transformer module. Finally, the output image will pass through a decoder composed of a convolutional neural network to obtain a fused image.
6. The face recognition method according to claim 1, characterized in that: The data mining module obtains face candidates in each modality through a regression method based on a convolutional neural network. Thereafter, the data mining module obtains the discriminative information contained in each candidate through a local attention mechanism, and enhances the face information contained therein through a feature enhancement module based on a convolutional neural network.
7. The face recognition method according to claim 4, characterized in that: The discriminator includes an intensity extraction module, a gradient extraction module, a face discriminator and a detail discriminator. The intensity extraction module and the gradient extraction module are used to generate a true value image for image identification; the intensity extraction module extracts pixels of thermal infrared images and visible light images to highlight the face while reducing the loss of facial structure information caused by occlusion of the face by obstructions.
8. The face recognition method according to claim 7, characterized in that: The intensity extraction module uses a minimum filter to extract image structure information. The formula is expressed as Formula 1, where IEB represents the intensity extraction module, I vi represents the visible light image, I ir Represents a thermal infrared image, and min() represents the minimum value operation. IEB(·) = min(I vi , I ir ) Formula 1; In the gradient extraction module, the gradient operation ▽ is used to calculate the gradients of the thermal infrared image and the visible light image respectively. The gradient extraction module uses the maximum filter to extract information. The formula is defined as Formula 2, where GEB represents the gradient extraction module and max() represents the maximum value operation; 9. A face recognition system, characterized in that: include: A memory, a processor and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the face recognition method according to any one of claims 1 to 8 when called by the processor.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the face recognition method according to any one of claims 1 to 8 when called by a processor.
