RGB-D face recognition and verification method based on generative heterogeneous feature complementary enhancement

Abstract

The invention discloses an RGB-D face recognition and verification method based on generative heterogeneous feature complementary enhancement, and the method comprises the steps: firstly obtaining a face RGB image and Depth image pair for model training, and carrying out the data enhancement operation; respectively training dimension reduction models of the RGB image and the Depth image, and constraining through an encoder-decoder network and a corresponding loss function; training a Depth-> RGB heterogeneous feature generation model based on a diffusion model by using the dimension reduction feature; generating pseudo RGB features through the heterogeneous feature generation model, and constructing a new training set; and finally, the RGB features, the pseudo RGB features and the Depth features are utilized to train a face feature generation network, feature complementary enhancement is realized by means of an RGB-RGB fusion module and a geometric enhancement module, face features for face recognition and verification are obtained, and recognition and verification are completed by calculating cosine similarity between the features in a test stage. The method gives full play to the heterogeneous feature complementarity of the RGB image and the Depth image, reduces the influence of the illumination and other factors on the recognition result, and improves the face recognition and verification performance.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and biometrics, and in particular to an RGB-D face recognition and verification method based on generative heterogeneous feature complementarity enhancement. Background Technology

[0002] Facial recognition technology, as an important component of the biometrics field, is widely used in security monitoring, financial payments, and smart access control due to its advantages such as non-contact and convenience. Traditional methods are mostly based on RGB images, extracting facial texture, color, and contour features for identity recognition and verification. However, RGB images are easily affected by lighting conditions. In complex lighting environments such as strong light, weak light, backlight, and uneven lighting, facial features may be severely distorted or missing, leading to a significant drop in recognition accuracy.

[0003] To overcome the influence of lighting conditions, researchers introduced depth images. Depth images reflect the three-dimensional geometry of the face, are unaffected by changes in lighting, and exhibit strong stability. By fusing RGB and depth images and utilizing the complementarity of modal features, the robustness of face recognition can be improved to some extent.

[0004] However, existing RGB-D face recognition methods have shortcomings in feature fusion utilization: First, some methods simply splice or weightedly fuse RGB and Depth features without fully exploring the inherent correlation and complementary potential of heterogeneous features, resulting in insufficient discriminative power of the fused features; second, RGB and Depth images belong to different modalities with large differences in feature distribution, and direct fusion can easily cause modal conflicts, affecting the recognition effect; third, in practical applications, the quality of single-modal data may be poor, and relying solely on the original dual-modal feature recognition makes it difficult to guarantee recognition stability and accuracy.

[0005] Due to the shortcomings of existing technologies, there is an urgent need for an effective method to achieve complementary enhancement of heterogeneous features of RGB and Depth, so as to improve the performance of face recognition and verification in complex environments. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide an RGB-D face recognition and verification method based on generative heterogeneous feature complementarity enhancement. By constructing a phased training dataset, designing RGB and Depth image dimensionality reduction models, a heterogeneous feature generation model based on a diffusion model, and an RGB-RGB fusion and geometric enhancement module, high-quality pseudo-RGB features are generated to supplement the insufficient RGB features under the influence of illumination. At the same time, the edge information of RGB features is used to enhance Depth features, realizing effective complementary enhancement of cross-modal features, and significantly improving the accuracy and robustness of face recognition and verification.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: an RGB-D face recognition and verification method based on generative heterogeneous feature complementarity enhancement, comprising the following steps:

[0008] Step S1: Obtain paired data of RGB and Depth images of the face used for model training;

[0009] Step S2: Train the dimensionality reduction model for RGB and Depth images;

[0010] Step S3: Train the Depth to RGB heterogeneous feature generation model using the dimensionality reduction features output by the image dimensionality reduction model;

[0011] Step S4: Generate pseudo-RGB features corresponding to the dimensionality-reduced Depth features using a heterogeneous feature generation model;

[0012] Step S5: Train a face feature generation network for face recognition and verification using the dimensionality-reduced RGB features, pseudo-RGB features, and dimensionality-reduced Depth features.

[0013] In a preferred embodiment, step S1 specifically includes the following steps:

[0014] Step S11: Simultaneously acquire face information using a ToF sensor or structured light camera and an optical camera to generate face depth and RGB images. Select image pairs (RGB, Depth) with good imaging quality from the acquired images to form the first training set, and perform mirror flipping, scaling and data augmentation operations on the images in the first training set.

[0015] In a preferred embodiment, step S2 specifically includes the following steps:

[0016] Step S21: Select a pair of images (RGB, Depth) from the first training set;

[0017] Step S22: Select the RGB image I from the training set image pair (RGB, Depth). RGB Reconstruction is performed using a ResNet-18-based encoder-decoder network. The original input image is encoded by the encoder to obtain intermediate features with reduced dimensions (C, H, W). These intermediate features are then decoded by the decoder into an I-value of the same size as the original input image. RGB The reconstructed image I is constrained by the MSE loss function and the SSIM loss function. RGB 'With the original input image I RGB Similarity is used to obtain intermediate features x that retain key information of the original image. RGB The objective function of the process is defined as follows:

[0018]

[0019] Among them, L MSE Represents the MSE loss function and L SSIM Represents the SSIM loss function;

[0020] Step S23: Select the Depth image I from the training set image pair (RGB, Depth). Depth The image is then reconstructed using a separate ResNet-18-based encoder-decoder network. The original input image is encoded by the encoder to obtain intermediate features with reduced C, H, and W dimensions. These intermediate features are then decoded by the decoder into an I-size image of the same size as the original input image. Depth The reconstructed image I is constrained by the MSE loss function and the SSIM loss function. Depth 'With the original input image I Depth Similarity is used to obtain intermediate features x that retain key information of the original image. Depth The size of the three dimensions C, H, and W of this feature is related to x. RGB The objective function of the process is defined as follows:

[0021]

[0022] Step S24: Repeat steps S21 to S23 until the model converges; after the model converges, remove the decoder part and freeze the parameters of the encoder part, and use it as an image dimensionality reduction model. The intermediate features output by the encoder are the dimensionality reduction features that retain the key information of the original image.

[0023] In a preferred embodiment, step S3 specifically includes the following steps:

[0024] Step S31: Using the RGB and Depth image dimensionality reduction model trained in Step S2, process the image pairs (RGB, Depth) of the first training set to obtain the second training set (x, y) composed of key information features after dimensionality reduction. RGB ,x Depth );

[0025] Step S32: Train the Depth→RGB heterogeneous feature generation model based on the diffusion model; take training set sample pairs (x RGB ,x Depth Randomly sample Gaussian noise ε of the same size as the features in the training set from a discrete uniform distribution {1,2,…,T} for time t; add the noise ε to x. RGB (The diffusion model is denoted as x0) RGB ), to obtain the noisy feature x t RGB ; to x t RGBInput a UNet-based diffusion model, and simultaneously x Depth The time t is used as a condition input to the diffusion model for generating denoised noise; for the noise Pred generated by the diffusion model... Noise The MSE loss function is used to constrain its similarity to the actual added noise ε; the objective function of the training process is as follows:

[0026]

[0027] in, It is a noisy RGB feature; It is a diffusion model based on UNet, with parameters as follows: ; It is the data distribution of the training set, i.e., the sample pairs (x RGB ,x Depth The distribution of t; t from a discrete uniform distribution Mid-sampling, This is the total number of time steps; From the standard multivariate Gaussian distribution Mid-sampling, dimension and x0 RGB same; These are noise scheduling parameters used to control the degree of noise addition;

[0028] Step S33: Repeat step S32 until the model converges; after training converges, obtain the model based on x. Depth Generate pseudo-RGB features x RGB A heterogeneous feature generation model that combines x Depth Randomly sampled Gaussian noise and time step t generate corresponding pseudo-x RGB .

[0029] In a preferred embodiment, step S4 specifically includes the following steps:

[0030] Step S41: Randomly sample multivariate Gaussian noise ε of the same dimension as the features of the training set; randomly sample time t from the discrete uniform distribution {1,2,…,T}; select sample pairs (x) from the second training set. RGB ,x Depth ), x Depth Inputting ε and t into a heterogeneous feature generation model generates corresponding pseudo-x. RGB Features; after repeating the operation multiple times, the resulting sample pairs (x RGB ,x Depth ,pseudo x RGB The set of (x) is used as the third training set; due to pseudox RGB By generating random Gaussian noise ε and repeating the process on the same sample pairs in the second training set, a pseudo-x can be obtained. RGB Diverse third training set sample pairs (xRGB ,x Depth ,pseudo x RGB ).

[0031] In a preferred embodiment, step S5 specifically includes the following steps:

[0032] Step S51: Select the sample pairs (x) from the third training set obtained in step S4. RGB ,x Depth ,pseudo x RGB ); to x RGB and pseudo-x RGB The input RGB-RGB fusion module enhances the original RGB features through pseudo-feature complementarity, resulting in enhanced features x that more accurately reflect facial structure. RGB ';

[0033] Step S52: For x RGB Edge features are extracted using the Canny algorithm, denoted as x. e RGB ; to x e RGB and x Depth The input geometry enhancement module utilizes the edge information extracted from RGB features and combines it with the structural characteristics of the Depth feature to complementaryly enhance the edge representation of the Depth feature, resulting in the enhanced edge feature x. Depth ';

[0034] Step S53: Concatenate x in the channel dimension RGB 'and x Depth The core feature Z of the face recognition and verification task is formed by using a fully connected layer in conjunction with the Softmax activation function to calculate the cross-entropy loss of the face recognition task and quantify the feature learning effect.

[0035] Step S54: Repeat steps S51 to S53 until the model converges; during the testing phase, calculate the cosine similarity of the features Z corresponding to different face image sample pairs (RGB, Depth) to complete face recognition and verification.

[0036] In a preferred embodiment, the specific operation of the RGB-RGB fusion module in step S51 is as follows: The pseudo-x... RGB and x RGB Features are concatenated along the channel dimension; a channel attention mechanism is introduced to perform channel weighting on the concatenated features, highlighting discriminative feature channels; then x is recovered through a 1×1 convolution. RGB The original channel dimension makes the output x RGB 'It integrates the advantages of pseudo-feature spectra, containing richer and more accurate spectral information.'

[0037] In a preferred embodiment, the specific operation of the geometry enhancement module in step S52 is as follows:

[0038] S521: Spatial Dissimilarity Weight Calculation: Calculate the Depth feature x Depth RGB edge features x e RGB Perform channel max pooling on each channel, and subtract the pooling results to obtain the spatial difference weight D. diff The spatial difference weight reflects the degree of complementarity between the two modes in the spatial dimension, i.e., x. e RGB With x Depth各自优势区域 ;

[0039] S522: Attention-guided feature complementarity: for x e RGB Apply self-attention to generate query tensor Q and key tensor K; compute Q. Obtain the basic attention similarity, and then... Element-wise multiplication and weighting yields geometric attention weights;

[0040] S523: Feature Enhancement and Fusion: Extraction middle Weaker than Spatial location, and Multiplication, reinforcement By extracting advantageous information and weakening its inferior information, we obtain Depth-enhanced features.

[0041] By concatenating "selective RGB features" and "Depth enhancement features," and compressing the dimensionality using a channel attention mechanism, the final output is the enhanced version. ;

[0042] S524: Formula for Geometric Attention Guidance Mechanism:

[0043] The core mechanism can be formally represented as follows:

[0044]

[0045] in, For x e RGB Stronger than x Depth Spatial difference weights.

[0046] The present invention also provides a computer device, comprising:

[0047] The processor, memory, and bus, wherein the memory stores machine-readable instructions executed by the processor;

[0048] When the computer device is running, the processor communicates with the memory via a bus, and when the machine-readable instructions are executed by the processor, the RGB-D face recognition and verification method based on generative heterogeneous feature complementarity enhancement is performed as described.

[0049] The present invention also provides a computer-readable storage medium, comprising:

[0050] The computer-readable storage medium contains a computer program;

[0051] The computer program is executed by the processor as described in the RGB-D face recognition and verification method based on generative heterogeneous feature complementarity enhancement.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] First, it effectively solves the problem of illumination interference. By using a diffusion-based Depth→RGB heterogeneous feature generation module, pseudo-RGB features are generated using Depth features that are unaffected by illumination. This compensates for the feature deviations caused by illumination changes in the original RGB image, significantly improving the recognition stability in complex lighting scenarios.

[0054] Secondly, cross-modal feature depth complementarity is achieved. Through the RGB-RGB fusion module and the geometry enhancement module, the complementarity of spectral features and geometric features is enhanced respectively. This preserves the texture details of RGB while utilizing the 3D structural information of Depth, thus solving the problem of insufficient modal feature fusion in traditional methods.

[0055] Finally, we improved the model's generalization ability and data utilization. By generating diverse pseudo-RGB features to expand the training set and combining them with the key information preserved by the dimensionality reduction model, we enhanced data diversity while reducing redundant computation. This enabled the model to maintain high recognition accuracy in different datasets and complex scenarios, and its robustness was significantly better than existing methods. Attached Figure Description

[0056] Figure 1 A flowchart (I) of a preferred embodiment of the present invention;

[0057] Figure 2 The flowchart (II) is a preferred embodiment of the present invention. Detailed Implementation

[0058] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0059] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0060] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application; as used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise; furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0061] like Figure 1 , 2 As shown, this embodiment provides an RGB-D face recognition and verification method based on generative heterogeneous feature complementarity enhancement, including the following steps:

[0062] Step S1: Obtain the face RGB image and depth image pair for model training.

[0063] In this invention, step S1 specifically includes the following steps:

[0064] Step S11: Simultaneously acquire facial information using a ToF sensor or a structured light camera and an optical camera to generate a depth image and an RGB image of the face. Select the image pair (RGB, Depth) with better imaging quality to form training set 1, and perform data augmentation operations such as mirror flipping and scaling on the images.

[0065] Step S2: Train the dimensionality reduction model for RGB and Depth images

[0066] In this invention, step S2 specifically includes the following steps:

[0067] Step S21: Select a pair of images (RGB, Depth) from training set 1.

[0068] Step S22: Select the RGB image I from the training set image pair (RGB, Depth). RGB Reconstruction is performed using a ResNet-18-based encoder-decoder network. The original input image is encoded by the encoder to obtain intermediate features with reduced dimensions (C, H, W). These intermediate features are then decoded by the decoder into an I-value of the same size as the original input image. RGB The reconstructed image I is constrained by the MSE loss function and the SSIM loss function. RGB 'With the original input image I RGB Similarity is used to obtain intermediate features x that retain key information from the original image. RGB The objective function for this process is defined as follows:

[0069]

[0070] Step S23: Select the Depth image I from the training set image pair (RGB, Depth). Depth The image is then reconstructed using a separate ResNet-18-based encoder-decoder network. The original input image is encoded by the encoder to obtain intermediate features with reduced dimensions (C, H, W). These intermediate features are then decoded by the decoder into an I-value of the same size as the original input image. Depth The reconstructed image I is constrained by the MSE loss function and the SSIM loss function. Depth 'With the original input image I Depth Similarity is used to obtain intermediate features x that retain key information from the original image. Depth The size of the three dimensions C, H, and W of this feature is related to x. RGB The same. The objective function for this process is defined as follows:

[0071]

[0072] Step S23: Repeat steps S21 to S23 until the model converges. After the model converges, remove the decoder part and freeze the parameters of the encoder part, using it as an image dimensionality reduction model. The intermediate features output by the encoder are the dimensionality reduction features that retain the key information of the original image.

[0073] Step S3: Train the Depth→RGB heterogeneous feature generation model using the dimensionality reduction features output by the image dimensionality reduction model.

[0074] In this invention, step S3 specifically includes the following steps:

[0075] Step S31: Using the RGB and Depth image dimensionality reduction model trained in Step S2, process the image pairs (RGB, Depth) of training set 1 to obtain a new training set 2 (x) consisting of dimensionality-reduced features containing key information. RGB ,x Depth ).

[0076] Step S32: Train the Depth→RGB heterogeneous feature generation model based on the diffusion model. Take training set sample pairs (x RGB ,x Depth Randomly sample Gaussian noise ε of the same size as the features in the training set from a discrete uniform distribution {1,2,…,T} for time t. Add the noise ε to x. RGB (In the diffusion model perspective, it is also called x0) RGB In ), we get x t RGB x t RGBInput a UNet-based diffusion model, and simultaneously x Depth The time t is used as a condition input to the diffusion model for generating denoised noise. For the noise Pred generated by the diffusion model... Noise The MSE loss function is used to constrain its similarity to the actual added noise ε. The objective function of the training process is as follows:

[0077]

[0078] in, It is a noisy RGB feature; It is a diffusion model based on UNet, with parameters as follows: ; It is the data distribution of the training set, i.e., the sample pairs (x RGB ,x Depth The distribution of t; t from a discrete uniform distribution Medium sampling ( (This is the total number of time steps). From the standard multivariate Gaussian distribution Mid-sampling, dimension and x0 RGB same; These are noise scheduling parameters (predefined, no learning required) used to control the degree of noise addition.

[0079] Step S33: Repeat step S32 until the model converges. After training converges, a model based on x is obtained. Depth Generate pseudo-RGB features x RGB The heterogeneous feature generation model can be based on x Depth And randomly sampled Gaussian noise, t generates the corresponding pseudo x RGB Since the acquisition of depth images is not affected by lighting conditions, x Depth The generated pseudo x RGB Having compared to x RGB Better detail rendering avoids facial feature distortion caused by lighting in RGB images. Meanwhile, due to x Depth It reflects 3D structural information, and the pseudo-x it generates RGB Features can exhibit diverse spectral characteristics, enabling x to be expressed without deviating from the 3D structure. RGB Provide useful supplementary information.

[0080] Step S4: Generate pseudo-RGB features corresponding to the Depth features using a heterogeneous feature generation model.

[0081] In this invention, step S4 specifically includes the following steps:

[0082] Step S41: Randomly sample multivariate Gaussian noise ε with the same dimension as the features in the training set, and randomly sample time t from the discrete uniform distribution {1,2,…,T}. Select sample pairs (x,y) from training set 2. RGB ,x Depth ), x Depth The heterogeneous feature generation model, with inputs of ε and t, generates corresponding pseudo-x. RGB Features. After repeating this step multiple times, the resulting sample (x) RGB ,x Depth ,pseudo x RGB The set of (x) is used as training set 3. Because of the pseudo-x RGB The generation of varies depending on the random sampling Gaussian noise ε; therefore, this step applies to sample pairs (x, y) in the same training set 2. RGB ,x Depth This process will be repeated to generate only pseudo-x. RGB Different training sets of 3 samples (x RGB ,x Depth ,pseudo x RGB This enhances the diversity of the dataset.

[0083] Step S5: Train a face feature generation network for face recognition and verification using RGB features, pseudo-RGB features, and depth features.

[0084] In this invention, step S5 specifically includes the following steps:

[0085] Step S51: Select samples (x) from training set 3 obtained in step S4. RGB ,x Depth ,pseudo x RGB x RGB and pseudo-x RGB Input RGB-RGB fusion module, utilize pseudo-x that contains accurate 3D features and is unaffected by lighting. RGB To complement and strengthen x RGB Features, yielding x that more accurately reflects facial structural features. RGB '.

[0086] Step S52: Place x RGB Edge extraction is performed using the Canny algorithm, and the extracted edge features x e RGB and x Depth Input together into the geometry enhancement module, utilizing higher-precision RGB features x RGB Extracted edge information to complement and enhance x Depth The edge features are used to obtain x. Depth '.

[0087] Step S53: Place x RGB 'and xDepth The face features Z, which are concatenated along the channel dimension, are used for face recognition and verification tasks. The face recognition cross-entropy loss is then calculated using a fully connected layer and a Softmax activation function.

[0088] Step S54: Repeat steps S51-53 until the model converges. During the testing phase, recognition and verification are achieved by calculating the cosine similarity between facial features Z of different face image sample pairs (RGB, Depth).

[0089] The specific operation of the RGB-RGB fusion module in step S51 is as follows: Figure 2 As shown, pseudo x RGB and x RGB Features are concatenated along the channel dimension, and a channel attention mechanism is used to weight the channels, selecting the feature channels with the most discriminative information. Then, a 1×1 convolution operation is performed on the channel-weighted concatenated features to recover x. RGB The original channel dimension. The resulting x. RGB It contains richer and more accurate spectral features.

[0090] The specific operation of the geometry enhancement module in step S52 is as follows: Figure 2 As shown, the Depth feature x Depth and the Canny operator from x RGB Extracted RGB edge features x e RGB Perform channel max pooling operations on each mode separately, and then subtract the pooling results to obtain the spatial difference weights between each mode and another mode. These differences reflect the degree of complementarity of the other mode to each mode in the spatial dimension. For x... e RGB To perform self-attention, first calculate the Q and K tensors of the attention (corresponding to partialSelf-Att in the diagram), multiply Q and K, and then multiply them by the spatial dissimilarity weights (x). e RGB Stronger than x Depth The spatial location is obtained by multiplying and weighting the elements one by one to get x. Depth The geometric information guides the attention similarity weights. These weights are then compared with the V tensor of the attention mechanism, i.e., x. e RGB Multiplying them together gives us the result used to complement x. Depth Selectivity x on e RGB Characteristics. Next, first reflect x Depth For x e RGB Complementarity (x e RGB Weaker than x DepthSpatial difference weights of spatial location and x e RGB Multiplication, enhancing x Depth Stronger information weakens weaker information, and then it is combined with selective x. e RGB Feature addition, spatial concatenation, and channel attention mechanism to compress channel dimensions yield the enhanced x. Depth The geometrically guided attention mechanism in this step is formulated as follows:

[0091]

[0092] in, It reflects x e RGB Stronger than x Depth Spatial difference weights of spatial location.

[0093] In a preferred embodiment of the present invention, the RGB-D face recognition and verification method based on generative heterogeneous feature complementarity enhancement, in step S1, image pair acquisition employs a ToF sensor and a high-definition optical camera for simultaneous image capture. Simultaneously, the depth image undergoes hole filling and edge smoothing to eliminate noise generated during acquisition. When selecting image pairs with better imaging quality, sharpness is evaluated by calculating the Laplacian variance, retaining image pairs with a Laplacian variance greater than 80 (for 1080P images). Data augmentation operations include ±15-degree rotation, 0.8-1.2x scaling, random brightness adjustment (±20%), and horizontal flipping.

[0094] An RGB-D face recognition system based on generative heterogeneous feature complementarity enhancement includes:

[0095] Acquisition module: Simultaneously acquires face depth images (640×480 resolution) and RGB images (1920×1080 resolution) using a ToF sensor and an optical camera, constructs an initial image dataset, and performs the aforementioned data augmentation operations.

[0096] Dimensionality Reduction Module: Deploys two ResNet-18-based encoder-decoder networks to perform dimensionality reduction on the RGB and Depth images respectively, outputting mid-level feature maps that preserve spatial dimensions: x RGB (H×W×128) and x Depth (H×W×128). The network is trained using the MSE loss function and the SSIM loss function, enabling the decoder to reconstruct an image similar to the original image. The encoder with frozen parameters is used as a dimensionality reduction model after training.

[0097] Generation module: Equipped with a UNet-based conditional diffusion model, using x Depth The feature map (middle-layer features) is used as conditional input to generate a function with x. RGB Same-dimensional pseudo-RGB feature maps. Noise scheduling parameters for the diffusion model. It decreases as t increases.

[0098] Feature fusion enhancement module: includes RGB-RGB fusion submodule and geometric enhancement submodule. The former fuses RGB features and pseudo-RGB features through channel attention mechanism, while the latter enhances Depth features by using Canny edge features and attention mechanism. Finally, the two enhanced features are spliced ​​together to form a fused feature.

[0099] Recognition and verification module: It takes fused features as input, trains face recognition through fully connected layers and the Softmax function, and realizes face verification by calculating the cosine similarity between features during the testing phase.

[0100] Figure 1 , 2 This is a schematic diagram of the method flow of the present invention, illustrating the complete process from face image acquisition, feature dimensionality reduction generation to fusion recognition and verification, including: RGB and Depth image dimensionality reduction model, Depth→RGB heterogeneous feature generation module based on diffusion model, RGB-RGB fusion module, geometric enhancement module and face feature generation network.

[0101] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the present invention should still fall within the scope of the present invention.

Claims

1. A method for RGB-D face recognition and verification based on generative heterogeneous feature complementarity enhancement, characterized in that, Includes the following steps: Step S1: Obtain paired data of RGB and Depth images of the face used for model training; Step S2: Train the dimensionality reduction model for RGB and Depth images; Step S3: Train the Depth to RGB heterogeneous feature generation model using the dimensionality reduction features output by the image dimensionality reduction model; Step S4: Generate pseudo-RGB features corresponding to the dimensionality-reduced Depth features using a heterogeneous feature generation model; Step S5: Train a face feature generation network for face recognition and verification using the dimensionality-reduced RGB features, pseudo-RGB features, and dimensionality-reduced Depth features.

2. The RGB-D face recognition and verification method based on generative heterogeneous feature complementarity enhancement according to claim 1, characterized in that, Step S1 specifically includes the following steps: Step S11: Simultaneously acquire face information using a ToF sensor or structured light camera and an optical camera to generate face depth and RGB images. Select image pairs (RGB, Depth) with good imaging quality from the acquired images to form the first training set, and perform mirror flipping, scaling and data augmentation operations on the images in the first training set.

3. The RGB-D face recognition and verification method based on generative heterogeneous feature complementarity enhancement according to claim 1, characterized in that, Step S2 specifically includes the following steps: Step S21: Select a pair of images (RGB, Depth) from the first training set; Step S22: Select the RGB image I from the training set image pair (RGB, Depth). RGB Reconstruction is performed using a ResNet-18-based encoder-decoder network. The original input image is encoded by the encoder to obtain intermediate features with reduced dimensions (C, H, W). These intermediate features are then decoded by the decoder into an I-value of the same size as the original input image. RGB The reconstructed image I is constrained by the MSE loss function and the SSIM loss function. RGB 'With the original input image I RGB Similarity is used to obtain intermediate features x that retain key information of the original image. RGB The objective function of the process is defined as follows: Among them, L MSE Represents the MSE loss function and L SSIM Represents the SSIM loss function; Step S23: Select the Depth image I from the training set image pair (RGB, Depth). Depth The image is then reconstructed using a separate ResNet-18-based encoder-decoder network. The original input image is encoded by the encoder to obtain intermediate features with reduced C, H, and W dimensions. These intermediate features are then decoded by the decoder into an I-size image of the same size as the original input image. Depth The reconstructed image I is constrained by the MSE loss function and the SSIM loss function. Depth 'With the original input image I Depth Similarity is used to obtain intermediate features x that retain key information of the original image. Depth The size of the three dimensions C, H, and W of this feature is related to x. RGB The objective function of the process is defined as follows: Step S24: Repeat steps S21 to S23 until the model converges; after the model converges, remove the decoder part and freeze the parameters of the encoder part, and use it as an image dimensionality reduction model. The intermediate features output by the encoder are the dimensionality reduction features that retain the key information of the original image.

4. The RGB-D face recognition and verification method based on generative heterogeneous feature complementarity enhancement according to claim 1, characterized in that, Step S3 specifically includes the following steps: Step S31: Using the RGB and Depth image dimensionality reduction model trained in Step S2, process the image pairs (RGB, Depth) of the first training set to obtain the second training set (x, y) composed of key information features after dimensionality reduction. RGB ,x Depth ); Step S32: Train the Depth→RGB heterogeneous feature generation model based on the diffusion model; take training set sample pairs (x RGB ,x Depth Randomly sample Gaussian noise ε of the same size as the features in the training set from a discrete uniform distribution {1,2,…,T} for time t; add the noise ε to x. RGB (The diffusion model is denoted as x0) RGB ), to obtain the noisy feature x t RGB ; will x t RGB Input a UNet-based diffusion model, and simultaneously x Depth The time t is used as a condition input to the diffusion model for generating denoised noise; for the noise Pred generated by the diffusion model... Noise The MSE loss function is used to constrain its similarity to the actual added noise ε; the objective function of the training process is as follows: in, It is a noisy RGB feature; It is a diffusion model based on UNet, with parameters as follows: ; It is the data distribution of the training set, i.e., the sample pairs (x RGB ,x Depth The distribution of t; t from a discrete uniform distribution Mid-sampling, This is the total number of time steps; From the standard multivariate Gaussian distribution Mid-sampling, dimension and x0 RGB same; These are noise scheduling parameters used to control the degree of noise addition; Step S33: Repeat step S32 until the model converges; after training converges, obtain the model based on x. Depth Generate pseudo-RGB features x RGB A heterogeneous feature generation model that combines x Depth Randomly sampled Gaussian noise and time step t generate corresponding pseudo-x RGB .

5. The RGB-D face recognition and verification method based on generative heterogeneous feature complementarity enhancement according to claim 1, characterized in that, Step S4 specifically includes the following steps: Step S41: Randomly sample multivariate Gaussian noise ε of the same dimension as the features of the training set; randomly sample time t from the discrete uniform distribution {1,2,…,T}; select sample pairs (x) from the second training set. RGB ,x Depth ), x Depth Inputting ε and t into a heterogeneous feature generation model generates corresponding pseudo-x. RGB Features; after repeating the operation multiple times, the resulting sample pairs (x RGB ,x Depth ,pseudo x RGB The set of (x) is used as the third training set; due to pseudox RGB By generating random Gaussian noise ε and repeating the process on the same sample pairs in the second training set, a pseudo-x can be obtained. RGB Diverse third training set sample pairs (x RGB ,x Depth ,pseudo x RGB ).

6. The RGB-D face recognition and verification method based on generative heterogeneous feature complementarity enhancement according to claim 5, characterized in that, Step S5 specifically includes the following steps: Step S51: Select the sample pairs (x) from the third training set obtained in step S4. RGB ,x Depth ,pseudo x RGB ); to x RGB and pseudo-x RGB The input RGB-RGB fusion module enhances the original RGB features through pseudo-feature complementarity, resulting in enhanced features x that more accurately reflect facial structure. RGB '; Step S52: For x RGB Edge features are extracted using the Canny algorithm, denoted as x. e RGB ; will x e RGB and x Depth The input geometry enhancement module utilizes the edge information extracted from RGB features and combines it with the structural characteristics of the Depth feature to complementaryly enhance the edge representation of the Depth feature, resulting in the enhanced edge feature x. Depth '; Step S53: Concatenate x in the channel dimension RGB 'and x Depth The core feature Z of the face recognition and verification task is formed by using a fully connected layer in conjunction with the Softmax activation function to calculate the cross-entropy loss of the face recognition task and quantify the feature learning effect. Step S54: Repeat steps S51 to S53 until the model converges; during the testing phase, calculate the cosine similarity of the features Z corresponding to different face image sample pairs (RGB, Depth) to complete face recognition and verification.

7. The RGB-D face recognition and verification method based on generative heterogeneous feature complementarity enhancement according to claim 6, characterized in that, The specific operation of the RGB-RGB fusion module in step S51 is as follows: The pseudo-x... RGB and x RGB Features are concatenated along the channel dimension; a channel attention mechanism is introduced to perform channel weighting on the concatenated features, highlighting the discriminative feature channels; Then recover x through 1×1 convolution RGB The original channel dimension makes the output x RGB 'It integrates the advantages of pseudo-feature spectra, containing richer and more accurate spectral information.' 8. The RGB-D face recognition and verification method based on generative heterogeneous feature complementarity enhancement according to claim 6, characterized in that, The specific operation of the geometry enhancement module in step S52 is as follows: S521: Spatial Dissimilarity Weight Calculation: Calculate the Depth feature x Depth RGB edge features x e RGB Perform channel max pooling on each channel, and subtract the pooling results to obtain the spatial difference weight D. diff The spatial difference weight reflects the degree of complementarity between the two modes in the spatial dimension, i.e., x. e RGB With x Depth各自优势区域 ; S522: Attention-guided feature complementarity: for x e RGB Apply self-attention to generate query tensor Q and key tensor K; compute Q. Obtain the basic attention similarity, and then... Element-wise multiplication and weighting yields geometric attention weights; S523: Feature Enhancement and Fusion: Extraction middle Weaker than Spatial location, and Multiplication, reinforcement By extracting advantageous information and weakening its inferior information, we obtain Depth-enhanced features. By concatenating "selective RGB features" and "Depth enhancement features," and then compressing the dimensionality using a channel attention mechanism, the final output is the enhanced version. ; S524: Formula for Geometric Guided Attention Mechanism: The core mechanism can be formally represented as follows: in, For x e RGB Stronger than x Depth Spatial difference weights.

9. A computer device, characterized in that, include: The processor, memory, and bus, wherein the memory stores machine-readable instructions executed by the processor; When the computer device is running, the processor communicates with the memory via a bus, and when the machine-readable instructions are executed by the processor, the RGB-D face recognition and verification method based on generative heterogeneous feature complementarity enhancement as described in any one of claims 1 to 8 is executed.

10. A computer-readable storage medium, characterized in that, include: The computer-readable storage medium contains a computer program; The computer program is executed by the processor to perform the RGB-D face recognition and verification method based on generative heterogeneous feature complementarity enhancement as described in any one of claims 1 to 8.

Images