High-precision face positioning method and system based on dual-light fusion, and storage medium

Abstract

The invention provides a high-precision face positioning method and system based on dual-light fusion and a storage medium, and the method comprises the steps: a training data obtaining step: collecting data through employing an infrared and visible light dual-light camera, carrying out the manual registration and alignment of a collected dual-light data set, and carrying out the median filtering and denoising of an infrared image; and a training step: taking the obtained image pairs as training data, sending the training data to a dual-light fusion face positioning network together, training by using the Mamba-YOLO network structure, fusing the obtained feature vectors through a DANet feature fusion network, sending the fused feature vectors to a PAFPN pixel attention feature pyramid network structure for feature mapping, and obtaining a face feature map. The mapped image features are sent to a layer of convolution to obtain a final positioning result; and a prediction step: obtaining a face positioning result by using the trained dual-light face positioning network. The method has the beneficial effects that the features can be extracted from the input dual-light image, and after the features are fused, the face position information is obtained from the dual-light image.

Description

Technical Field

[0001] The present invention relates to the field of face recognition technology, and in particular to a high-precision face positioning method, system and storage medium based on dual-light fusion. Background Art

[0002] Face positioning technology is an important application in the field of computer vision, playing a key role in many scenarios such as human-computer interaction, biometrics, and intelligent monitoring. Accurate and reliable face positioning can provide valuable input data for these applications and lay the foundation for the realization of system functions. However, there are certain limitations in face positioning under single visible light conditions. Visible light images are easily affected by changes in ambient light. Under insufficient or strong light, facial features will be blocked or distorted, resulting in a rapid decrease in positioning accuracy and success rate, and are easily interfered by noise. Therefore, the dual-light fusion technology of infrared and visible light can effectively supplement the shortcomings of single visible light face positioning. Infrared imaging can extract the heat distribution of the face and maintain good detection effects even in dark environments or complex backgrounds. Combining the color texture information of visible light with the structural characteristics of infrared can help build a more robust and accurate face positioning algorithm, significantly improving the positioning performance in various complex environments.

[0003] The core goal of the dual-light fusion face positioning method is to achieve high-precision, all-weather face positioning based on the dual-light fusion of infrared and visible light. Through face positioning, the face position information is obtained to prepare for the subsequent feature extraction and classification recognition steps of face recognition.

[0004] The traditional dual-light fusion face localization method is to use traditional target recognition methods to locate faces on the aligned infrared image and visible light image respectively, and then select a more reliable localization result as the final result based on the confidence level.

[0005] When performing target recognition on infrared images and visible light images respectively, the traditional method first uses sliding windows of different sizes and proportions to slide across the entire image at a certain step length, and extracts the feature vectors of each sliding window, such as Haar features, HOG features, etc., and then uses a classifier for recognition, such as the commonly used SVM model.

[0006] Although the traditional method of dual-light face localization is simple to implement, it has many problems. First, the method performs face localization on dual-light images separately, which cannot capture the correlation and complementarity between the two domains in a limited way, which limits the ability of the method to fully utilize dual-light information. Secondly, classifiers such as SVM rely on manually designed features, which often have strong scene dependence and are difficult to generalize to complex practical application scenarios. Finally, the face localization method using sliding windows requires sliding the detection window on the image, extracting features and classifying each window, which has high computational complexity and is difficult to achieve real-time localization.

[0007] Due to the widespread application of deep learning, some methods have begun to use neural networks to achieve face positioning with dual-light fusion. One method is to fuse the features of the dual-light image, and then obtain the positioning result through the target recognition network such as YOLO and R-CNN. Another method is to use the target recognition network to obtain the face positioning information in the dual-light image, and then select the positioning result with the highest confidence as the final output result, or select and combine the ranges defined by the two according to a certain strategy, such as obtaining the intersection and union.

[0008] Compared with the traditional dual-light face positioning method, the dual-light face positioning method based on deep learning can provide higher accuracy and faster speed. However, the current face positioning method still has many problems.

[0009] First, most face positioning methods based on dual-light fusion directly use the data collected by the dual-light camera to train the neural network without any preprocessing operations other than alignment. However, since infrared cameras are prone to noise, direct training will cause the neural network to learn incorrect mapping relationships, thus affecting positioning accuracy.

[0010] Secondly, most of the current face positioning networks use the structure proposed in the YOLO series. However, with the continuous development of the YOLO network, the time required for its neural network reasoning is getting longer and longer, making it difficult to achieve real-time performance.

[0011] Thirdly, when faced with visible light and infrared dual modalities, most methods use simple feature splicing or weighted fusion, which makes it difficult to adaptively adjust the importance of different features. The results obtained are only applicable to one or a few limited scenarios, and the generalization performance is poor.

[0012] Finally, most of the localization methods based on neural networks use pyramid sampling for multi-scale feature mapping. Although this method can use multi-scale features for feature mapping, its structure is relatively simple and it is difficult to learn more complex relationships between features. Summary of the invention

[0013] In order to solve the problems in the prior art, the present invention provides a high-precision face positioning method based on dual-light fusion, which not only improves the positioning accuracy, but also effectively improves the inference speed of the algorithm.

[0014] The present invention provides a high-precision face positioning method based on dual-light fusion, comprising performing the following steps in sequence:

[0015] Training data acquisition steps: Data is collected by using an infrared and visible light dual-light camera, and then the collected dual-light data set is manually aligned, and the infrared image is simply denoised using a median filter for use in subsequent operations;

[0016] Training step: the image pairs obtained in the training data acquisition step are used as training data, and are sent together into the dual-light fusion face positioning network, and the proposed Mamba-YOLO network structure is used for training. The proposed Mamba-YOLO structure first sends the visible light image and the infrared image into the ODMamba Backbone structure for feature extraction, and fuses the obtained feature vectors through the DANet feature fusion network, and then sends them to the PAFPN pixel attention feature pyramid network structure for feature mapping, and finally sends the mapped image features into a layer of convolution to obtain the final positioning result;

[0017] Prediction step: The bi-optical image captured and aligned by the bi-optical camera is passed through the trained bi-optical face localization network to obtain the face localization result.

[0018] As a further improvement of the present invention, in the feature extraction part of the training step, the input image is first preprocessed by the Simple Stem structure, and the processed feature map is then subjected to deep feature extraction by the ODSSBlock unit. Finally, the Vision Club Merge module further fuses and optimizes these features to ensure that the features extracted from different levels can be effectively combined to enhance the positioning capability.

[0019] As a further improvement of the present invention, the ODSSBlock unit includes an LSBlock module, an RGBlock module, and an SS2D module. The calculation steps of the features through the ODSSBlock unit are shown in Formulas 5-1 to 5-3, wherein Φ represents the activation function, Conv represents the convolution layer, BN represents the batch normalization operation, LN, LS, and RG represent the layer normalization operation, the local space block, and the residual gating block, respectively. X1 and X4 are the input and output of the structure, respectively, and X2 and X3 are the intermediate results of the algorithm.

[0020]

[0021] X3=SS2D(LN(LS(X2)))+X2 (5-2)

[0022] X4=RG(LN(X3))+X3 (5-3).

[0023] As a further improvement of the present invention, in the DANet feature fusion network of the training step, the features extracted from the visible light and infrared images are first spliced, then passed through the ResNet network, and then sent to the position attention module and the channel attention module respectively to obtain the fused features, and then the two results are summed to obtain the final result.

[0024] As a further improvement of the present invention, in the feature mapping part of the training step, the fused feature vector is input into the PAFPN network for multi-scale processing. At this stage, the feature map realizes spatial resolution adjustment and refinement of the feature by combining upsampling and downsampling operations, as well as additional convolution processing. Through such a configuration, the model can simultaneously process high-resolution large-scale features and low-resolution small-scale features, and can more accurately identify and locate facial images of different sizes that appear in the picture at the same time.

[0025] The present invention also discloses a high-precision face positioning system based on dual-light fusion, comprising: a memory, a processor, and a computer program stored on the memory, wherein the computer program is configured to implement the steps of the high-precision face positioning method of the present invention when called by the processor.

[0026] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the high-precision face positioning method of the present invention when called by a processor.

[0027] The beneficial effects of the present invention are: the present invention can extract features from the input bi-optical image, and after the feature fusion, obtain the face position information therefrom, specifically:

[0028] (1) In the data preprocessing stage, the infrared image in the aligned bi-optical image is subjected to median filtering and denoising to prevent the neural network from learning the wrong mapping relationship, thereby improving the accuracy of face location.

[0029] (2) In the network design stage, the Mamba network structure is used for feature extraction, so that the model has better long-distance dependency performance and relatively low time complexity;

[0030] (3) During the network design phase, the DANet network structure is used for adaptive feature fusion. Position attention and channel attention modules are added to the network structure to improve the feature fusion performance and obtain better feature fusion effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is the Mamba-YOLO network architecture diagram of the present invention;

[0032] Figure 2 This is the Simple Stem module architecture diagram of the present invention;

[0033] Figure 3 It is the module architecture diagram of ODSSBlock of the present invention;

[0034] Figure 4 It is the architecture diagram of the Vision Club Merge module of the present invention;

[0035] Figure 5 This is a diagram of the DANet feature fusion network architecture of the present invention;

[0036] Figure 6 This is a diagram of the PAFPN feature mapping network architecture of the present invention. DETAILED DESCRIPTION

[0037] Chinese explanation:

[0038] Mamba-YOLO: Mamba-You Only Look Once, Mamba-You Only Look Once;

[0039] ODMamba Backbone: Overlapped Depthwise Mamba Backbone, overlapping depth wise Mamba network backbone structure;

[0040] DANet: Dual Attention Network for Scene Segmentation, dual attention network for scene segmentation;

[0041] PAFPN: Pixel Attention Feature Pyramid Network, pixel attention feature pyramid network;

[0042] Simple Stem: simple interference block;

[0043] ODSSBlock: Overlapped Depthwise Separable Convolution Block, overlapping depth-wise separable convolution block;

[0044] Vision Club Merge: Vision aggregation module;

[0045] LSBlock: Local Spatial Block, local spatial block;

[0046] RGBlock: Residual Gated Block, residual gated block;

[0047] SS2D: Spatial Semantic 2D, 2D spatial semantics;

[0048] ResNet:Residual Network,Residual Network;

[0049] The present invention relates to a high-precision face positioning method based on dual-light feature fusion, and its potential application scenarios include public safety, intelligent security, military and other fields. The application method is to deploy the algorithm and model to the computing device and position the face image in the input "infrared + visible light" dual-channel video stream.

[0050] The present invention discloses a high-precision face positioning method based on dual-light fusion, which specifically comprises the following steps performed in sequence:

[0051] Training data acquisition steps: Data is collected by using an infrared and visible light dual-light camera, and then the collected dual-light data set is manually aligned, and the infrared image is simply denoised using a median filter for use in subsequent operations;

[0052] Training step: the image pairs obtained in the training data acquisition step are used as training data, and are sent together into the dual-light fusion face positioning network, and the proposed Mamba-YOLO network structure is used for training. The proposed Mamba-YOLO structure first sends the visible light image and the infrared image into the ODMamba Backbone structure for feature extraction, and fuses the obtained feature vectors through the DANet feature fusion network, and then sends them to the PAFPN pixel attention feature pyramid network structure for feature mapping, and finally sends the mapped image features into a layer of convolution to obtain the final positioning result;

[0053] Prediction step: The bi-optical image captured and aligned by the bi-optical camera is passed through the trained bi-optical face localization network to obtain the face localization result.

[0054] 1. Median filter denoising operation

[0055] Since thermal infrared images usually have a low resolution, some noise often occurs. Currently, most face localization methods directly feed the aligned bi-optical images into the face localization network without denoising the training data. If there is enough noise in the input image, it may affect the accuracy of the neural network. Therefore, we perform denoising in the data preprocessing stage, which can make the mapping relationship learned by the neural network more accurate, thereby improving the localization effect.

[0056] During image acquisition, the infrared imaging system is affected by environmental factors, detector defects, etc., and impulse noise and speckle noise often occur. These noises are often non-Gaussian. Median filtering is a nonlinear filtering method that has a good suppression effect on impulse noise and speckle noise.

[0057] 2. Feature extraction network structure based on Mamba

[0058] Currently, the introduction of the Transformer structure has enhanced the global modeling capability of the model, but its high computational complexity is still a challenge. The Mamba network structure can solve the long-distance dependency problem by introducing the state space model SSM, while reducing the computational burden through its linear time complexity advantage. In this network, the ODMamba Backbone structure is used for feature extraction, and the ODSSBlock structure from Mamba is added to the PAFPN network in the feature fusion stage to apply this network structure.

[0059] In the proposed Mamba-YOLO structure, the visible light and infrared images are first extracted through the ODMambaBackbone structure, and the obtained features are fused using the DANet network. The fused network is then mapped through the PAFPN, and the mapped features are passed through the convolution layer to obtain the positioning results. The entire network structure is shown in the figure. Figure 1 shown.

[0060] In the feature extraction part, the input image is first preprocessed through the Simple Stem structure. This step includes using convolutional layers with larger strides to reduce the spatial size of the image. This module helps reduce the dimensionality of the input data, thereby reducing the computational burden of subsequent processing, and preliminarily extracts image features. The network structure of the Simple Stem part is as follows: Figure 2 shown.

[0061] Next, the processed feature map is passed through a series of ODSSBlocks for deep feature extraction. These modules use state-space model technology to effectively capture features at different levels. ODSSBlock is the core unit of Mamba-YOLO, which mainly includes three modules: LSBlock, RGBlock, and SS2D. The batch normalization, layer normalization, and residual connection design of ODSSBlock allow the model to flow effectively during deep stacking training. The network structure of the ODSSBlock part is as follows Figure 3 shown.

[0062] The calculation steps of the features through the ODSSBlock module are shown in Formula 5-1 to Formula 5-3. Among them, Φ represents the activation function, Conv represents the convolution layer, BN represents the batch normalization operation, LN, LS, RG are the layer normalization operation, local space block, and residual gating block respectively, X1 and X4 are the input and output of the structure respectively, and X2 and X3 are the intermediate results of the algorithm.

[0063]

[0064] X3=SS2D(LN(LS(X2)))+X2 (5-2)

[0065] X4=RG(LN(X3))+X3 (5-3).

[0066] Finally, the Vision Club Merge module further integrates and optimizes these features to ensure that the features extracted from different levels can be effectively combined to enhance the positioning ability. The network structure of the Vision Club Merge part is as follows: Figure 4 shown.

[0067] 3. Feature fusion network structure based on DANet

[0068] In the proposed Mamba-YOLO architecture, a dual attention network DANet is used to adaptively integrate local features and global dependencies. Two types of attention modules are attached to the traditional dilated FCN to simulate semantic interdependencies in spatial and channel dimensions respectively.

[0069] In this structure, the features extracted from the visible light and infrared images are first concatenated, and then sent to the position attention module and the channel attention module through the ResNet network to obtain the fused features. The two results are then summed to obtain the final result. Figure 5 shown.

[0070] 4. Feature mapping network structure based on PAFPN

[0071] In the feature mapping step, the fused feature vector is input into the PAFPN network for multi-scale processing. At this stage, the feature map adjusts and refines the spatial resolution of the feature by combining upsampling and downsampling operations, as well as additional convolution processing. With this configuration, the model can simultaneously process high-resolution large-scale features and low-resolution small-scale features, and can more accurately identify and locate facial images of different sizes that appear in the picture at the same time. Compared with the feature pyramid commonly used in other face positioning methods, the PAFPN structure adds the ODSSBlock structure, which can fit more complex feature relationships and enable positioning to achieve higher accuracy. Its network structure is as follows: Figure 6 shown.

[0072] The specific innovations of the present invention are as follows:

[0073] (1) In the data preprocessing stage, after aligning the dual-light images, a simple denoising operation is performed on the infrared image using a median filter to avoid the problem that the face localization algorithm regards the noise as valid input and performs incorrect mapping;

[0074] (2) In the network structure design stage, the ODMamba Backbone structure in the Mamba network is used for feature extraction. By introducing the state space model, the computational burden brought by the quadratic complexity of the self-attention mechanism is effectively reduced, and the accuracy of the face localization network is improved;

[0075] (3) In the network structure design stage, the DANet network is used for feature fusion. By adding two types of attention modules on top of the traditional expanded FCN, a more accurate fusion result is obtained;

[0076] (4) In the network structure design stage, the PAFPN pixel attention feature pyramid structure is used for feature mapping. By adding a more complex ODSSBlock module to the feature pyramid, a better mapping effect can be obtained than the general feature pyramid structure.

[0077] The present invention also discloses a high-precision face positioning system based on dual-light fusion, comprising: a memory, a processor, and a computer program stored on the memory, wherein the computer program is configured to implement the steps of the high-precision face positioning method of the present invention when called by the processor.

[0078] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the high-precision face positioning method of the present invention when called by a processor.

[0079] The beneficial effects of the present invention are: the present invention can extract features from the input bi-optical image, and after the feature fusion, obtain the face position information therefrom, specifically:

[0080] (1) In the data preprocessing stage, the infrared image in the aligned bi-optical image is subjected to median filtering and denoising to prevent the neural network from learning the wrong mapping relationship, thereby improving the accuracy of face location.

[0081] (2) In the network design stage, the Mamba network structure is used for feature extraction, so that the model has better long-distance dependency performance and relatively low time complexity;

[0082] (3) During the network design phase, the DANet network structure is used for adaptive feature fusion. Position attention and channel attention modules are added to the network structure to improve the feature fusion performance and obtain better feature fusion effects.

[0083] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A high-precision face positioning method based on dual-light fusion, characterized in that: It includes executing the following steps in sequence: Training data acquisition steps: Data is collected by using an infrared and visible light dual-light camera, and then the collected dual-light data set is manually aligned, and the infrared image is simply denoised using a median filter for use in subsequent operations; Training step: the image pairs obtained in the training data acquisition step are used as training data, and are sent together into the dual-light fusion face positioning network, and the proposed Mamba-YOLO network structure is used for training. The proposed Mamba-YOLO structure first sends the visible light image and the infrared image into the ODMamba Backbone structure for feature extraction, and fuses the obtained feature vectors through the DANet feature fusion network, and then sends them to the PAFPN structure for feature mapping, and finally sends the mapped image features into a layer of convolution to obtain the final positioning format; Prediction step: The bi-optical image captured and aligned by the bi-optical camera is passed through the trained bi-optical face localization network to obtain the face localization result.

2. The high-precision face positioning method according to claim 1, characterized in that: In the feature extraction part of the training step, the input image is first preprocessed by the Simple Stem structure, and the processed feature map is then subjected to deep feature extraction by the ODSSBlock unit. Finally, Vision Club Merge further fuses and optimizes these features to ensure that the features extracted from different levels can be effectively combined to enhance the positioning capability.

3. The high-precision face positioning method according to claim 2, characterized in that: The ODSSBlock unit includes the LSBlock module, the RGBlock module, and the SS2D module. The calculation steps of the features through the ODSSBlock unit are shown in Formula 5-1 to Formula 5-3, where Φ represents the activation function, Conv represents the convolution layer, BN represents the batch normalization operation, LN represents the layer normalization operation, LS represents the local space block, RG represents the residual gating block, X1 and X4 are the input and output of the structure respectively, and X2 and X3 are the intermediate results of the algorithm; X3=SS2D(LN(LS(X2)))H+x2 (5-2) X4=RG(LN(X3))+X3 (5-3).

4. The high-precision face positioning method according to claim 1, characterized in that: In the training step, in the DANet feature fusion network, the features extracted from the visible light and infrared images are first spliced, then passed through the ResNet network, and then sent to the position attention module and the channel attention module respectively to obtain the fused features, and then the two results are summed to obtain the final result.

5. The high-precision face positioning method according to claim 1, characterized in that: In the feature mapping part of the training step, the fused feature vector is input into the PAFPN network for multi-scale processing. At this stage, the feature map realizes spatial resolution adjustment and refinement of the features by combining upsampling and downsampling operations and additional convolution processing.

6. A high-precision face positioning system based on dual-light fusion, characterized in that: include: A memory, a processor and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the high-precision face positioning method according to any one of claims 1 to 5 when called by the processor.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the high-precision face positioning method according to any one of claims 1 to 5 when called by a processor.

Images