Thermal infrared face image texture enhancement method and system for recognition process, and storage medium
Through high-resolution thermal infrared cameras, data is collected and supervised learning is added to the texture enhancement network, combining median filtering and residual networks, the problem of infrared face image texture enhancement in face recognition is solved, and high-quality thermal Infrared face images improve the recognition of features and reduce the difficulty of data acquisition.
Patent Information
- Application Number
- CN202411032472.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-08
AI Technical Summary
The existing infrared image texture enhancement technology is difficult to effectively improve image quality in the process of face recognition. In particular, the infrared face image texture enhancement algorithm for deep learning has training data problems, general scene-oriented problems, and model design is difficult and easy. There are problems with gradient explosion and gradient disappearance.
High-resolution thermal infrared cameras are used to collect data, generate low-resolution images through simulation operations, and supervised learning is added to the texture enhancement network, using median filtering and denoising processing, designing texture enhancement structures based on residual networks, combined with visible light people The loss function of the face recognition network is trained.
A high-resolution thermal infrared face image is generated, which improves the recognition characteristics of the image, retains the original identity information, reduces the difficulty of data acquisition, improves the accuracy of texture enhancement, and avoids gradient problems.
Smart Images

Figure CN118918024A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a thermal infrared face image texture enhancement method, system and storage medium for a recognition process. Background Art
[0002] Infrared images are the product of the combination of infrared technology and imaging technology. They are mainly used in military and security fields. For example, in the field of monitoring, in the absence of light, monitoring equipment equipped with infrared cameras can still obtain environmental information within the monitoring field of view. However, due to the imaging mechanism of infrared images and the infrared imaging system itself, infrared images, compared with visible light images, have the characteristics of low image contrast, blurred images, and low signal-to-noise ratio. This is extremely unfavorable for subsequent feature extraction, recognition or tracking, so it is necessary to suppress noise, improve image signal-to-noise ratio, adjust infrared image contrast, and enhance infrared image edges and lines. Therefore, it is necessary to process the image appropriately, so infrared image enhancement is very necessary.
[0003] The main purpose of infrared image texture enhancement technology is to process a given infrared image so that its result is more suitable for a specific application than the original image. Through enhancement, the quality of infrared images can be improved and preparation can be provided for subsequent operations on infrared images.
[0004] Traditional infrared image texture enhancement technology mainly refers to the use of traditional image processing methods such as grayscale transformation, linear and nonlinear transformation, image smoothing filtering, etc., to process the original blurred or even indistinguishable original image into a clear usable image rich in useful information, effectively remove the noise in the image, and enhance the edges or other areas of interest in the image. Among them, the most commonly used infrared texture enhancement algorithm is the DDE (Digital Detail Enhancement) algorithm. When the DDE algorithm performs image processing, a special filter is first used to separate the background layer (low-frequency part) and the detail layer (high-frequency part) of the image, and then the extracted background layer and detail layer are respectively subjected to corresponding grayscale enhancement and noise suppression processing, and then the dynamic range of each layer is adjusted and compressed, and finally an 8-bit output image is synthesized. In this way, the large dynamic temperature difference in the 14-bit image and the local detail information of the target can be displayed within the 8-bit range.
[0005] Due to the widespread application of deep learning, some methods have begun to use neural networks to achieve infrared image enhancement. One method uses a fully convolutional network. This method generally extracts the feature information of the infrared image through a fully convolutional network, performs feature fusion through skip connections, enriches the detail information, and achieves an enhanced effect. Another method uses a generative adversarial network, which uses a real visible light grayscale image as a condition, relies on the recognition subnetwork D to supervise the generation subnetwork G, and achieves infrared image enhancement through the mutual game between the generation subnetwork and the recognition subnetwork.
[0006] Currently, there is no deep learning-based infrared face image texture enhancement algorithm for the face recognition process.
[0007] Compared with traditional infrared image texture enhancement methods, infrared image texture enhancement methods based on deep learning can provide higher accuracy and wider application scenarios. However, the current infrared image texture enhancement technology still has many problems.
[0008] First of all, in the general training process of deep learning-based infrared texture enhancement methods, pairs of low-resolution and high-resolution images are required as training sets, and the perspective, content, etc. between the image pairs are required to be completely consistent. Such a large number of strictly restricted image pairs are obviously difficult to obtain by shooting separately with two infrared cameras.
[0009] Secondly, most infrared texture enhancement methods based on deep learning are oriented to general scenes. When these enhancement methods are used on face images, they cannot achieve good texture enhancement effects.
[0010] Finally, it is very difficult to develop thermal infrared facial texture enhancement technology for the face recognition process. In the model design process, it is difficult to use the face recognition results to effectively supervise the texture enhancement process; and in the model training process, it is easy to cause training failure due to problems such as gradient explosion and gradient disappearance. Summary of the invention
[0011] The present invention provides a thermal infrared face image texture enhancement method for recognition process, comprising the following steps:
[0012] Step 1, training data acquisition step: obtain a thermal infrared image by using a high-resolution thermal infrared camera to collect data; then, through simulation operation, the grayscale value of the high-resolution thermal infrared image is stepped according to the temperature resolution calibrated by the infrared camera to obtain its corresponding low-resolution thermal infrared image;
[0013] Step 2, training step: the image pairs obtained in the training data acquisition step are used as training data and sent to the texture enhancement network for training. During the training process, the output result of the texture enhancement network is sent to the visible light face recognition network again, and the weighted sum of the last inferred classification cross entropy loss and the output result of the texture enhancement network and the L2 loss of the high-resolution thermal infrared image is used as the loss function of the texture enhancement network to complete the supervision of the texture enhancement network;
[0014] Step 3, prediction step: using only the face image taken by the low temperature resolution infrared camera, obtain its corresponding texture enhanced image through the texture enhancement network.
[0015] As a further improvement of the present invention, in step 1, the thermal infrared image obtained by the high-resolution thermal infrared camera is denoised to obtain a clearer thermal infrared image.
[0016] As a further improvement of the present invention, in step 1, the thermal infrared image obtained by the high-resolution thermal infrared camera is denoised by a median filtering method to obtain a clearer thermal infrared image.
[0017] The present invention also provides a thermal infrared facial image texture enhancement system for a recognition process, comprising: a memory, a processor, and a computer program stored on the memory, wherein the computer program is configured to implement the steps of the thermal infrared facial image texture enhancement method of the present invention when called by the processor.
[0018] The present invention also provides a computer-readable storage medium, which stores a computer program. The computer program is configured to implement the steps of the thermal infrared facial image texture enhancement method of the present invention when called by a processor.
[0019] The beneficial effects of the present invention are as follows: the present invention can generate a high-resolution thermal infrared face image from a low-resolution thermal infrared face image through a texture enhancement algorithm, and can ensure that its features are enhanced in a direction that is conducive to recognition, and the original identity information is retained. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 This is a framework diagram of the texture enhancement task;
[0021] Figure 2 It is a diagram of the texture enhancement network structure. DETAILED DESCRIPTION
[0022] In order to solve the problem in the background technology, the present invention provides an end-to-end thermal infrared face image texture enhancement method for the recognition process, which improves the performance of the thermal infrared face image texture enhancement algorithm.
[0023] The present invention discloses a thermal infrared face image texture enhancement method for recognition process, which specifically comprises the following steps:
[0024] Step 1, training data acquisition step: by using a high-resolution uncooled infrared lens to collect data, and then denoising it by median filtering, a clearer thermal infrared image is obtained. Then, through simulation operations, the grayscale value of the high-resolution thermal infrared image is stepped according to the temperature resolution calibrated by the infrared camera to obtain its corresponding low-resolution thermal infrared image.
[0025] Step 2, training step: the image pairs obtained in the training data acquisition step are used as training data and sent to the texture enhancement network for training. During the training process, the output of the texture enhancement network is sent to the visible light face recognition network again, and the weighted sum of the final inferred classification cross entropy loss and the output of the texture enhancement network and the L2 loss of the high-resolution thermal infrared image is used as the loss function of the texture enhancement network to complete the supervision of the texture enhancement network; at the same time, a specific residual network structure is proposed to achieve high stability training.
[0026] Step 3, prediction step: Use only the face image taken by the low-temperature resolution infrared camera to obtain its corresponding texture-enhanced image through our texture enhancement network.
[0027] 1. The process of obtaining low-resolution thermal infrared images through simulation:
[0028] In texture enhancement algorithms based on deep learning, a large number of image pairs before and after degradation are usually required to learn the mapping relationship between them. In addition, the perspective and content between the image pairs are required to be completely consistent, with only the image quality being different. Such a large number of image pairs with strict restrictions are obviously difficult to obtain through actual shooting.
[0029] Thermal infrared images are different from visible light images. The brightness of the images reflects the surface temperature of the object being photographed. The brighter the image, the higher the temperature, and the darker the image, the lower the temperature. Therefore, the texture in the thermal infrared face image reflects the distribution of the surface temperature of the face being photographed. The temperature distribution of each part of the face is slightly different, and its distribution image should be rich in texture and other feature information. However, in the actual acquisition process, due to the insufficient temperature resolution of ordinary thermal infrared cameras, it is difficult to capture these subtle temperature distribution differences, resulting in the collected face images containing a large number of flat areas, that is, texture loss. In other words, the grayscale values obtained by high-resolution infrared cameras and ordinary infrared cameras at the same temperature are different.
[0030] The temperature resolution of thermal infrared cameras is calibrated before leaving the factory. Therefore, you only need to align the lowest and highest temperatures of two thermal infrared images and perform step-by-step processing on the grayscale values of the high-resolution thermal infrared images according to their calibrated temperature resolutions according to the formula to generate a large number of corresponding image pairs.
[0031] 2. Median filter denoising processing:
[0032] When collecting thermal infrared images, there is usually some noise. In most texture enhancement algorithms based on deep learning, the training data is not denoised. If there is enough noise in the image, the neural network may learn the wrong mapping relationship. Therefore, we perform denoising during data preprocessing to make the mapping relationship learned by the neural network more accurate.
[0033] During the image acquisition process, due to the thermal motion of the infrared sensor itself, the acquired thermal infrared images usually have thermal noise showing random brightness changes. For this kind of noise, using median filtering can greatly smooth the image and reduce the impact of noise.
[0034] 3. Loss function for the recognition process:
[0035] Texture enhancement of thermal infrared face images, that is, the estimation and solution of high temperature resolution images, is not arbitrary and must be effectively supervised. First, texture enhancement must be in a direction that is conducive to subsequent face recognition. The result of the enhancement needs to be optimized to improve the accuracy of face recognition. The impact of different enhancement results on the final recognition success rate must be fully analyzed to ensure that the enhanced texture is required for the face recognition task. Secondly, in order to prevent the loss of feature information in the original thermal infrared face image during the enhancement process, that is, the identity information cannot be maintained, it is necessary to constrain the changes in identity information that may occur before and after the enhancement. In order to achieve this form of supervision, we implemented a loss function to effectively supervise the model.
[0036] The so-called method of achieving texture enhancement is to find a method framework that inputs ordinary low-temperature resolution thermal infrared face images and eventually outputs high-temperature resolution thermal infrared images rich in texture. Naturally, we think of using the high-temperature resolution thermal infrared face images in the laboratory as a reference to make the final high-temperature resolution image closer to it. From a formal point of view, this is actually very close to the restoration task of degraded images, so we first use the L2 loss function commonly used in image restoration tasks, as shown in Formula 5-1. Where h and w are the height and width of the image, respectively, x is a low-resolution thermal infrared image generated by simulation, Y is a high-resolution thermal infrared image, f is a texture enhancement network, and θ is a network parameter.
[0037]
[0038] In addition to selecting the L2 loss of the restored image and the target image as the basic loss function for training during the training process, it is also necessary to impose supervision based on the recognition success rate on the results of texture enhancement to ensure that during the texture enhancement process, its features are improved in a direction that is conducive to recognition and the original identity information is retained. By using a thermal infrared or even visible light face recognition network that has been pre-trained with low-resolution images, the supervision of the texture enhancement network can be completed based on its final inferred classification cross entropy loss. Here, we use the vgg network as the visible light face recognition network, where the classification cross entropy loss is shown in formulas 5-2 and 5-3. Where k is the total number of face categories, and Yk is the face category to which image Y belongs.
[0039] l2(θ,x,Y)=-∑kYklog(softmax(vgg(f(x)))) (5-2)
[0040]
[0041] The final texture enhancement task framework and loss function are as follows: Figure 1 As shown in Formula 5-4.
[0042] l(θ,x,Y)=l1(θ,x,Y)+l2(θ,x,Y) (5-4)
[0043] 4. Texture enhancement network structure based on residual network:
[0044] In order to improve the accuracy of texture enhancement and minimize the problems of gradient vanishing and gradient exploding, we propose a convolutional neural network for texture enhancement of thermal infrared face images based on residual learning. The network structure we propose is as follows: Figure 2 As shown in Figure 3, the network uses residual learning to directly learn the mapping relationship between high-resolution thermal infrared images and low-resolution thermal infrared images.
[0045] The network consists of three modules: feature extraction, mapping, and image reconstruction. In addition to the image reconstruction layer of the last layer, each convolutional layer is followed by a PReLu layer, where the operation of the PreLu layer is shown in Formula 5-5. The input image resolution is 1xHxW. First, in the feature extraction part, two Conv3x3 networks are used to generate a 56xHxW feature map. As a high-scale feature map, the feature map passes through a Conv1x1 network, two Conv3x3 networks, and a Conv1x1 network to generate a new 56xHxW feature map. Finally, the feature map passes through a Conv3x3 network in turn, and the obtained 1xHxW image is summed with the original low-resolution image pixel by pixel to generate the final high-resolution thermal infrared image.
[0046]
[0047] The beneficial effects of the present invention are as follows:
[0048] 1. In the training data acquisition step (data set preparation stage), only one high-resolution thermal infrared camera is needed to collect facial images, and by aligning the lowest and highest temperatures in the two thermal infrared images, the grayscale values of the high-resolution thermal infrared images are stepped according to their calibrated temperature resolutions to generate a large number of corresponding low-resolution images, thereby reducing the difficulty of image acquisition. The present invention generates corresponding low-resolution thermal infrared images from high-resolution thermal infrared images through simulation operations, so compared with other texture enhancement algorithms based on deep learning, the present invention can greatly reduce the difficulty of data collection.
[0049] 2. In the training data acquisition step (data set preparation stage), the original thermal infrared image is denoised by using a median filter to avoid the problem that the texture enhancement algorithm enhances noise points as textures. The present invention denoises the original thermal infrared image, so the present invention can reduce the noise of the input image to a certain extent and improve the accuracy of the result.
[0050] 3. In the training step (training phase), a loss function for the recognition process is designed. The loss function simultaneously incorporates the classification cross entropy loss obtained by the high-resolution image generated by the texture enhancement network in the face recognition network and the L2 loss of the generated image and the original high-resolution image into the calculation of the loss function to ensure that the generated image obtains texture enhancement that is conducive to face recognition while retaining the original identity information. Therefore, the features of the high-resolution thermal infrared image generated by the present invention can be improved in a direction that is conducive to recognition, and the original identity information is retained.
[0051] 4. In the training step (training phase), the present invention designs a texture enhancement neural network based on residual learning. By using the residual learning structure, the accuracy of texture enhancement can be effectively improved, and the problem of gradient vanishing and gradient explosion can be avoided. Therefore, the present invention can generate more refined thermal infrared images.
[0052] In summary, the present invention relates to a method for texture enhancement of thermal infrared face images, and its potential application scenarios include public safety, intelligent security, military and other fields. The application method is to use the algorithm to perform texture enhancement on the infrared image before using the thermal infrared face image for training or reasoning of models such as face recognition, thereby improving the effect of models such as face recognition.
[0053] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.
Claims
1. A thermal infrared face image texture enhancement method for recognition process, characterized in that: The steps include: Step 1, training data acquisition step: obtain a thermal infrared image by using a high-resolution thermal infrared camera to collect data; then, through simulation operation, the grayscale value of the high-resolution thermal infrared image is stepped according to the temperature resolution calibrated by the infrared camera to obtain its corresponding low-resolution thermal infrared image; Step 2, training step: the image pairs obtained in the training data acquisition step are used as training data and sent to the texture enhancement network for training. During the training process, the output result of the texture enhancement network is sent to the visible light face recognition network again, and the weighted sum of the last inferred classification cross entropy loss and the output result of the texture enhancement network and the L2 loss of the high-resolution thermal infrared image is used as the loss function of the texture enhancement network to complete the supervision of the texture enhancement network; Step 3, prediction step: using only the face image taken by the low temperature resolution infrared camera, obtain its corresponding texture enhanced image through the texture enhancement network.
2. The thermal infrared facial image texture enhancement method according to claim 1, characterized in that: In step 1, the thermal infrared image obtained by the high-resolution thermal infrared camera is denoised to obtain a clearer thermal infrared image.
3. The thermal infrared facial image texture enhancement method according to claim 2, characterized in that: In step 1, the thermal infrared image obtained by the high-resolution thermal infrared camera is denoised by a median filtering method to obtain a clearer thermal infrared image.
4. The thermal infrared facial image texture enhancement method according to claim 1, characterized in that: The L2 loss function is formula 5-1, where h and w are the height and width of the image, respectively, x is the low-resolution thermal infrared image generated by simulation, Y is the high-resolution thermal infrared image, f is the texture enhancement network, and θ is the network parameter; 5. The thermal infrared facial image texture enhancement method according to claim 4, characterized in that: The visible light face recognition network is a VGG network.
6. The thermal infrared facial image texture enhancement method according to claim 5, characterized in that: The classification cross entropy loss is formula 5-2 and 5-3, where k is the total number of face categories, Y k is the face category to which image Y belongs; l2(θ,x,Y)=-Σ k AND k log(softmax(vgg(x)))) (5-2) The loss function of the texture enhancement network is formula 5-4, l(θ,x,Y)=l1(θ,x,Y)+l2(θ,x,Y) (5-4).
7. The thermal infrared facial image texture enhancement method according to claim 1, characterized in that: The texture enhancement network is a thermal infrared face image texture enhancement convolutional neural network based on residual learning, which adopts residual learning to directly learn the mapping relationship between high-resolution thermal infrared images and low-resolution thermal infrared images.
8. The thermal infrared facial image texture enhancement method according to claim 7, characterized in that: The texture enhancement network consists of a feature extraction module, a mapping module and an image reconstruction module. Except for the image reconstruction module, each convolution layer is followed by a PReLu layer, where the operation of the PreLu layer is shown in Formula 5-5. The input image resolution is 1xHxW. First, in the feature extraction module, two Conv3x3 networks are sequentially used to generate a 56xHxW feature map. The feature map is respectively passed through a Conv1x1 network, two Conv3x3 networks, and a Conv1x1 network to generate a new 56xHxW feature map. Finally, the new 56xHxW feature map is sequentially passed through a Conv3x3 network, and the obtained 1xHxW image is summed with the original low-resolution image by pixel to generate the final high-resolution thermal infrared image.
9. A thermal infrared face image texture enhancement system for recognition process, characterized in that: include: A memory, a processor and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the thermal infrared facial image texture enhancement method according to any one of claims 1 to 8 when called by the processor.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the thermal infrared facial image texture enhancement method according to any one of claims 1 to 8 when called by a processor.
Citation Information
Patent Citations
Super-resolution reconstruction model and method for infrared image of power equipment
CN115456877A
Infrared thermal imaging super-resolution network training method for strengthening texture information
CN116883244A
Image super-resolution enhancement method and device, training method, equipment and medium
CN117575913A
Image enhancement method for thermal infrared imaging, image processing equipment and storage medium
CN118037613A
System and method of unveiling high-resolution visible face images from low-resolution face images
WO2024105196A1
Cited By
Power grid operation site face recognition method based on deep learning
CN120635960A
A power grid operation site face recognition method based on deep learning
CN120635960B
