A method for segmenting and detecting salient human bodies in thermal imaging of nighttime airport terminals

A lightweight U-Net neural network with encoder-decoder structure addresses the challenges of human body detection in nighttime airport terminals, enhancing detection accuracy and reducing resource demands for real-time performance.

CN114067273BActive Publication Date: 2024-12-03JIANGSU UNIV OF SCI & TECH
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202111391643.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-23
Publication Date
2024-12-03
Estimated Expiration
2041-11-23

AI Technical Summary

Technical Problem

Existing human body detection methods in thermal imaging of nighttime airport terminals face challenges such as high labor intensity, labor costs, misrecognition due to posture changes and occlusions, and the need for robust detection in low-light environments, which are not adequately addressed by traditional and deep learning methods.

Method used

A method based on a lightweight U-Net neural network with an encoder-decoder structure, incorporating VGG neural network features and supervised learning, to enhance detection accuracy and reduce computing resources, handling multi-posture, multi-scale, and occluded human bodies in thermal imaging.

Benefits of technology

The method achieves accurate and real-time detection of human bodies in complex airport environments, reducing missed detections and computing resource demands, enabling efficient human recognition tasks in nighttime terminals.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114067273B_ABST
    Figure CN114067273B_ABST
Patent Text Reader

Abstract

The present invention discloses a thermal imaging significant human body segmentation detection method suitable for nighttime airport terminals. The specific steps are as follows: (1) progressively extracting encoder feature maps of five different scales from thermal imaging images; (2) using an embedding module to change the number of channels of the encoder feature map obtained in step (1) to obtain a new output feature map; (3) superimposing the output feature map obtained in step (2) with the original decoder feature map of the previous layer to obtain an original decoder feature map of the same scale; (4) extracting a thermal feature branch and a significant feature branch from the feature map obtained in step (3); (5) using a fusion module to fuse the two feature branches obtained in step (4) to obtain a significant decoder feature map; (6) after all the five significant decoder feature maps obtained in step (5) are upsampled to the original scale of the input image, a final output feature map is obtained by superimposing and fusing operations.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a salient human body segmentation detection method, and specifically to a salient human body segmentation detection method for thermal imaging of a nighttime airport terminal, which aims to detect salient human body targets with multiple postures, multiple scales and partial occlusion from thermal imaging images obtained under a nighttime airport terminal environment using thermal imaging technology. Background Art

[0002] With the vigorous development of civil aviation, terminal safety inspection has gradually become one of the important tasks to ensure airport safety. The existing terminal safety inspection method is mainly manual inspection, supplemented by information technology means such as inspection personnel positioning and central control room video monitoring. Under such inspection methods, airport managers must face the difficulties of high labor intensity of staff and rising labor costs year by year. In addition, since the inspection of the terminal is mainly performed by people, the sense of responsibility of employees has become a key factor affecting the inspection effect, and the distraction of employees during the inspection process can easily cause safety hazards in the terminal. In recent years, inspection robots have begun to be widely used in many fields such as the power industry, mining, machine room inspection and warehouse security. For this reason, the use of robots to carry out inspections has become an inevitable trend in the development of intelligent security technology in terminals.

[0003] In order to meet the needs of nighttime terminal inspections, inspection robots should have the ability to accurately identify human bodies in low-light or even dark environments. For this reason, the use of thermal imaging cameras as monitoring cameras for inspection robots has become an inevitable choice. In essence, the core of robot inspection is the process of human body detection from images captured by the camera. Traditional human body detection algorithms mostly rely on manual features, such as directional gradient histograms, integral channel features, and deformable component features. Using such methods in visible light scenes can achieve certain results, but if they are directly applied to thermal imaging environments, it is often difficult to achieve good results. However, the reason is that compared with visible light images, thermal imaging images have many disadvantages, including missing texture features, blurred visual effects, low resolution and signal-to-noise ratio, etc.

[0004] Traditional human detection methods in thermal imaging scenes are often directly extended and applied from saliency detection methods in visible light scenes. However, these methods only utilize shallow features of images. In order to complete the detection task, these traditional methods need to design features for a class of targets in a specific scene, and these features need to be manually extracted from the image. Although features can be quickly extracted from images manually, these features are often unable to cope with misrecognition caused by other factors such as human posture changes and occlusion in nighttime airport terminal scenes. In 2013, Yan et al. proposed the Hsaliency method (Yan Q, Li X, Shi J, et al. Hierarchical Saliency Detection [C] / / Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on. IEEE, 2013.) as a traditional saliency method. When it is applied to scenes with strong contrast between the target and the background, it has a certain effect, but it cannot meet the actual needs of robot inspection in nighttime airport terminal scenes.

[0005] Deep neural networks can directly use training samples to deeply mine the potential features of data by learning end-to-end through multi-layer neural networks. To this end, the feature representation of human bodies in different states in the night airport terminal scene can be obtained by autonomous learning using deep models, which effectively avoids the shortcomings of traditional manually designed features. However, when using image segmentation methods to detect human bodies in infrared images, some objective factors that actually exist in the terminal and affect target recognition need to be considered. First, the human body in the terminal has a variety of postures such as standing, walking, sitting, squatting, etc. At the same time, fixed objects such as seats, beams and columns in the terminal will also partially block the human body target. Secondly, when using infrared cameras to obtain target images, heat sources such as light sources and display screens in the terminal will also be imaged in infrared images. Furthermore, when the inspection robot detects the surrounding environment from a horizontal perspective, due to the difference in distance from the camera, the human body target will appear at different scales in the image. In addition, in order to adapt to the limited computing resources of the airport inspection robot platform, the human body detection method in the night airport terminal scene must be able to use limited computing resources to achieve accurate detection of human bodies.

[0006] Amulet(Zhang P,D Wang,Lu H,et al.Amulet:Aggregating Multi-levelConvolutional Features for Salient Object Detection[C] / / IEEE ComputerSociety,2017.), BASNet(Qin Recognition (CVPR). IEEE, 2019.), CPD (Cascaded Partial Decoder for Fast and Accurate Salient Object Detection [C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, 2019.) and SRM (Wang T, Borji A, Zhang L, et al. A Stagewise Refinement Model for Detecting SalientObjects in Images[C] / / 2017IEEE International Conference on Computer Vision (ICCV). IEEE, 2017.) are four classic infrared image human body detection methods based on deep models proposed in recent years. From the running results of these four methods on the data sets actually collected at the airport, in general, the SRM method has no special advantages in detection accuracy and detection speed; the Amulet method has high detection accuracy and good robustness, but the Amulet method has high requirements on computing power and it is difficult to meet the requirements of real-time detection of patrol robots; although the CPD method has certain advantages in detection speed, its overall detection accuracy is the worst among these deep models. The BASNet method pays more attention to the segmentation effect of the target edge pixels while paying attention to lightweight, so the model has both good robustness and fast running speed. However, the BASNet model often misses detection when dealing with small-scale targets at a distance.

[0007] Compared with the target detection model that is prone to missed detection and has high requirements for computing resources, the target segmentation model, such as the U-Net neural network model (Ronneberger O, Fischer P, Brox TU-Net: Convolutional Networks for Biomedical Image Segmentation [J]. Springer, Cham, 2015.), is more suitable for the terminal night inspection task with higher requirements for safety and real-time performance. To this end, in order to cope with unfavorable factors such as multi-posture, multi-scale, local occlusion and heat source interference, and reduce the occurrence of missed detection, the present invention discloses a significant human segmentation detection method for thermal imaging of night airport terminal. The method of the present invention is based on the U-Net neural network model, improves the decoder structure of the U-net network, improves the sensitivity of the method to the high temperature area in the image, reduces the adverse effects of various interference factors on the human detection performance in the night airport terminal scene, and improves the detection accuracy in the actual application scene. In addition, the method of the present invention makes a series of lightweight improvements to the deep network structure, improves the detection effect of the model detecting human targets, reduces the demand for computing resources of the method, and adapts to the actual situation of limited computing resources of the inspection robot. Summary of the invention

[0008] The purpose of the present invention is to address the defects of the prior art and propose a method for segmenting and detecting significant human bodies in thermal imaging of airport terminals at night. The method integrates the thermal features in thermal imaging images into a deep neural network method that conforms to the encoder-decoder structure, so that the method can perform real-time detection on a robot platform with limited computing resources, and can more accurately detect significant human bodies with high heat in complex situations in the terminal, reducing the occurrence of missed detections.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0010] A method for detecting significant human body segmentation in thermal imaging of an airport terminal at night, comprising the following steps:

[0011] a. Based on the encoder-decoder structure, the model is lightweight and improved. The original thermal image I is input, and the VGG neural network is used to progressively extract encoder module feature maps E of 5 different scales from the thermal image. i ;

[0012] b. Convert the encoder module feature map E in step a i After the lightweight embedding module, the corresponding feature map A of the embedding module output is obtained i ;

[0013] c. Output feature map A of the embedding module obtained in step bi Compared with the upsampled original decoder feature map D of the previous layer i+1 Superposition and combination to obtain the original decoder feature map D of the same scale i ;

[0014] d. Use the original decoder feature map D of 5 different scales obtained in step c i , learn the heat feature information and significant feature information contained therein, and extract each feature map D i The corresponding thermal feature branch H (i) and the salient feature branch S (i) ;

[0015] e. In the fusion module, the two feature branches H of each scale obtained in step d are combined (i) and S (i) The channel number change operation is performed, which is different from the previous scale feature map S that has been upsampled. i+1 Fusion, get the significant decoder feature map S i ;

[0016] f. The five salient decoder feature maps S obtained in step e are i All are upsampled to the scale of the input image and combined to obtain the final output feature map S0;

[0017] In the above, i=1,2,3,4,5.

[0018] Further description, the lightweight improvement method of the model includes the following steps:

[0019] (1) Discard the last fully connected layer in the VGG network and only use the first 5 layers of the VGG neural network to extract encoder feature maps of different scales;

[0020] (2) All input images of different scales are adjusted to 320×320, which reduces the number of operation parameters and ensures that the method has good effects on input images of various scales;

[0021] (3) In step b, the lightweight embedding module is implemented by aggregating the maximum value of each group of channels in order to change the number of channels. The specific generation process adopts the calculation formula:

[0022]

[0023] Among them A i Represents the feature map E i The corresponding feature map after the embedding module operation is compared with E i , A iThe number of channels has changed. i is the index value of the encoder feature map, and its value range is [1,5]. The term represents the encoder feature map E i No. channels, where j and k are integers, n and m represent the number of input and output channels respectively, and the number of original channels in each group is n / m;

[0024] (4) Reduce the feature map with 256 or 128 channels in the decoder module to a feature map with 64 channels, thereby reducing method parameters;

[0025] (5) The upsampling method in the decoder module uses the bilinear interpolation method. In the process of calculating the final output feature map S0, when the output scale reaches 80×80, the upsampling method uses the faster nearest neighbor method.

[0026] Further description, the original decoder feature map D in step c i The generation of needs to be done through supervised learning methods, and perform superposition and upsampling operations. The specific generation process uses the calculation formula:

[0027]

[0028] Where T represents supervised learning, which consists of a series of convolutional layers and activation functions. The subscripts represent the corresponding modules, cat represents the superposition operation, and up represents the upsampling operation. Note that when i is equal to 5, D in the above formula i+1 Equal to A5, and no upsampling operation is performed.

[0029] Further description, the thermal feature branch H in step d (i) and the saliency map branch S (i) The heat map and saliency map are used respectively and obtained through supervised learning. The specific generation process adopts the calculation formula:

[0030]

[0031]

[0032] Where T represents supervised learning, which consists of a series of convolutional layers and activation functions. The subscripts represent the corresponding modules. The saliency map is obtained by manual annotation, while the heat map comes from the high grayscale area in the thermal imaging image and is generated by threshold segmentation. The grayscale threshold θ during segmentation is TA The standard deviation σ of the input image I I and average The specific calculation process is calculated using the formula:

[0033]

[0034] Further description, the significant decoder feature map S in step e i The generation of needs to be done through supervised learning methods, and perform superposition and upsampling operations. The specific generation process uses the calculation formula:

[0035]

[0036] Where cp represents the same operation of changing the number of channels as the embedding module, cat represents the superposition operation, and up represents the upsampling operation. Note that when i is equal to 5, S in the above formula i+1 Equal to D5, and no upsampling operation is performed.

[0037] To further describe, the final output feature map S0 in step f first needs to be supervised by using the manually annotated saliency map, and then perform superposition and upsampling operations. The specific generation process uses the calculation formula:

[0038]

[0039] Where T represents supervised learning, which consists of a series of convolutional layers and activation functions. The subscripts represent the corresponding modules, cat represents the superposition operation, and up represents the upsampling operation.

[0040] Further description, the method designs the prediction graph generation process and model loss function, and the specific method is as follows:

[0041] (h-1), branch the heat map H (i) The feature map and saliency map branch S (i) The number of channels of the feature map and the final output feature map S0 are all modified to 1 to calculate the corresponding prediction map. The corresponding prediction map P i H , and The specific generation process uses the calculation formula:

[0042] P i H =cp(H (i) )

[0043]

[0044]

[0045] Where cp represents the same method of changing the number of channels as the embedded module;

[0046] (h-2), by calculating the heat map branch H (i) and the saliency map branch S (i)The difference between the branch prediction graph and the real image at each scale is used to design the loss functions on these two branches respectively. and The specific calculation method uses the following formula:

[0047]

[0048]

[0049] Where L represents the loss value of the branch, the superscript L represents the branch name, and S (i) represents a significant branch, H (i) represents the heat branch, n represents the total number of pixels, m represents the index value of the pixel, the symbol G represents the real image, the superscript G represents the image name on the corresponding branch, and the subscript represents the location of the current pixel, the symbol P represents the predicted image, the superscript H represents the name of the predicted image on the corresponding branch, and the subscript represents the location of the current pixel. It should be noted that and The value of is either 0 or 1, and and The value of is in the interval [0,1];

[0050] (h-3) Design the loss function of the output feature map S0 When , only the predicted image of feature map S0 and the real image of the first branch in the salient branch are calculated. The difference between them, and through the loss function Added The item is used to increase the weight of pixels in the high temperature area of ​​the heat map. The specific calculation method uses the following formula:

[0051]

[0052] Where n represents the total number of pixels, and m represents the index value of the pixel;

[0053] (h-4), through and The overall loss L of the model is calculated by the weighted sum of the three types of losses. The specific calculation method uses the following formula:

[0054]

[0055] in, for The weight is set to 1.5, w i is the weight of the overall loss on these two branches. According to the principle that the larger the image scale, the greater the weight, w i(i=1,2…,5) are set to 0.2, 0.4, 0.6, 0.8, 1 respectively.

[0056] Beneficial effects: Compared with the existing technology, the nighttime airport terminal thermal imaging significant human body segmentation detection method of the present invention has the following advantages:

[0057] The present invention uses a segmentation method to perform human detection tasks in thermal imaging scenes, and can effectively detect human targets in various postures such as standing, walking, sitting, squatting, etc. in complex airport environments, and can efficiently handle the segmentation of significant human targets in multi-scale and partially occluded situations, greatly improving the situation where existing methods often miss detection, and efficiently realizing human recognition tasks in different scenes of night terminals. In addition, the present invention reduces the model's demand for robot platform computing resources through a series of model lightweight designs, and enables the method of the present invention to obtain detection results at a speed of more than 40FPS, meeting the needs of real-time detection of inspection robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flow chart of the present invention for the segmentation and detection of significant human bodies in thermal imaging of an airport terminal at night;

[0059] Figure 2 This is a detailed structural diagram of the present invention for the segmentation and detection of significant human bodies in thermal imaging of an airport terminal at night;

[0060] Figure 3 is the input original thermal imaging image;

[0061] Figure 4 is the heat map obtained from the original thermal imaging image;

[0062] Figure 5 Manually labeled saliency maps;

[0063] Figure 6 It is the flow chart of the fusion module;

[0064] Figure 7 This is a diagram of significant human body detection results;

[0065] Figure 8 This is a visual comparison chart;

[0066] The solid arrows represent data flow, and the dotted arrows represent supervised learning. DETAILED DESCRIPTION

[0067] The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0068] The simulation experiment conducted by the present invention is implemented by programming on a PC test platform with a GPU model of RTX 2080Ti and a memory of 16G.

[0069] like Figure 1 As shown in the figure, a method for detecting significant human body segmentation in thermal imaging of a nighttime airport terminal is described in detail in the following steps:

[0070] (1) Based on the encoder-decoder structure, such as Figure 2 As shown, input the original thermal image I, such as Figure 3 As shown in Figure 2, the VGG neural network is used to progressively extract encoder module feature maps E of 5 different scales from thermal imaging images. i ;

[0071] (2) The encoder module feature map E in step (1) i After the lightweight embedding module, the corresponding feature map A of the embedding module output is obtained i ;

[0072] (3) Output feature map A of the embedding module obtained in step (2) i Compared with the upsampled original decoder feature map D of the previous layer i+1 Superposition and combination, then obtain the original decoder feature map D of the same scale i ;

[0073] (4) Using the original decoder feature maps D of five different scales obtained in step (3) i , learn the heat feature information and significant feature information contained therein, and extract each feature map D i The corresponding thermal feature branch H (i) and the salient feature branch S (i) ;

[0074] (5) In the fusion module, the two feature branches H of each scale obtained in step (4) are combined (i) and S (i) The channel number change operation is performed, which is different from the previous scale feature map S that has been upsampled. i+1 Fusion, get the significant decoder feature map S i ;

[0075] (6) The five salient decoder feature maps S obtained in step (5) are i All are upsampled to the scale of the input image and combined to obtain the final output feature map S0;

[0076] Among them, i=1,2,3,4,5.

[0077] The lightweight improvement method of the model includes the following steps:

[0078] (1) Discard the last fully connected layer in the VGG network and only use the first 5 layers of the VGG neural network to extract encoder feature maps of different scales;

[0079] (2) All input images of different scales are adjusted to 320×320, which reduces the number of operation parameters and ensures that the method has good effects on input images of various scales;

[0080] (3) Figure 2 In order to change the number of channels, the embedding module is implemented by aggregating the maximum value of each group of channels. The specific generation process adopts the calculation formula:

[0081]

[0082] Among them A i Represents the feature map E i The corresponding feature map after the embedding module operation is compared with E i , A i The number of channels has changed. i is the index value of the encoder feature map, and its value range is [1,5]. The term represents the encoder feature map E i No. channels, where j and k are integers, n and m represent the number of input and output channels respectively, and the number of original channels in each group is n / m;

[0083] (4) Reduce the feature map with 256 or 128 channels in the decoder module to a feature map with 64 channels, thereby reducing method parameters;

[0084] (5) The upsampling method in the decoder module uses the bilinear interpolation method. In the process of calculating the final output feature map S0, when the output scale reaches 80×80, the upsampling method uses the faster nearest neighbor method.

[0085] The original decoder feature map D i The generation of needs to be done through supervised learning methods, and perform superposition and upsampling operations. The specific generation process uses the calculation formula:

[0086]

[0087] Where T represents supervised learning, which consists of a series of convolutional layers and activation functions. The subscripts represent the corresponding modules, cat represents the superposition operation, and up represents the upsampling operation. Note that when i is equal to 5, D in the above formula i+1 Equal to A5, and no upsampling operation is performed.

[0088] The thermal characteristic branch H (i)and the saliency map branch S (i) Use heat map and saliency map respectively, such as Figure 4 , Figure 5 As shown, it is obtained through supervised learning. The specific generation process uses the calculation formula:

[0089]

[0090]

[0091] Where T represents supervised learning, which consists of a series of convolutional layers and activation functions. The subscripts represent the corresponding modules. The saliency map is obtained by manual annotation, while the heat map comes from the high grayscale area in the thermal imaging image and is generated by threshold segmentation. The grayscale threshold θ during segmentation is TA The standard deviation σ of the input image I I and average The specific calculation process is calculated using the formula:

[0092]

[0093] The salient decoder feature map S i The generation of needs to be done through supervised learning methods, and perform superposition and upsampling operations, such as Figure 6 As shown, the specific generation process adopts the calculation formula:

[0094]

[0095] Where cp represents the same operation of changing the number of channels as the embedding module, cat represents the superposition operation, and up represents the upsampling operation. Note that when i is equal to 5, S in the above formula i+1 Equal to D5, and no upsampling operation is performed.

[0096] The final output feature map S0 first needs to be supervised by using a manually annotated saliency map, and then perform superposition and upsampling operations. The specific generation process uses the calculation formula:

[0097]

[0098] Where T represents supervised learning, which consists of a series of convolutional layers and activation functions. The subscripts represent the corresponding modules, cat represents the superposition operation, and up represents the upsampling operation.

[0099] The method designs the prediction graph generation process and model loss function as follows:

[0100] (1) Branch the heat map H (i) The feature map and saliency map branch S (i)The number of channels of the feature map and the final output feature map S0 are all changed to 1 to calculate the corresponding prediction map. P H , and The specific generation process uses the calculation formula:

[0101]

[0102]

[0103]

[0104] Where cp represents the same method of changing the number of channels as the embedded module;

[0105] (2) By calculating the heat map branch H (i) and the saliency map branch S (i) The difference between the branch prediction graph and the real image at each scale is used to design the loss functions on these two branches respectively. and The specific calculation method uses the following formula:

[0106]

[0107]

[0108] Where L represents the loss value of the branch, the superscript L represents the branch name, and S (i) represents a significant branch, H (i) represents the heat branch, n represents the total number of pixels, m represents the index value of the pixel, the symbol G represents the real image, the superscript G represents the image name on the corresponding branch, and the subscript represents the location of the current pixel, the symbol P represents the predicted image, the superscript H represents the name of the predicted image on the corresponding branch, and the subscript represents the location of the current pixel. It should be noted that and The value of is either 0 or 1, and and The value of is in the interval [0,1];

[0109] (3) Design the loss function of the output feature map S0 When , only the predicted image of feature map S0 and the real image of the first branch in the salient branch are calculated. The difference between them, and through the loss function Added The item is used to increase the weight of pixels in the high temperature area of ​​the heat map. The specific calculation method uses the following formula:

[0110]

[0111] Where n represents the total number of pixels, and m represents the index value of the pixel;

[0112] (4) Through and The overall loss L of the model is calculated by the weighted sum of the three types of losses. The specific calculation method uses the following formula:

[0113]

[0114] Among them, w s0 for The weight is set to 1.5, w i is the weight of the overall loss on these two branches. According to the principle that the larger the image scale, the greater the weight, w i (i=1,2…,5) are set to 0.2, 0.4, 0.6, 0.8, 1 respectively.

[0115] Finally, the final output prediction image is multiplied by 255 and restored to the input image scale to obtain the visual output result, such as Figure 7 shown.

[0116] In order to verify the efficiency and effectiveness of the model, we conducted experiments on four datasets: ATH, OSU, KAIST, and FLIR. OSU, KAIST, and FLIR are public datasets, and the ATH dataset is an actual dataset collected in an airport terminal. The ATH dataset includes nearly 4,000 images collected in actual airport scenes, including a variety of complex human samples such as multi-pose, multi-scale, and occluded.

[0117] We experimentally compared the performance of the proposed method with that of five other saliency methods, including Hsaliency, Amulet, BASNet, CPD, and SRM, on four datasets, namely OSU, KAIST, FLIR, and ATH. OSU, KAIST, and FLIR are three public datasets, while ATH is a dataset actually collected in an airport terminal.

[0118] Judging from the F-measure value and mIOU value, the method of the present invention has excellent overall performance on the three public data sets and has the best performance on the airport actual data set ATH data set. In general, although the performance of the Amulet method and the method of the present invention on the four data sets is relatively close, and both have high accuracy and robustness, the Amulet method has high requirements on computing power, and its FPS value is only 17, which is difficult to meet the requirements of real-time detection of inspection robots. In addition, although the CPD method is relatively close to the method of the present invention in detection speed, and the FPS values ​​of both are above 40, its overall detection accuracy is much lower than that of the method of the present invention. It can be seen that the method of the present invention has an overall balanced advantage in detection speed and detection accuracy, especially in the actual data set ATH of the terminal building, it has obvious performance advantages.

[0119] Figure 8 Some visualization examples of 6 experimental methods on 4 data sets are shown. The BASNet method pays more attention to the segmentation effect of the target edge pixels while paying attention to lightweight, so the model has both good robustness and fast running speed. However, the BASNet model often misses detections when dealing with distant small-scale targets. In comparison, the method of the present invention can pay more attention to the pixels in the high-temperature area, thereby ensuring that the model still has good results even when facing small-scale targets. In addition, since the present invention has a lightweight design for the method, our model can use fewer parameters and computing power to complete the model's calculations. For this reason, the method of the present invention can detect results at a speed of more than 40FPS, which is fully capable of meeting real-time detection tasks under thermal imaging cameras.

Claims

1. A method for segmenting and detecting significant human bodies in thermal imaging of nighttime airport terminals, characterized in that: The steps include: a. Using the encoder-decoder structure, the original thermal image I is input, and the VGG neural network is used to progressively extract encoder module feature maps E of 5 different scales from the thermal image i ; b. Convert the encoder module feature map E in step a i After the lightweight embedding module, the corresponding feature map A is output through the embedding module i ; c. Output feature map A of the embedding module obtained in step b i Compared with the upsampled original decoder feature map D of the previous layer i+1 Superposition and combination, get the original decoder feature map D of the same scale i ; d. The original decoder feature maps D of 5 different scales obtained in step c i , learn the heat feature information and significant feature information contained therein, and extract each feature map D i The corresponding thermal feature branch H (i) and the salient feature branch S (i) ; e. The two feature branches H of each scale obtained in step d are (i) and S (i) After the channel number change operation is performed in the fusion module, it is compared with the previous scale feature map S that has been upsampled. i+1 Fusion, get the significant decoder feature map S i ; f. The five salient decoder feature maps S obtained in step e are i All are upsampled to the scale of the input image and combined to obtain the final output feature map S0; The following steps are also included: (1) Discard the last fully connected layer in the VGG network and only use the first 5 layers of the VGG neural network to extract encoder feature maps of different scales; (2) All input images of different scales are adjusted to 320×320 to reduce the number of operation parameters; (3) In order to achieve the change of the number of channels, the embedding module is implemented by aggregating the maximum value of each group of channels. The specific generation process adopts the calculation formula: Among them A i Represents the feature map E i The corresponding feature map after the embedding module operation is compared with E i , A i The number of channels has changed. i is the index value of the encoder feature map, and its value range is [1,5]. The term represents the encoder feature map E i No. channels, where j and k are integers, n and m represent the number of input and output channels respectively, and the number of original channels in each group is n / m; (4) Reduce the feature map with 256 or 128 channels in the decoder module to a feature map with 64 channels; (5) The upsampling method in the decoder module uses the bilinear interpolation method. In the process of calculating the final output feature map S0, when the output scale reaches 80×80, the upsampling method uses the faster nearest neighbor method.

2. The method for detecting significant human body segmentation in thermal imaging of nighttime airport terminals according to claim 1 is characterized in that: The original decoder feature map D in step c i The generation of needs to be done through supervised learning methods, and perform superposition and upsampling operations. The specific generation process uses the calculation formula: Where T represents supervised learning, which consists of a series of convolutional layers and activation functions. The subscripts represent the corresponding modules, cat represents the superposition operation, and up represents the upsampling operation. When i is equal to 5, D in the above formula i+1 Equal to A5, and no upsampling operation is performed.

3. The method for detecting significant human body segmentation in thermal imaging of nighttime airport terminals according to claim 1, characterized in that: The thermal feature branch H in step d (i) and the saliency map branch S (i) The heat map and saliency map are obtained through supervised learning, and the specific generation process adopts the calculation formula: Where T represents supervised learning, which consists of a series of convolutional layers and activation functions. The subscripts represent the corresponding modules. The saliency map is obtained by manual annotation, while the heat map comes from the high grayscale area in the thermal imaging image and is generated by threshold segmentation. The grayscale threshold θ during segmentation is TA The standard deviation σ of the input image I I and average The specific calculation process is calculated using the formula:

4. The method for detecting significant human body segmentation in thermal imaging of nighttime airport terminals according to claim 1, characterized in that: The salient decoder feature map S of the fusion module in step e i The generation needs to be done through supervised learning methods, and perform superposition and upsampling operations. The specific generation process uses the calculation formula: Where cp represents the same channel number change operation as the embedding module, cat represents the superposition operation, and up represents the upsampling operation. When i is equal to 5, S in the above formula i+1 Equal to D5, and no upsampling operation is performed.

5. The method for detecting significant human body segmentation in thermal imaging of nighttime airport terminal buildings according to claim 1, characterized in that: The final output feature map S0 in step f first needs to be supervised by using the manually annotated saliency map, and then perform superposition and upsampling operations. The specific generation process uses the calculation formula: Where T represents supervised learning, which consists of a series of convolutional layers and activation functions. The subscripts represent the corresponding modules, cat represents the superposition operation, and up represents the upsampling operation.

6. The method for detecting significant human body segmentation in thermal imaging of nighttime airport terminals according to claim 1, characterized in that: The specific method is as follows: (h-1), branch the heat map H (i) The feature map and saliency map branch S (i) The number of channels of the feature map and the final output feature map S0 are all modified to 1 to calculate the corresponding prediction map. The corresponding prediction map P i H , and The specific generation process uses the calculation formula: P i H =cp(H (i) ) Where cp represents the same method of changing the number of channels as the embedded module; (h-2), by calculating the heat map branch H (i) and the saliency map branch S (i) The difference between the branch prediction graph and the real image at each scale is used to design the loss functions on these two branches respectively. and The specific calculation method uses the following formula: Where L represents the loss value of the branch, the superscript L represents the branch name, and S (i) represents a significant branch, H (i) represents the heat branch, n represents the total number of pixels, m represents the index value of the pixel, the symbol G represents the real image, the superscript G represents the image name on the corresponding branch, and the subscript represents the location of the current pixel, the symbol P represents the predicted image, the superscript H represents the name of the predicted image on the corresponding branch, and the subscript represents the location of the current pixel. It should be noted that and The value of is either 0 or 1, and and The value of is in the interval [0,1]; (h-3) Design the loss function of the output feature map S0 When , only the predicted image of feature map S0 and the real image of the first branch in the salient branch are calculated. The difference between them, and through the loss function Add The item is used to increase the weight of pixels in the high temperature area of ​​the heat map. The specific calculation method uses the following formula: Where n represents the total number of pixels, and m represents the index value of the pixel; (h-4), through and The overall loss L of the model is calculated by the weighted sum of the three types of losses. The specific calculation method uses the following formula: in, for The weight is set to 1.5, w i is the weight of the overall loss on these two branches. According to the principle that the larger the image scale, the greater the weight, w i Set to 0.2, 0.4, 0.6, 0.8 and 1 respectively, i=1,2…,5.

Citation Information

Patent Citations

  • Human body action real-time identification method based on infrared image

    CN110363140A

  • Improved deep learning pornographic image recognition method

    CN111199233A

  • Context-aware multi-view three-dimensional reconstruction system and method based on deep learning

    CN112489217A

  • RGB-T saliency target detection method based on modal correlation and double-perception decoder

    CN113298094A

  • Human action recognition method

    WO2021129569A1