Method and system for detecting salient human targets in thermal infrared images

Abstract

The embodiment of the present invention provides a method and system for detecting significant human targets in thermal infrared images, which belongs to the field of image processing and recognition technology. The method comprises: in a pre-built significant human detection deep network, using processed historical images to train a significant human target detection model to obtain a significant human target detection model; wherein the processed historical images are generated by processing historical information of thermal infrared images containing human targets based on a preset data enhancement algorithm; obtaining an infrared image to be processed; importing the infrared image to be processed as input data into the significant human target detection model, performing model training, and outputting the training results; identifying human targets according to the training results. The scheme of the present invention realizes accurate human target recognition in infrared images, and improves the accuracy and intelligence of significant human target recognition in infrared images.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and recognition, and in particular to a method for detecting significant human targets in thermal infrared images and a system for detecting significant human targets in thermal infrared images. Background Art

[0002] Salient human object detection (SOD) is to segment out the most attractive visual target in the image, and has a wide range of applications in visual tracking, image segmentation and other fields. Traditional salient object detection algorithms mostly use manual feature methods such as superpixel similarity, histogram, pixel gradient ratio, etc. for detection. However, the detection effect is poor in the case of small objects and objects being occluded. Thermal infrared images are widely used because of their special imaging principles, especially in military reconnaissance, resource exploration and other fields. However, thermal infrared images have obvious shortcomings, such as poor resolution, low contrast, low signal-to-noise ratio, blurred regional boundaries, and no linear relationship between grayscale distribution and target reflection characteristics. Therefore, the difficulty of salient object detection in thermal infrared images is higher. The existing salient object detection algorithms cannot meet the accuracy requirements of human object detection in thermal infrared images, so it is necessary to create a new salient human object detection method for thermal infrared images. Summary of the invention

[0003] The purpose of the embodiments of the present invention is to provide a method and system for detecting significant human targets in thermal infrared images, so as to at least solve the problem that the existing significant target detection algorithms cannot meet the accuracy requirements of human target detection in thermal infrared images.

[0004] In order to achieve the above-mentioned purpose, the first aspect of the present invention provides a method for detecting significant human targets in thermal infrared images, the method comprising: in a pre-built significant human detection deep network, using processed historical images to train a significant human target detection model to obtain a significant human target detection model; wherein the processed historical images are generated by processing historical information of thermal infrared images containing human targets based on a preset data enhancement algorithm; obtaining an infrared image to be processed; importing the infrared image to be processed as input data into the significant human target detection model, performing model training, and outputting the training results; and identifying human targets according to the training results.

[0005] Optionally, the preset data enhancement algorithm includes at least: color transformation, geometric transformation and cropping operations.

[0006] Optionally, the pre-built salient human body detection deep network is built based on a HED edge detection model.

[0007] Optionally, the pre-built salient human body detection deep network includes: a backbone network, a codec layer and a fusion layer.

[0008] Optionally, the method further includes: constructing the backbone network; deleting the fully connected layer of the VGG network; and adding multiple convolutions to the VGG network with the fully connected layer deleted to obtain the backbone network.

[0009] Optionally, the codec layer is a U-type codec of a single scale or an FPN connection codec between different scales.

[0010] Optionally, the fusion layer is constructed based on the HED model fusion preset loss function; the fusion layer includes an upsampling operation and a Concat operation; the calculation formula of the upsampling operation is:

[0011]

[0012] Among them, W input is the width of the input image; H input is the height of the input image; W i , H i They are the width and height of the image output by the codec layer respectively.

[0013] Optionally, the preset loss function expression is:

[0014]

[0015] Among them, lk is the loss of the kth output result; K represents the number of output results; α k is the weight for each loss function.

[0016] A second aspect of the present invention provides a thermal infrared image significant human target detection system, the system comprising: an acquisition unit, used to obtain historical information of thermal infrared images containing human targets, and used to obtain infrared images to be processed; a processing unit, used to train a significant human target detection model using processed historical images in a pre-built significant human detection deep network, to obtain a significant human target detection model; wherein the processed historical images are generated by processing historical information of thermal infrared images containing human targets based on a preset data enhancement algorithm; a training unit, used to import the infrared images to be processed as input data into the significant human target detection model, perform model training, and output training results; and an identification unit, used to identify human targets according to the training results.

[0017] On the other hand, the present invention provides a computer-readable storage medium having instructions stored thereon, which, when executed on a computer, enables the computer to execute the above-mentioned method for detecting significant human targets in thermal infrared images.

[0018] Through the above technical solution, a large amount of historical information of thermal infrared images containing human targets is enhanced, and then a significant human target detection model is trained in the pre-built significant human detection deep network. Based on the significant human target detection model obtained through training, human recognition is performed in subsequent infrared images. Human target recognition in infrared images is realized, and the accuracy and intelligence of infrared image significant human target recognition are improved.

[0019] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0021] Figure 1 It is a flowchart of the steps of a method for detecting significant human targets in thermal infrared images provided by one embodiment of the present invention;

[0022] Figure 2 is a flowchart of steps for training a significant human target detection model provided by one embodiment of the present invention;

[0023] Figure 3 It is a system structure diagram of a thermal infrared image salient human target detection system provided by an embodiment of the present invention.

[0024] Description of Reference Numerals

[0025] 10-acquisition unit; 20-processing unit; 30-training unit; 40-recognition unit. DETAILED DESCRIPTION

[0026] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0027] Figure 3 1 is a system structure diagram of a thermal infrared image salient human target detection system provided by an embodiment of the present invention. Figure 3As shown, an embodiment of the present invention provides a thermal infrared image significant human target detection system, the system comprising: an acquisition unit 10, used to obtain historical information of thermal infrared images containing human targets, and used to obtain infrared images to be processed; a processing unit 20, used to train a significant human target detection model using processed historical images in a pre-built significant human detection deep network, and obtain a significant human target detection model; a training unit 30, used to import the infrared image to be processed as input data into the significant human target detection model, perform model training, and output training results; and an identification unit 40, used to identify human targets according to the training results.

[0028] Figure 1 1 is a flowchart of a method for detecting significant human targets in thermal infrared images provided by an embodiment of the present invention. Figure 1 As shown, an embodiment of the present invention provides a method for detecting significant human targets in thermal infrared images, the method comprising:

[0029] Step S10: In the pre-built salient human detection deep network, the processed historical head portraits are used to train a salient human target detection model to obtain a salient human target detection model. Figure 2 , including the following steps:

[0030] Step S101: Acquire historical information of thermal infrared images containing human targets, and perform historical image processing based on a preset data enhancement algorithm.

[0031] Specifically, significant human target detection (SOD) is to segment the most attractive visual target in the image, and it has a wide range of applications in the fields of visual tracking and image segmentation. Limited by the poor detection effect, low resolution and low contrast of the traditional significant target detection algorithm, the scheme of the present invention adopts VGG and MobileNet networks as the backbone network. The U-shaped hole convolution is used as the encoding and decoding layer, and then the multi-layer decoding feature fusion method is used to obtain the final detection result. It is well known that in order to obtain a training model close to reality, a large amount of historical data containing target laws is required as training samples. Therefore, before the model training, the scheme of the present invention needs to obtain a large amount of historical data as training samples. These historical data are thermal infrared images containing human targets in the database. These images are widely stored in the public network library, the fire library and the monitoring equipment library. Through the corresponding acquisition unit 10, they are connected with these resource libraries, and the associated images are downloaded through these resource libraries. After obtaining a large amount of historical image information, because the thermal infrared images containing human targets stored in different acquisition devices and storage devices have large differences in resolution, contrast, signal-to-noise ratio, regional boundaries, grayscale distribution and anti-features. If these image information are used directly for model training, they are bound to cause mutual interference, and the training model obtained by training must be very different from the actual needs. In order to avoid this situation, it is preferred that after obtaining the historical image information, image processing is performed through a preset data enhancement algorithm. This can not only enhance the features in the image, making the subsequent training effect higher, but also process all historical image information into images under the same standard to avoid training interference.

[0032] Preferably, the preset data enhancement algorithm includes at least: color transformation, geometric transformation and cropping operation. Among them, the core enhancement algorithm is geometric transformation and cropping operation, which mainly targets objects of different shapes and angles in infrared images, enhances image features through random flipping and geometric changes, and then uses cropping operation to simulate significant human detection targets blocked by obstacles.

[0033] Step S102: Perform model training based on historical image processing.

[0034] Specifically, model training requires corresponding deep network support. Based on the requirements of significant human target detection, the present invention designs a deep network adapted for significant human target detection model training, which is called a significant human detection deep network. Specifically, it includes the following parts:

[0035] 1) Backbone network:

[0036] Specifically, the backbone network of the scheme of the present invention is built based on the traditional VGG network. The VGG network uses several consecutive 3x3 convolution kernels to replace the larger convolution kernels in AlexNet, which are 11x11, 7x7, and 5x5 respectively. For a given receptive field (the local size of the input image related to the output), it is better to use a stacked small convolution kernel than a large convolution kernel, because multiple layers of nonlinear layers can increase the depth of the network to ensure learning more complex patterns, and because there are fewer parameters, the cost is relatively small. In short, in VGG, three 3x3 convolution kernels are used to replace the 7x7 convolution kernel, and two 3x3 convolution kernels are used to replace the 5×5 convolution kernel. The main purpose of doing so is to increase the depth of the network while ensuring the same receptive field, and to improve the effect of the neural network to a certain extent. Among them, there is at least one fully connected layer at the end of the VGG network. The fully connected layer has certain disadvantages in image cutting. Preferably, this problem can be avoided by replacing the original fully connected layer with conv. Because conv can complete the operation by adjusting the stride, conv performs well in image cutting. For example, when the image size is 256×256 during training, a feature map of 7×7×512 is obtained before inputting fc or conv. If the image size is changed to 512×512, a feature map of 14×14×512 is obtained before inputting fc or conv. In this case, fc cannot process such a feature map because the weight matrix is ​​fixed. Based on this, the scheme of the present invention discards the last FC fully connected layer of the original VGG network and instead adds two convolution operations distributed as Conv2d (input=512, output=1024, kernel_size=3) and Conv2d (input=1024, output=512, kernel_size=1). The improved network can take into account the detection of human targets of different sizes. The hierarchical input / output of the backbone network designed by the present invention is shown in Table 1:

[0037] Operation Name Input size Output size Whether to output Input Data 224*224*3 no Conv_1(3*3) 224*224*3 224*224*64 no Conv_2(3*3) 224*224*64 224*224*64 no MaxPool(2*2) 224*224*64 112*112*64 no Conv_3(3*3) 112*112*64 112*112*128 no Conv_4(3*3) 112*112*128 112*112*128 yes MaxPool(2*2) 112*112*128 56*56*128 no Conv_5(3*3) 56*56*128 56*56*256 no Conv_6(3*3) 56*56*256 56*56*256 no Conv_7(3*3) 56*56*256 56*56*256 yes MaxPool(2*2) 56*56*256 28*28*256 no Conv_8(3*3) 28*28*256 28*28*512 no Conv_9(3*3) 28*28*512 28*28*512 no Conv_10(3*3) 28*28*512 28*28*512 yes MaxPool(2*2) 28*28*512 14*14*512 no Conv_11(3*3) 14*14*512 14*14*512 no Conv_12(3*3) 14*14*512 14*14*512 no Conv_13(3*3) 14*14*512 14*14*512 yes MaxPool(2*2) 14*14*512 7*7*512 no Conv_add1(3*3) 7*7*512 7*7*1024 no Conv_add2(3*3) 7*7*1024 7*7*512 yes

[0038] Table 1 Backbone network VGG network layer input / output

[0039] 2) Codec layer:

[0040] Specifically, when the present invention designs the codec layer, there are two design ideas, the first is a single-scale U-type codec method; the second is an FPN connection method between different scales. First, when designing a single-scale U-type codec method, the designed U-type codec network is compared with the traditional U-Net network structure. The U-type codec network discards Skip-Connections and uses dilation convolution (Dilation Conv) instead of Conv to reduce the depth of the network. The traditional U-Net network uses Skip-Connections to increase the global context information, taking into account both shallow and deep features, but the cost is that the amount of data in the network is constantly increasing. Although Skip-Connections are discarded here, in order to ensure that the global context information is not lost, the present invention uses Dilation Conv instead of Conv, and increases the range of the receptive field by setting a gradually increasing dilation coefficient, so as to achieve the effect of increasing the global context information.

[0041] Then there is the FPN connection method between different scales. The FPN structure is mainly used for multi-scale target detection. The size of human targets varies greatly. With the increase of network depth and multiple downsampling operations, the information of small targets is seriously lost, and it is impossible to accurately distinguish them at the pixel level, which easily leads to false detection and missed detection. The FPN structure is top-down, and deep features flow to the shallow network. It can take into account both deep features and shallow features, making up for the defect of small target information loss and greatly avoiding false detection and missed detection.

[0042] 3) Fusion layer:

[0043] Specifically, the fusion of the solution of the present invention is based on the design concept of the HED network, combined with a specific loss function to predict accurate edge information. In the U-type codec network, the input and output sizes of the codec layer of a single scale are consistent, while the output size of the codec layer of each scale is different from the size of the input image. The reason is the 5 MaxPool (2*2) operations in the feature extraction layer. Therefore, for output fusion, an upsampling operation must be performed first to unify all output sizes to the size of the input image. Assume that the input image is I and the width is W. input , height H input After feature extraction and encoding / decoding layer operations, the output results are O i , i∈[1,5], the corresponding sizes are as follows:

[0044]

[0045] Among them, W i , H iare the width and height of the codec layer output respectively. Therefore, the upsampling rate of each layer output is 2 i , i∈[1,5], the output tensor size after upsampling is [1,W input ,H input ].

[0046] Then use the Concat operation to merge all upsampled features into a [5,W input ,H input ] tensor. Then a 1*1Conv kernel is used for dimensionality reduction, and finally Sigmoid is used as the activation function to output the final detection result.

[0047] The loss function of the solution of the present invention adopts the weighted linear sum form of multi-layer mixed loss function, and the formula is:

[0048]

[0049] Among them, l k is the loss of the kth output result; K represents the number of outputs; α k is the weight of each loss function. In this network, set α k =1 / K, K=5, including the output results of the encoding and decoding layer and the fusion results. In the scheme of the present invention, binary cross entropy is used as the loss function of a single layer, and the formula is:

[0050]

[0051] in, Corresponding to Pixel-level, is the commonly used binary cross entropy, the formula is:

[0052]

[0053] Among them, G(x,y)∈{0,1} indicates whether the pixel at position (x,y) is the Ground Truth Label (GTLabel), and S(x,y) indicates the probability of predicting that the pixel at position (x,y) is the detected object. bce By calculating the binary classification entropy of each pixel, the probability of distinguishing the foreground and the background is obtained. Based on this, the loss function of the scheme of the present invention adopts the weighted linear sum form of the multi-layer mixed loss function and can also be expressed as:

[0054]

[0055] Among them, the loss function adjusts the multiple output results in the form of linear combination, which ensures the accuracy of the fused results on the boundary to the greatest extent.

[0056] Through the above-designed salient human detection deep network, the processed thermal infrared image containing human targets is used as input data to train a salient human target detection model to obtain a salient human target detection model.

[0057] Step S20: obtaining the infrared image to be processed, and importing the infrared image to be processed as input data into the salient human target detection model, performing model training, and outputting the training result.

[0058] Specifically, once a significant human target detection model is obtained, subsequent infrared image detection can be performed based on this model to identify whether there is a human body in the image, and if there is a human body, locate the position of the human body and highlight the human body outline. The acquisition unit 10 acquires the infrared image to be processed through the infrared image acquisition module, and then transmits the acquired infrared image to the training unit 30. The training unit 30 uses the acquired infrared image as input data of the significant human target detection model, performs infrared image analysis based on the model process, until the model training is completed, and outputs the training results.

[0059] Step S30: Identify human targets according to the training results.

[0060] Specifically, the training results are output by the significant human target detection model, and the output unit determines whether there is a human body in the result and the outline of the actual position of the human body based on the training results, and then outputs the recognition result of outlining the human body outline to the monitoring end for relevant personnel to view.

[0061] In the embodiment of the present invention, the human targets in the thermal infrared images are detected by the designed HED-like thermal infrared saliency human target network. The human targets in the infrared images can be clearly detected by the network, and can be clearly detected in the case of occlusion and targets of different sizes. The invention can be applied to fire rescue, military target detection and other fields.

[0062] In a possible implementation, the thermal infrared image significant human target detection method and system proposed in the scheme of the present invention are used for firefighting and rescue. The acquisition unit 10 performs indiscriminate infrared image acquisition at the firefighting and rescue acquisition site, and the image acquisition covers various positions and angles of the rescue site. Then these data are automatically imported into the processing unit 20. The processing unit 20 performs preset preprocessing of the image, and then sends the preprocessed infrared image to the training unit 30. The training unit 30 performs human recognition based on the preset significant human target detection model. If a human body is recognized in a certain infrared image, the output unit locates the serial number of the infrared image in which the human body exists, and collects the acquisition angle of the acquisition position of the image in parallel. While outputting the human body image information, an alarm message is triggered. Then the infrared image that triggers the alarm message is annotated and processed, and independently pushed to the monitoring point of each rescuer, reminding the rescuer to go to the located image acquisition position to rescue people, thereby improving the rescue efficiency at the firefighting and rescue site.

[0063] An embodiment of the present invention further provides a computer-readable storage medium, on which instructions are stored, which, when executed on a computer, enable the computer to execute the above-mentioned thermal infrared image significant human target detection method.

[0064] Those skilled in the art will understand that all or part of the steps in the method for implementing the above-mentioned embodiments can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including several instructions for making a single-chip microcomputer, a chip or a processor (processor) perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0065] The optional embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above embodiments. Within the technical concept of the embodiments of the present invention, the technical scheme of the embodiments of the present invention can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the embodiments of the present invention. It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the embodiments of the present invention will not further describe various possible combinations.

[0066] In addition, various embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the embodiments of the present invention, they should also be regarded as the contents disclosed in the embodiments of the present invention.

Claims

1. A method for detecting significant human targets in thermal infrared images, characterized in that: The method comprises: In the pre-built salient human detection deep network, the processed historical images are used to train the salient human target detection model to obtain the salient human target detection model; wherein the processed historical images are generated by processing the historical information of thermal infrared images containing human targets based on a preset data enhancement algorithm; wherein, The pre-built salient human body detection deep network is built based on the HED edge detection model; The pre-built salient human body detection deep network includes: a backbone network, a codec layer and a fusion layer; The backbone network construction rules are: Delete the fully connected layer of the VGG network; In the VGG network with the fully connected layer deleted, multiple convolutions are added to obtain the backbone network; The codec layer is a single-scale U-type codec or a FPN connection codec between different scales; The fusion layer is constructed based on the HED model fusion preset loss function; the fusion layer includes upsampling operation and Concat operation; the calculation formula of the upsampling operation is: in, is the width of the input image; is the height of the input image; , The width and height of the image output by the codec layer respectively; is the upsampling rate of each layer output; The preset loss function expression is: in, is the loss of the kth output result; K represents the number of output results; is the weight for each loss function; is the binary cross entropy; Acquire an infrared image to be processed; Importing the infrared image to be processed as input data into the salient human target detection model, performing model training, and outputting training results; Recognize human targets based on training results.

2. The method according to claim 1, characterized in that The preset data enhancement algorithm includes at least: color transformation, geometric transformation and clipping operations.

3. A thermal infrared image salient human target detection system, characterized in that: The system comprises: An acquisition unit, used for acquiring historical information of thermal infrared images containing human targets, and for acquiring infrared images to be processed; The processing unit is used to train a significant human target detection model using the processed historical images in a pre-built significant human target detection deep network to obtain a significant human target detection model; wherein, The pre-built salient human body detection deep network is built based on the HED edge detection model; The pre-built salient human body detection deep network includes: a backbone network, a codec layer and a fusion layer; The backbone network construction rules are: Delete the fully connected layer of the VGG network; In the VGG network with the fully connected layer deleted, multiple convolutions are added to obtain the backbone network; The codec layer is a single-scale U-type codec or a FPN connection codec between different scales; The fusion layer is constructed based on the HED model fusion preset loss function; the fusion layer includes upsampling operation and Concat operation; the calculation formula of the upsampling operation is: in, is the width of the input image; is the height of the input image; , The width and height of the image output by the codec layer respectively; is the upsampling rate of each layer output; The preset loss function expression is: in, is the loss of the kth output result; K represents the number of output results; is the weight for each loss function; is the binary cross entropy; A training unit, used for importing the infrared image to be processed as input data into the salient human target detection model, performing model training, and outputting a training result; The recognition unit is used to recognize human targets according to the training results.

4. A computer-readable storage medium having instructions stored thereon, which, when executed on a computer, enables the computer to execute the thermal infrared image salient human target detection method as described in any one of claims 1 or 2.

Images