A passive terahertz image fixed background human automatic tracking method

Abstract

The application discloses a kind of human automatic tracking methods under fixed background of passive terahertz image, its technical key points are in, design a kind of for passive terahertz image in figure and background carries out semantic segmentation, completes the automatic tracking of whole process of human motion under fixed background Multi-feature fusion neural network model, the model is based on the semantic segmentation network of encoder-decoder structure, on the basis thereof adds attention pyramid module and multi-scale filter fusion module.Firstly, the passive terahertz image of input semantic segmentation network is carried out multi-scale filtering and then enters the encoder, the feature image after encoder is sequentially input into attention pyramid module and multi-scale filter fusion module, finally, the feature image is input into the decoder of semantic segmentation network, and the semantic segmentation result graph is output, realizes the semantic segmentation of figure and background in real-time acquisition passive terahertz image, completes the automatic tracking of whole process of human motion under fixed background.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of passive terahertz image processing, and particularly relates to a human automatic tracking method in a fixed background of a passive terahertz image. BACKGROUND

[0002] Terahertz waves refer to electromagnetic waves between microwaves and light waves, with a frequency of 0.1-10 THz. Therefore, terahertz has the following competitive advantages. Terahertz has good penetration for most non-metallic and non-polar materials, thereby enabling it to realize perspective imaging of opaque objects. The radiation energy of terahertz is maintained at the level of millielectron volts, which is much lower than the kiloelectron volt energy generated by X-rays. Therefore, terahertz radiation is almost unable to penetrate human skin or produce harmful ionization reactions, thereby ensuring its highly safe application to humans and other living beings. The terahertz spectrum of a substance contains rich physical and chemical information, which is unique to each other. The above three performance advantages of terahertz waves make it particularly suitable for security inspection.

[0003] Security inspection should be the most attractive application of terahertz at the present stage, because X-ray systems can detect objects hidden in ordinary clothes through extremely high radiation levels that are harmful to the human body. Therefore, X-ray systems are only used to detect hand-held items. In contrast, terahertz imaging technology not only can classify hidden objects, but also can ensure the safe use of the human body. Generally, terahertz imaging systems work in active mode or passive mode. Active terahertz imaging requires a special terahertz transmitter to project terahertz waves onto the human body and collect echoes to achieve imaging. Although the power of terahertz waves is moderate and does not harm health, the terahertz transmitter increases the cost and size of the system. In contrast, passive terahertz imaging directly detects terahertz waves generated by blackbody radiation, so it is completely harmless, and the security instrument is also smaller and more concealed. However, due to the weak natural terahertz radiation and the accompanying noise interference, the quality of passive terahertz images is very low, and passive terahertz images are prone to fixed background bands, thereby hindering the concealment of objects and accurate identification.

[0004] Based on the above background conditions and technical problems, there is an urgent need for a passive terahertz image processing method with high segmentation degree training learning and adaptive subtraction of fixed background. SUMMARY

[0005] To solve the above problems, the application provides a human automatic tracking method in a passive terahertz image fixed background, which has the advantages that a multi-feature fusion semantic segmentation network is constructed, the network comprises an attention pyramid module and a multi-scale filter fusion module, different noise scales and different clarity passive terahertz images can be efficiently processed, the passive terahertz image is effectively segmented, and the automatic tracking of the whole human motion process in the fixed background is completed.

[0006] To achieve the above technical purposes, the technical scheme adopted by the application is as follows:

[0007] The human automatic tracking method in a passive terahertz image fixed background comprises the following steps:

[0008] In some embodiments, the human automatic tracking method in a passive terahertz image fixed background comprises the following steps:

[0009] S1: performing multi-scale filtering on the passive terahertz image input into the semantic segmentation network;

[0010] S2: inputting the image filtered in S1 into the encoder of the semantic segmentation network;

[0011] S3: inputting the feature image of the encoder in S2 into the attention pyramid module;

[0012] S4: inputting the feature image of the attention pyramid module in S3 into the multi-scale filter fusion module;

[0013] S5: inputting the feature image of the multi-scale filter fusion module in S3 into the decoder of the semantic segmentation network, and finally outputting a semantic segmentation result image, so as to realize the semantic segmentation of the characters and the background in the real-time collected passive terahertz image and complete the automatic tracking of the whole human motion process in the fixed background.

[0014] In some embodiments, the attention pyramid module is formed by adding an attention module to each layer of the feature image of the spatial pyramid module in the semantic segmentation network.

[0015] The original hollow space pyramid was:

[0016] E ASPP (I)=C[W1(I),W2(I),W3(I),W4(I),W5(I)]

[0017] Where I represents the input feature image, and W1, W2, W3, W4, W5 represent feature extraction at different scales in the hollow spatial pyramid;

[0018] The attention pyramid is:

[0019] E PA (I)=C[W CBAM (W1(I′)),W CBAM (W2(I)),W CBAM (W3(I)),W CBAM (W4(I)),W CBAM (W5(I))]

[0020] Among them W CBAM This represents the convolution operation of the attention module.

[0021] In some embodiments, the multi-scale filtering fusion module is used to perform multi-scale filtering on the passive terahertz image data I input to the neural network as {I,F(I,θ1),F(I,θ2),…,F(I,θ)}. n )}, where F represents the filter function and θ represents different filter scales;

[0022] The encoder feature extraction of the original semantic segmentation network is as follows:

[0023] E(I)=W(I)

[0024] Where W represents the feature extraction in the encoder, and its feature image parameters change from (b,3,h,w) to (b,c,h′,w′), where b is the batch size, 3 represents the RGB three channels of the image, h and h′ represent the lengths of the feature images before and after the change, respectively, and w and w′ represent the widths of the feature images before and after the change, respectively.

[0025] The feature extraction part of the multi-scale filtering and fusion module is as follows:

[0026] E MSF (I)=W′{C[E PA (W(I)),E PA (W(F(I,θ1))),E PA (W(F(I,θ2))),…,E PA (W(F(I,θ n)))]}

[0027] Where C represents the concatenation and fusion operation in the convolutional neural network, and W′ represents the convolution operation with a kernel size of 1×1. This operation adjusts the number of channels and convolves the concatenated feature images of multiple filters into a single convolutional image. The feature image parameters are (b,3,h,w) after feature extraction and concatenation and fusion, becoming (b,n×c,h′,w′), and then (b,c,h′,w′) after a 1×1 convolution operation.

[0028] The beneficial effects of this invention are:

[0029] 1. A multi-feature fusion convolutional neural network for semantic segmentation of passive terahertz images was constructed, proposing an attention pyramid module and a multi-scale filtering fusion module. The attention pyramid module ensures that each layer (i.e., each scale) of the hollow spatial pyramid receives channel and spatial attention, highlighting useful information and ignoring invalid information in the feature image, which aids in feature extraction and enriches the edge details of the passive terahertz image, resulting in smoother segmentation edges. The multi-scale filtering fusion module extracts features from passive terahertz images at different filtering scales, and then automatically selects appropriate features through 1×1 convolution, thereby achieving accurate semantic segmentation of passive terahertz images with varying noise levels.

[0030] 2. Effectively segment people and backgrounds in passive terahertz images, and automatically track the entire process of human movement against a fixed background. Attached Figure Description

[0031] Figure 1 This is a diagram of the semantic segmentation network structure of the present invention;

[0032] Figure 2 This is a comparison chart of the semantic segmentation results of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0035] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0036] Example:

[0037] The acquired passive terahertz image data I is labeled. In this step, labels are manually created using the labelme software. After labeling, the data is saved, generating label data. The label data is then mapped one-to-one with the images to generate {I, label}, which serves as the training set, validation set, and test set for the learning network model. In this embodiment, the training set, validation set, and test set are divided into three portions with ratios of 60%, 20%, and 20%, respectively.

[0038] A semantic segmentation network model for passive terahertz images was constructed. Based on the Deeplabv3+ encoder-decoder structure, an attention pyramid module and a multi-scale filtering fusion module were added. The specific network structure diagram is shown below. Figure 1 As shown, the attention pyramid module and the multi-scale filtering fusion module are specifically as follows:

[0039] The Attention Pyramid Module adds an Attention Module (CBAM) to each layer of feature images in the Atrous Spatial Pyramid Pooling (ASPP) module of the Deeplabv3+ semantic segmentation network.

[0040] The original hollow space pyramid was:

[0041] E ASPP (I)=C[W1(I),W2(I),W3(I),W4(I),W5(I)]

[0042] Where I′ represents the input feature image, and W1, W2, W3, W4, W5 represent feature extraction at different scales in the hollow spatial pyramid.

[0043] The attention pyramid is:

[0044] E PA (I)=C[W CBAM (W1(I)),W CBAM (W2(I)),W CBAM (W3(I)),WCBAM (W4(I)),W CBAM (W5(I))]

[0045] Among them W CBAM This represents the convolution operation of the attention module.

[0046] The multi-scale filtering and fusion module inputs the passive terahertz image data I into the neural network and performs three-scale filtering on the original image to generate {I,F(I,θ1),F(I,θ2),F(I,θ3)}, where F represents the filtering function for the passive terahertz image and θ represents different filtering scales.

[0047] The original semantic segmentation network, after feature extraction, is as follows:

[0048] E(I)=W(I)

[0049] Here, W represents the convolution operation in the encoder, where the feature image parameters change from (4,3,512,512) to (4,320,32,32) and (4,24,128,128), where 4 is the batch size and 3 represents the RGB channels of the image. The original input image to the network has dimensions of (512,512), which are downsampled by 16 times by the backbone encoder to (32,32), resulting in 320 channels. The feature image size of the shallow layer is (128,128), with 24 channels.

[0050] The feature extraction part of the multi-scale filtering and fusion module is as follows:

[0051] E MSF (I)=W′{C[E PA (W(I)),E PA (W(F(I,θ1))),E PA (W(F(I,θ2))),E PA (W(F(I,θ3)))]}

[0052] Here, C represents the concatenation and fusion operation in the convolutional neural network, and W′ represents a convolution operation with a kernel size of 1×1. This operation adjusts the number of channels, convolves the concatenated feature images from multiple filters into a single convolutional image. The parameters of the backbone feature image change from (4,3,512,512) to (4,4×320,32,32) after feature extraction and concatenation, and then to (4,320,32,32) after a 1×1 convolution operation. The parameters of the shallow feature image change from (4,3,512,512) to (4,4×24,128,128) after feature extraction and concatenation, and then to (4,24,128,128) after a 1×1 convolution operation.

[0053] The semantic segmentation network constructed in the embodiment is trained using the training set in the dataset. The network is trained for 100 epochs using the training set in the dataset, and cross-validation is performed using the training set and validation set at the same time. The training parameters are adjusted, and the above process is repeated until the model achieves good results on the test set.

[0054] A trained convolutional neural network model is used to perform semantic segmentation on the image to be segmented. The trained convolutional neural network model described in S3 is used to perform semantic segmentation on people and backgrounds in real-time acquired passive terahertz images, completing the automatic tracking of the entire process of human movement against a fixed background. The semantic segmentation result is shown in the figure below. Figure 2 As shown.

[0055] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A passive terahertz image automatic human tracking method in a fixed background, characterized in that: The application discloses a multi-feature fusion neural network model of a semantic segmentation network based on an encoder-decoder structure, which is used for semantic segmentation of a person and a background in a passive terahertz image and automatic tracking of a whole human motion process in a fixed background. The attention pyramid module is formed by adding an attention module to each layer of feature images of a spatial pyramid module in the semantic segmentation network. The original spatial pyramid is as follows: E ASPP (I) = C[W1(I), W2(I), W3(I), W4(I), W5(I)] wherein I represents input feature images, W1, W2, W3, W4 and W5 represent feature extraction of different scales in the spatial pyramid. The attention pyramid is as follows: E PA (I) = C[W CBAM (W1(I′)),W CBAM (W2(I)),W CBAM (W3(I)),W CBAM (W4(I)),W CBAM (W5(I))] wherein W CBAM represents a convolution operation of the attention module; The multi-scale filtering fusion module is used for multi-scale filtering of passive terahertz image data I of an input neural network into {I, F(I, θ1), F(I, θ2), …, F(I, θ n )}, wherein F represents a filtering function, and θ represents different filtering scales. Feature extraction of the original semantic segmentation network encoder is as follows: E(I)=W(I) wherein W represents feature extraction in the encoder, the feature image parameters of which are changed from (b, 3, h, w) to (b, c, h', w'), wherein b is a batch-size, 3 represents RGB three channels of the image, h and h' represent the length of the feature image before and after change respectively, and w and w' represent the width of the feature image before and after change respectively, Feature extraction of the multi-scale filter fusion module is as follows: E MSF (I) = W'(C[E PA (W(I)),E PA (W(F(I,θ1))),E PA (W(F(I,θ2))),…,E PA (W(F(I,θ n )))]} wherein C represents a splicing fusion operation in a convolutional neural network, W' represents a convolution operation with a convolution kernel size of 1×1, the feature image after splicing fusion of the multi-filter fusion is convolved into one image after the size of the convolution, the feature image parameters of which are changed from (b, 3, h, w) to (b, n×c, h', w') after feature extraction and splicing fusion, and then to (b, c, h', w') after the 1×1 convolution operation.

2. The method according to claim 1, wherein the method is characterized in that: The specific tracking method comprises the following steps: S1: performing multi-scale filtering on the passive terahertz image input into the semantic segmentation network; S2: inputting the image after the multi-scale filtering in S1 into the encoder of the semantic segmentation network; S3: inputting the feature image after the encoder in S2 into the attention pyramid module; S4: inputting the feature image after the attention pyramid module in S3 into the multi-scale filter fusion module; S5: inputting the feature image after the multi-scale filter fusion module in S3 into the decoder of the semantic segmentation network, and finally outputting a semantic segmentation result image, so as to realize semantic segmentation of a person and a background in a real-time collected passive terahertz image and complete automatic tracking of a whole human motion process in a fixed background.

Images