A terahertz imaging processing method based on time series convolutional neural network

Abstract

The invention relates to the fields of terahertz imaging and artificial intelligence, and is a terahertz imaging processing method based on a time series convolutional neural network. The method comprises the following steps: using a terahertz imaging device and a visible light imaging device at the same position and angle to simultaneously photograph and collect scenes in the same range to form a terahertz image and a visible light image; performing median filtering on the terahertz image to remove salt and pepper noise in the terahertz image; stacking the processed terahertz image below the visible light image; inputting a sample image and an initial noise feature as inputs into a convolutional neural network based on a time series; calculating a target recognition frame in the sample image in combination with a priori frame Anchor and a position offset, and calculating a target position and category information in the sample image in combination with the target recognition frame and a category confidence.

Description

Technical Field

[0001] The present invention relates to the fields of terahertz imaging and artificial intelligence, and in particular to a terahertz imaging processing method based on a time series convolutional neural network. Background Art

[0002] Due to the characteristics of terahertz waves such as high spatiotemporal coherence, low photon energy, good directionality and wide spectrum, they have developed rapidly in recent years. Terahertz linear scanners using terahertz imaging technology have been initially applied in the fields of security and medical treatment.

[0003] However, the original grayscale image collected by the terahertz passive imaging system is not conducive to the intuitive display of imaging due to the influence of temperature drift of the front-end device, internal noise of the circuit, and stripe noise formed by the linear scanning of the servo turntable, so image enhancement processing is required. Traditional image enhancement methods are slow and have poor robustness. At present, convolutional neural networks have surpassed traditional algorithms in many fields. Therefore, it is necessary to try to use convolutional neural networks to enhance terahertz imaging. Summary of the invention

[0004] In order to solve the deficiencies in the prior art, the present invention provides a terahertz imaging processing method based on a time series convolutional neural network, which can reduce the processing time of terahertz imaging, improve imaging quality, and has good robustness.

[0005] In order to achieve the above objectives, the present invention adopts the following technical solution: a terahertz imaging processing method based on a time series convolutional neural network, comprising the following steps:

[0006] Step 1: Use a terahertz imaging device and a visible light imaging device to take photos of the same range of scenes at the same position and angle to form a terahertz image and a visible light image, wherein the visible light image is a three-channel RGB or BGR image, the visible light image is W1 pixels wide and H1 pixels high, and the terahertz image is a single-channel grayscale image, the terahertz image is W2 pixels wide and H2 pixels high;

[0007] Step 2: Perform median filtering on the terahertz image to remove salt and pepper noise in the terahertz image;

[0008] Step 3: Use bilinear interpolation to scale the width W2 and height H2 of the terahertz image processed in step 2 to a width W1 and a height H1, and then overlay the terahertz image under the visible light image so that the positions of the same content scenes on the terahertz image and the visible light image are aligned vertically to form a superimposed four-channel sample image, where the sample image is W1 pixels wide and H1 pixels high;

[0009] Step 4: Use the sample image and the initial noise feature as inputs to a time series-based convolutional neural network. The initial noise feature is a three-dimensional array with a width of W1 / 4, a height of H1 / 4, and 8 channels, and all values of the initial noise feature are 0. The time series-based convolutional neural network outputs the class confidence, position offset, and cyclic noise feature of the sample image. The size of the cyclic noise feature is the same as the width, height, and number of channels of the initial noise feature.

[0010] Step 5: Calculate the target recognition box in the sample image by combining the prior box Anchor and the position offset, and calculate the target position and class information in the sample image by combining the target recognition box and the class confidence.

[0011] In Step 2, when performing median filtering on the terahertz image, set the sampling window as a rectangle of M×M, where M is an odd number greater than 1. Align the center of the sampling window with the terahertz image, and slide the window pixel by pixel from left to right and from top to bottom. After each slide, set the pixel value at the center of the window as the median of all pixel values within the M×M rectangle.

[0012] In Step 3, use bilinear interpolation to scale the width W2 and height H2 of the terahertz image processed in Step 2 to width W1 and height H1. That is, the size of the terahertz image processed in Step 2 is W2×H2, and the size of the scaled terahertz image is W1×H1. For each pixel point P1=(x,y) in the scaled terahertz image, the corresponding coordinates on the terahertz image processed in Step 2 are P2=(x*W2 / W1,y*H2 / H1). Obviously, this corresponding coordinate is generally not an integer. There are four pixel points Q 11 =(x1,y1), Q 12 =(x1,y2), Q 21 =(x2,y1), and Q 22 =(x2,y2) around the pixel point P2 on the terahertz image processed in Step 2. The values of these four pixel points are f(Q 11 ), f(Q 12 ), f(Q 21 ), f(Q 22 ). Taking the lower left corner of the image as the origin, x, x1, x2 as the horizontal coordinate values of the image, and y, y1, y2 as the vertical coordinate values of the image, where 0≤x1<x*W2 / W1<x2≤W2, 0≤y1<y*H2 / H1<y2≤H2, 0≤x≤W1, 0≤y≤H1, the value of the corresponding pixel point P1 on the scaled terahertz image

[0013]

[0014] f(P1) is rounded to an integer.

[0015] In step 4, the main body of the time series convolutional neural network is divided into a ShuffleNetV2 network, which includes a convolution layer Conv, a maximum pooling layer MaxPool and three network blocks Stage2, Stage3 and Stage4 composed of the basic structure of ShuffleNetV2. The input of the main body is the sample image, and the output is connected to the detection head and the time series part respectively. The detection head part is divided into two branches, each of which is composed of a convolution layer and a ReLU activation function. The two branches output the category confidence and position offset of the sample image respectively. The time series part includes three convolution layers. In addition to the output of the main body, the input of the time series part also has the initial noise feature, and the output is the cyclic noise feature.

[0016] In step 5, the target recognition frame in the sample image is calculated by combining the prior frame Anchor and the position offset, where the prior frame Anchor is a predefined n rectangular frame, n is a positive integer, and each rectangular frame is denoted by A i ={cx i , cy i , w i ,h i},0≤i≤n,cx i is the x coordinate of the center point of the prior box, cy i is the y coordinate of the center point of the prior box, w i is the width of the prior box, h i is the height of the prior frame. The size of the prior frame is preset according to the size of the object in the sample image to be processed. n is 4 or 5. The convolutional neural network based on time series outputs a total of m position offsets, where m is a positive integer. For each position offset O output j ={offset_x j , offset_y j , offset_w j , offset_h j}, 0≤j≤m, offset_x j is the offset of the x coordinate of the center point of the prior box, offset_y j is the offset of the y coordinate of the center point of the prior box, offset_w j is the offset of the prior box width, offset_h j is the offset of the height of the prior frame, and finally m*n target recognition frames are obtained, each of which is R k ={real_x k , real_y k , real_w k , real_h k}, 0≤k≤m*n, where:

[0017]

[0018]

[0019]

[0020] .

[0021] In step 5, the target position and category information in the sample image are calculated by combining the target recognition frame and category confidence. There are m*n target recognition frames, and the category confidences output by the convolutional neural network based on time series are m*n*c. Each target recognition frame corresponds to c category confidences. The category recognized by each target recognition frame is the category with the highest confidence among the c category confidences corresponding to it. The confidence of each target recognition frame is the confidence of the category it recognizes. The category confidence threshold is set to Ct. Only when the confidence of the target recognition frame is greater than Ct can the target recognition frame be considered credible. Then the credible target recognition frame is non-maximum. The NMS operation is used to remove target recognition frames with too high overlap. The remaining target recognition frames and their corresponding categories are the final identified target positions and categories. The specific operation of non-maximum suppression is to sort the confidence of all target recognition frames, select the target recognition frame with the highest confidence, and traverse the remaining target recognition frames. If the overlapping area with the current target recognition frame with the highest confidence is greater than the threshold Nt, the traversed target recognition frame will be deleted. After the traversal is completed, the target recognition frame with the highest confidence is processed, and the one with the highest confidence is selected from the unprocessed target recognition frames. Repeat the above process until all target recognition frames are processed.

[0022] In the convolutional neural network based on time series, the initial noise feature is input for the first time, and the cyclic noise feature of the previous output is used to replace the initial noise feature when input again thereafter, and is input together with the new sample image. This allows the convolutional neural network based on time series to learn the complete noise features in the time series and improve the quality of terahertz imaging, wherein the new sample image is an image acquired at the same position and angle at the next time point using the same terahertz imaging device and the same visible light imaging device through steps one, two, and three.

[0023] Beneficial effects achieved by the invention: The present invention applies a time series-based convolutional neural network to terahertz imaging enhancement processing. The use of a convolutional neural network reduces the terahertz imaging processing time. The use of a time series can effectively suppress the interference of terahertz imaging noise on terahertz imaging and improve robustness. The input sample image combines visible light information and terahertz information, thereby improving the accuracy of detecting targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of median filtering;

[0025] Figure 2 It is a schematic diagram of a four-channel sample picture;

[0026] Figure 3 It is a schematic diagram of the convolutional neural network structure. DETAILED DESCRIPTION

[0027] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0028] A terahertz imaging processing method based on a time series convolutional neural network comprises the following steps:

[0029] Step 1: Use a terahertz imaging device and an ordinary camera (visible light imaging device) to simultaneously collect terahertz imaging and visible light imaging at the same position and angle (the best way is to use a device that has both terahertz imaging and visible light imaging), and store them as terahertz images and visible light images. The size of the visible light image is 3x 500*240, that is, 500 pixels high and 240 pixels wide, which is RGB three channels. The terahertz image is 1x 400*50, that is, 400 pixels high and 50 pixels wide, which is a grayscale single channel.

[0030] Step 2: Perform median filtering on the terahertz image. The sampling window is a rectangle of 3 pixels × 3 pixels. The median filtering process is as follows: Figure 1 As shown in the figure, taking a picture with a length of 5 pixels and a width of 5 pixels as an example, the thick black frame in the figure represents the sampling window, and the black bottom frame represents the center of the sampling window. The center of the sampling window is aligned with the pixel of terahertz imaging, and the window is slid from left to right and from top to bottom on the pixel of terahertz imaging. After each slide, the pixel value of the center of the window is set to the median value of all pixel values ​​in the 3×3 rectangle. Figure 1 The pixel values ​​in the middle neighborhood are sorted as [56, 66, 90, 91, 93, 94, 95, 97, 101], and the median is 93, so the original pixel value 56 of the center point is replaced by 93;

[0031] Step 3: Use bilinear interpolation to interpolate the terahertz image from 1x 400x50 to 1x 500x240. For the value f(P1) of each pixel point P1=(x,y) of the new terahertz image, we know that the original terahertz image is around the pixel point P2=(x*50 / 240,y*400 / 500)Q 11 =(x1,y1),Q12=(x1,y2),Q 21=(x2, y1) and Q 22 =(x2, y2) the values of the four pixel points f(Q 11 ), f(Q 12 ), f(Q 21 ), f(Q 22 ), with the lower left corner of the picture as the origin, x, x1, x2 as the horizontal coordinate points of the picture, y, y1, y2 as the vertical coordinate points of the picture, 0 ≤ x1 < x * 50 / 240 < x2 ≤ 50, 0 ≤ y1 < y * 400 / 500 < y2 ≤ 400, 0 ≤ x ≤ 240, 0 ≤ y ≤ 50, we get:

[0032]

[0033] Then stack the terahertz picture behind the visible light picture to form a four-channel sample picture (when stacking, ensure that the same targets of the two pictures are completely aligned up and down). The size is 4x500x240, as Figure 2 shown, the white rectangle represents the visible light picture, and the black striped rectangle represents the terahertz imaging picture;

[0034] Step 4: Transmit the sample picture and an initial noise feature as inputs to a time series-based convolutional neural network. The initial noise feature is a three-dimensional array with a width of 60, a height of 125, and 8 channels. The values of the initial noise feature are all 0. The time series-based convolutional neural network outputs the class confidence, position offset of the sample picture, and a cyclic noise feature. The size of the cyclic noise feature is the same as that of the initial noise feature. The structure of the time series convolutional neural network is as Figure 3 shown. The backbone part of the time series convolutional neural network is the ShuffleNetV2 network, which consists of a convolutional layer (Conv), a max pooling layer (MaxPool), and three network blocks (Stage2, Stage3, Stage4) composed of the basic structure of ShuffleNetV2 in series. The input of the backbone part is the sample picture, and the outputs are respectively connected to the detection head part and the time series part. The detection head part is divided into two branches, each branch consists of a convolutional layer and a ReLU activation function, and the two branches respectively output the class confidence and position offset of the sample picture. The time series part includes three convolutional layers. The input of the time series part includes not only the output of the backbone part but also the initial noise feature, and the output is the cyclic noise feature;

[0035] Step 5: Combining the prior box and the position offset, the target recognition box in the sample picture can be calculated. The prior box is 4 predefined rectangular boxes, and each rectangular box is denoted as A i ={cx i , cy i , wi , h i},0≤i≤4,cx i is the x coordinate of the center point of the prior box, cy i is the y coordinate of the center point of the prior box, w i is the width of the prior box, h i is the height of the prior frame. The convolutional neural network based on time series outputs a total of 10 position offsets. For each position offset O j ={offset_x j ,offset_y j , offset_w j , offset_h j}, 0≤j≤10, offset_x j is the offset of the x coordinate of the center point of the prior box, offset_y j is the offset of the y coordinate of the center point of the prior box, offset_w j is the offset of the prior box width, offset_h j is the offset of the height of the prior frame, and finally 40 target recognition frames are obtained, each of which is R k ={real_x k , real_y k , real_w k , real_h k}, 0≤k≤40, where:

[0036]

[0037] Then, the target position and category information in the sample image can be calculated by combining the target recognition frame and category confidence. There are 40 target recognition frames and 120 category confidences output by the time series-based convolutional neural network. Each target recognition frame corresponds to 3 category confidences. The category recognized by each target recognition frame is the category with the highest confidence among the 3 corresponding category confidences. The confidence of each target recognition frame is the confidence of the category it recognizes. The category confidence threshold is set to 0.5. Only when the confidence of the target recognition frame is greater than 0.5 can the target recognition frame be considered credible. Then, the credible target recognition frame is subjected to non-maximum suppression (NMS) operation to remove the target recognition frames with high overlap. The remaining target recognition frames and their corresponding categories are the final identified target positions and categories. The specific operation of non-maximum suppression is to sort the confidences of all target recognition frames, select the target recognition frame with the highest confidence, traverse the remaining target recognition frames, and if the overlapping area with the current target recognition frame with the highest confidence is greater than the threshold 0.3, delete the traversed target recognition frame. After the traversal is completed, the target recognition frame with the highest confidence is processed, and the target recognition frame with the highest confidence is selected from the unprocessed target recognition frames. Repeat the above process until all target recognition frames are processed.

[0038] It is only necessary to input the initial noise feature when using the time series-based convolutional neural network for the first time. When using the time series-based convolutional neural network again, it is only necessary to replace the initial noise feature with the cyclic noise feature outputted last time and input it together with the new sample image, such as Figure 3 As shown by the dotted line, the new sample image is obtained by steps one, two and three using the same terahertz imaging device and the same ordinary camera at the same position and angle at the next time point.

[0039] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A terahertz imaging processing method based on time series convolutional neural network, characterized in that: The steps include: Step 1: Use a terahertz imaging device and a visible light imaging device to take photos of the same range of scenes at the same position and angle to form a terahertz image and a visible light image, wherein the visible light image is a three-channel RGB or BGR image, the visible light image is W1 pixels wide and H1 pixels high, and the terahertz image is a single-channel grayscale image, the terahertz image is W2 pixels wide and H2 pixels high; Step 2: Perform median filtering on the terahertz image to remove salt and pepper noise in the terahertz image; Step 3: Use bilinear interpolation to scale the width W2 and height H2 of the terahertz image processed in step 2 to a width W1 and a height H1, and then overlay the terahertz image under the visible light image so that the positions of the same content scenes on the terahertz image and the visible light image are aligned vertically to form a superimposed four-channel sample image, where the sample image is W1 pixels wide and H1 pixels high; Step 4: Input the sample image and initial noise features as input to the time series-based convolutional neural network. The initial noise feature is a three-dimensional array with a width of W1 / 4, a height of H1 / 4, and 8 channels. The values ​​of the initial noise feature are all 0. The time series-based convolutional neural network outputs the category confidence, position offset, and cyclic noise features of the sample image. The size of the cyclic noise feature is the same as the width, height, and number of channels of the initial noise feature. Step 5: Combine the prior box Anchor and the position offset to calculate the target recognition box in the sample image, and combine the target recognition box and category confidence to calculate the target position and category information in the sample image.

2. The terahertz imaging processing method based on time series convolutional neural network according to claim 1, characterized in that: In step 2, when the terahertz image is subjected to median filtering, the sampling window is set to an M×M rectangle, where M is an odd number greater than 1, the center of the sampling window is aligned with the terahertz image, and the window is slid from left to right and from top to bottom on the pixels of the terahertz image. After each slide, the pixel value at the center of the window is set to the median value of all pixel values ​​in the M×M rectangle.

3. The terahertz imaging processing method based on time series convolutional neural network according to claim 1, characterized in that: In step 3, use bilinear interpolation to scale the width W2 and height H2 of the terahertz image processed in step 2 to width W1 and height H1. That is, the size of the terahertz image processed in step 2 is W2×H2, and the size of the scaled terahertz image is W1×H1. For each pixel point P1=(x,y) of the scaled terahertz image, the corresponding coordinates on the terahertz image processed in step 2 are P2=(x*W2 / W1, y*H2 / H1). Obviously, this corresponding coordinate is generally not an integer. Around the pixel point P2 on the terahertz image processed in step 2, there are four pixel points Q 11 =(x1,y1), Q 12 =(x1,y2), Q 21 =(x2,y1), and Q 22 =(x2,y2) with values of f(Q 11 ), f(Q 12 ), f(Q 21 ), f(Q 22 ). Taking the lower left corner of the image as the origin, x, x1, x2 are the coordinate values in the horizontal direction of the image, and y, y1, y2 are the coordinate values in the vertical direction of the image. 0≤x1<x*W2 / W1<x2≤W2, 0≤y1<y*H2 / H1<y2≤H2, 0≤x≤W1, 0≤y≤H1, to obtain the value of the corresponding pixel point P1 on the scaled terahertz image , f(P1) is rounded to the nearest integer.

4. The terahertz imaging processing method based on time series convolutional neural network according to claim 1, characterized in that: In step 4, the main body of the time series convolutional neural network is divided into a ShuffleNetV2 network, which includes a convolution layer Conv, a maximum pooling layer MaxPool and three network blocks Stage2, Stage3 and Stage4 composed of the basic structure of ShuffleNetV2. The input of the main body is the sample image, and the output is connected to the detection head and the time series part respectively. The detection head part is divided into two branches, each of which is composed of a convolution layer and a ReLU activation function. The two branches output the category confidence and position offset of the sample image respectively. The time series part includes three convolution layers. In addition to the output of the main body, the input of the time series part also has the initial noise feature, and the output is the cyclic noise feature.

5. The terahertz imaging processing method based on time series convolutional neural network according to claim 1, characterized in that: In step 5, the target recognition frame in the sample image is calculated by combining the prior frame Anchor and the position offset, where the prior frame Anchor is a predefined n rectangular frame, n is a positive integer, and each rectangular frame is denoted by A i ={cx i , cy i , w i ,h i },0≤i≤n,cx i is the x coordinate of the center point of the prior box, cy i is the y coordinate of the center point of the prior box, w i is the width of the prior box, h i is the height of the prior frame. The size of the prior frame is preset according to the size of the object in the sample image to be processed. n is 4 or 5. The convolutional neural network based on time series outputs a total of m position offsets, where m is a positive integer. For each position offset O output j ={offset_x j , offset_y j , offset_w j , offset_h j }, 0≤j≤m, offset_x j is the offset of the x coordinate of the center point of the prior box, offset_y j is the offset of the y coordinate of the center point of the prior box, offset_w j is the offset of the prior box width, offset_h j is the offset of the height of the prior frame, and finally m*n target recognition frames are obtained, each of which is R k ={real_x k , real_y k , real_w k , real_h k }, 0≤k≤m*n, where: 。 6. The terahertz imaging processing method based on time series convolutional neural network according to claim 5, characterized in that: In step 5, the target position and category information in the sample image are calculated by combining the target recognition frame and category confidence. There are m*n target recognition frames, and the category confidences output by the convolutional neural network based on time series are m*n*c. Each target recognition frame corresponds to c category confidences. The category recognized by each target recognition frame is the category with the highest confidence among the c category confidences corresponding to it. The confidence of each target recognition frame is the confidence of the category it recognizes. The category confidence threshold is set to Ct. Only when the confidence of the target recognition frame is greater than Ct can the target recognition frame be considered credible. Then the credible target recognition frame is non-maximum. The NMS operation is used to remove target recognition frames with too high overlap. The remaining target recognition frames and their corresponding categories are the final identified target positions and categories. The specific operation of non-maximum suppression is to sort the confidence of all target recognition frames, select the target recognition frame with the highest confidence, and traverse the remaining target recognition frames. If the overlapping area with the current target recognition frame with the highest confidence is greater than the threshold Nt, the traversed target recognition frame will be deleted. After the traversal is completed, the target recognition frame with the highest confidence is processed, and the one with the highest confidence is selected from the unprocessed target recognition frames. Repeat the above process until all target recognition frames are processed.

7. The terahertz imaging processing method based on time series convolutional neural network according to claim 1, characterized in that: In the convolutional neural network based on time series, the initial noise feature is input for the first time, and the cyclic noise feature of the previous output is used to replace the initial noise feature when input again thereafter, and is input together with the new sample image. This allows the convolutional neural network based on time series to learn the complete noise feature in the time series and improve the quality of terahertz imaging, wherein the new sample image is an image acquired at the same position and angle at the next time point using the same terahertz imaging device and the same visible light imaging device through steps one, two, and three.

Images