A method and system for detecting target centroid of infrared sequence images

Abstract

The present invention relates to the field of image processing technology, and in particular to a method and system for detecting the target centroid of infrared sequence images. The method comprises: performing subsequence splitting on the infrared sequence images, taking five consecutive frames of images as subsequence images; performing background matching preprocessing on the first subsequence image; calculating a label centroid map with the same size as the preprocessed image based on a simplified two-dimensional Gaussian formula and the true position of the target centroid; inputting the preprocessed sequence image into an improved U‑Net neural network, performing feature extraction on the preprocessed sequence image, and outputting a predicted centroid map; inferring the rough position of the target centroid based on the predicted centroid map; performing centroid correction on the predicted rough position of the target centroid to obtain the target centroid position; and repeating the above steps until all subsequence images are processed. The advantages are: directly outputting a centroid map containing target centroid information based on the sequence image, and calculating the accurate position of the target centroid; the centroid positioning accuracy is high and the real-time performance is good.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for detecting target centroids of infrared sequence images. Background Art

[0002] In the application fields of infrared early warning, missile interception and trajectory estimation, it is more important to detect the target's centroid than to detect the target's bounding box. The existing centroid detection algorithms first detect the target's bounding box, and then calculate the target's centroid using centroid calculation methods such as the grayscale centroid method. This method is less effective when calculating the centroid of irregularly shaped targets. Therefore, the present invention proposes a centroid detection network that can directly output a centroid map containing the target's centroid information based on the input image. Based on this centroid map, we can calculate the exact position of the target's centroid.

[0003] Since in the field of infrared remote sensing, the target is very far from the imaging device, the target appears as a faint dot in the image. There are often many interference sources in the infrared image, and it is difficult to distinguish the target and false alarm in the image by relying on a single frame image alone. The existing method of removing false alarms is to first detect all targets and false alarms, and then use a post-processing method to remove false alarms. This method has poor real-time performance. Therefore, the present invention proposes a sequence image target centroid detection network, which can extract the motion characteristics of the target according to the input sequence image, thereby easily distinguishing the target and false alarm. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a method and system for detecting the target centroid of infrared sequence images.

[0005] The first object of the present invention is to provide a method for detecting the centroid of an object in an infrared sequence image, which specifically comprises the following steps:

[0006] S1. splitting the infrared sequence image into subsequences, taking five consecutive frames of images as subsequence images, and obtaining a plurality of subsequence images; performing background matching preprocessing on the first subsequence image;

[0007] S2. Calculate a label centroid map with the same size as the preprocessed sequence image based on the simplified two-dimensional Gaussian formula and the true position of the target centroid;

[0008] S3. Input the preprocessed sequence images into the improved U-Net neural network, perform convolution operation on the preprocessed sequence images, extract image features, and output a predicted centroid map;

[0009] S4. Inferring the rough position of the target centroid using a simplified two-dimensional Gaussian formula based on the predicted centroid map;

[0010] S5. Use a centroid correction algorithm to perform centroid correction on the predicted rough position of the target centroid to obtain the final target centroid position;

[0011] S6. Repeat steps S1-S5 until the processing of all sub-sequence images in the infrared sequence image is completed.

[0012] Preferably, step S1 specifically includes the following sub-steps:

[0013] S101. Collect continuous images from the first frame to the fifth frame in the infrared sequence image, and mark image 1, image 2, image 3, image 4 and image 5; take a pixel block in the central area of ​​image 1, find a pixel block in the central area of ​​image 5 that is most similar to the taken pixel block, and calculate the relative displacement of the pixel block; determine the image of the retained part in image 5 according to the displacement, and paste the retained part of the image in the upper left corner of image 1 to obtain a background matching image; the background matching process of image 2, image 3 and image 4 is the same as that of image 1, and finally obtain 4 background matching images;

[0014] S102. Concatenate the four background matching images and image 5 by channel.

[0015] Preferably, the simplified two-dimensional Gaussian formula in step S2 is as follows:

[0016] (1)

[0017] in, It represents the value corresponding to the position in the i-th row and j-th column of the centroid map, x and y represent the horizontal and vertical coordinates of the target centroid, and j and i represent the horizontal and vertical coordinates of each position in the centroid map.

[0018] Preferably, the improved U-Net neural network includes a feature extraction module CBR consisting of a convolution kernel of size 3×3, a pooling layer, and a Relu non-linear activation layer.

[0019] Preferably, the convolution operation of step S3 is specifically as follows: the input feature map is sequentially passed through the convolution kernel, the pooling layer, and the Relu nonlinear activation layer to obtain a deeper feature map; and the feature map is then downsampled three times and upsampled three times before output.

[0020] Preferably, downsampling uses a convolution kernel with a stride of 2 and a size of 2×2 to find the maximum value within a 2×2 window on the feature map as the output.

[0021] Preferably, step S4 specifically includes: finding the pixel with the largest value in the predicted centroid map as the centroid position, expanding three pixels outward from the centroid position to obtain pixel block B; correcting pixel block B according to the centroid correction algorithm to solve the offset C; adding the offset C and the centroid position obtained from the predicted centroid map to obtain the final target centroid position.

[0022] Preferably, the calculation formula of the offset C in the centroid correction algorithm is as follows:

[0023] (2)

[0024] (3)

[0025] (4)

[0026] (5)

[0027] (6)

[0028] Among them, C represents the offset of the final centroid in the xy direction, and x and y represent the horizontal and vertical coordinates of other pixels in the rectangular coordinate system established with the center point of pixel block B. Indicates the pixel value at this position in pixel block B.

[0029] Preferably, the step size of the subsequence images is 1.

[0030] The second object of the present invention is to provide a target centroid detection system for infrared sequence images, comprising:

[0031] The sequence image data preprocessing module is used to perform background matching and channel splicing on the infrared sequence image data, and use the spliced ​​sequence image as the input of the U-Net neural network;

[0032] A sequence image feature extraction module is used to extract image features from the spliced ​​sequence images and obtain a predicted centroid map;

[0033] The centroid calculation module is used to perform centroid correction on the predicted centroid map to obtain the final target centroid position.

[0034] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0035] The method and system proposed in the present invention can directly output a centroid map containing target centroid information according to the input image, and the accurate position of the target centroid can be calculated according to the centroid map; the centroid positioning accuracy is high and the real-time performance is good. The target centroid is accurately detected through a lightweight neural network system, and the lightweight neural network is convenient to be transplanted into embedded devices for use. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The present invention is a flowchart of a method for detecting the target centroid of infrared sequence images provided by an embodiment of the present invention.

[0037] Figure 2It is an overall schematic diagram of the technical process of the target centroid detection method of infrared sequence images provided by an embodiment of the present invention.

[0038] Figure 3 1 is a schematic diagram of background matching of a method for detecting target centroids of infrared sequence images provided by an embodiment of the present invention.

[0039] Figure 4 is a label centroid diagram provided according to an embodiment of the present invention.

[0040] Figure 5 Schematic diagram of the U-Net neural network structure for extracting features from sequence images provided according to an embodiment of the present invention.

[0041] Figure 6 It is a predicted centroid map output by the U-Net neural network provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are represented by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, the detailed description thereof will not be repeated.

[0043] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0044] The present invention proposes a method for detecting the target centroid of infrared sequence images. For a real-time transmitted video stream, firstly, 1 to 5 frames of continuous images are subjected to background matching so that a convolutional neural network can extract the motion characteristics of the target. Then, the 5 images after background matching are spliced ​​by channel and sent to the neural network, and the neural network will predict a centroid map with the same size as the input image. This centroid map represents the centroid information of the 5th frame of the 1 to 5 frames of images, and the centroid position of the target in the 5th frame of the image can be inferred based on this centroid map. In order to reduce the error of the target centroid calculated by the centroid map, a centroid correction algorithm is also designed to improve the centroid positioning accuracy. The 2nd to 6th frames of images are processed according to the above process until all images are processed.

[0045] The present invention provides a method for detecting the target centroid of infrared sequence images ( Figure 1-Figure 2 The detection process is shown in FIG. 1 ), which specifically includes the following steps:

[0046] S1. splitting the infrared sequence image into subsequences, taking five consecutive frames of images as subsequence images, and obtaining a plurality of subsequence images; performing background matching preprocessing on the first subsequence image; specifically comprising the following sub-steps:

[0047] S101. Collect continuous images from the first frame to the fifth frame in the infrared sequence image, mark image 1, image 2, image 3, image 4 and image 5, and perform background matching between image 1, image 2, image 3, image 4 and image 5 respectively; specifically, taking image 1 as an example, take a pixel block in the central area of ​​image 1, find a pixel block in the central area of ​​image 5 that is most similar to the taken pixel block, and calculate the relative displacement of the pixel block; determine the image of the retained part in image 5 according to the displacement, paste the retained part of the image in the upper left corner of image 1, and obtain a background matching image; compared with image 1, the background matching image has the same background position and a different target position; the background matching process of image 2, image 3 and image 4 is consistent with that of image 1, and finally obtains 4 background matching images;

[0048] Figure 3 shows the background matching process. Figure 3 Take a pixel block from the central area of ​​a ( Figure 3 As shown in the solid line part in a), Figure 3 The central area of ​​c ( Figure 3 c) to find and Figure 3 The pixel block most similar to the pixel block taken from a ( Figure 3 Calculate the relative displacement of the pixel blocks in the two images, and determine Figure 3 The part of c to be retained ( Figure 3 As shown in the white dotted box in c), paste this part of the image into Figure 3 After the upper left corner of a, we get Figure 3 The image shown in b;

[0049] S102. Concatenate the four background matching images and image 5 by channel.

[0050] S2. Calculate a label centroid map with the same size as the preprocessed sequence image based on the simplified two-dimensional Gaussian formula and the true position of the target centroid;

[0051] Specifically, a simplified two-dimensional Gaussian formula is used to make a centroid map. The calculation formula of the centroid map is as follows:

[0052] (1)

[0053] in, It represents the value corresponding to the position in the i-th row and j-th column of the centroid map, x and y represent the horizontal and vertical coordinates of the target centroid, and j and i represent the horizontal and vertical coordinates of each position in the centroid map.

[0054] The purpose of the centroid detection task is to detect the centroid of the target in the image. In order to make the neural network more susceptible to convergence, we use a simplified two-dimensional Gaussian formula to create a centroid map, and let the neural network output the centroid map based on the input sequence image. Assuming that the centroid coordinates are (3.3, 4.6) and the input image size is 8×8, then the centroid map can be calculated according to the above formula as follows: Figure 4 As shown, the value of each position in the centroid map represents the proximity of the position to the target centroid.

[0055] S3. Input the preprocessed sequence images into the improved U-Net neural network, perform convolution operation on the preprocessed sequence images, extract image features, and output a predicted centroid map;

[0056] The improved U-Net neural network includes a feature extraction module CBR consisting of a convolution kernel of size 3×3, a pooling layer, and a Relu nonlinear activation layer;

[0057] The specific process of convolution operation is as follows:

[0058] The feature extraction module CBR consists of a 3×3 convolution kernel, a pooling layer, and a Relu nonlinear activation layer. The input feature map is sequentially subjected to a 2D convolution with a 3×3 convolution kernel, a normalization layer, and a nonlinear activation layer to obtain a deeper feature map. Specifically, three downsampling operations are performed first, and each downsampling operation uses a 2×2 convolution kernel with a step size of 2 to find the maximum value in a 2×2 window on the feature map as the output. Three upsampling operations are then performed, and each upsampling operation uses a deconvolution operation. Figure 5 As shown in the figure, downsampling × 2 and upsampling × 2 indicate that the downsampling and upsampling ratios are 2 times; the gray solid line in the figure indicates that the feature map obtained in the previous step is upsampled or downsampled and then given to the next feature extraction module, and the black dotted line indicates that the feature map obtained in the previous step is directly given to the next feature extraction module without being processed; if the feature extraction module CBR in the figure has two inputs, it means that the two input feature maps are spliced ​​by channel and then input into the feature extraction module CBR; if the feature extraction module CBR in the figure has two outputs, the two outputs are the same feature map.

[0059] Principle description: Since the size of the target to be detected in the application scenario of the present invention is very small, usually only a few pixels in size, this places high demands on feature extraction and positioning accuracy. In the traditional U-Net structure, the network captures feature information of different scales through multiple downsampling and upsampling. However, in the present invention, since the target size is extremely small, there is no need to perform too many downsampling operations to avoid losing too much detail information in the process of reducing the resolution. Therefore, the designed U-Net network only performs three downsamplings, which can ensure sufficient spatial resolution to locate small targets while maintaining computational efficiency.

[0060] The input of the neural network is a sequence of images after background matching by channel splicing. This input method can make full use of the spatiotemporal information in the sequence of images, which helps the network learn the motion characteristics of the target. Through the convolution operation, the network can gradually extract the high-level semantic features in the image, which are crucial for the subsequent centroid detection. Although in the present invention, we use a convolutional neural network with a small number of parameters, experiments have shown that such a network structure is effective enough to extract the motion characteristics of the target. By training the network, we can let the network learn the mapping relationship from the input image to the target centroid map.

[0061] Assuming the performance of the neural network is good enough, the predicted centroid map should be exactly the same as the labeled centroid map. Figure 6 As shown, find the pixel with the largest value in the centroid map, and then find the pixel with larger value in the horizontal direction and the vertical direction in its four neighborhoods. If the prediction is accurate enough, since these three pixels are calculated according to the calculation formula of the centroid map, we can determine the vertical coordinate of the target centroid based on the pixel with a value of 0.88250 and the pixel with a value of 0.79852, and determine the horizontal coordinate of the target centroid based on the pixel with a value of 0.88250 and the pixel with a value of 0.72253. The calculation process is as follows:

[0062]

[0063]

[0064] According to these two formulas, the y value can be calculated. Similarly, the x value can be calculated according to the pixel with a value of 0.88250 and the pixel with a value of 0.72253.

[0065] However, in actual operation, the predicted target center of mass is still a little different from the actual center of mass, so the next step is needed.

[0066] S4. Based on the predicted centroid map, the rough position of the target centroid is inferred using a simplified two-dimensional Gaussian formula.

[0067] S5. Use the centroid correction algorithm to perform centroid correction on the rough position of the predicted target centroid to obtain the final target centroid position; specifically, find the pixel with the largest value in the predicted centroid map as the centroid position, expand three pixels outward from the centroid position to obtain a 7×7 pixel block B, and ensure that all target pixels are included in the 7×7 pixel block; correct the pixel block B according to the centroid correction algorithm to solve the offset C; add the offset C and the centroid position obtained from the predicted centroid map to obtain the final target centroid position; the calculation formula of the offset C in the centroid correction algorithm is as follows:

[0068] (2)

[0069] (3)

[0070] (4)

[0071] (5)

[0072] (6)

[0073] Among them, C represents the offset of the final centroid in the xy direction, and x and y represent the horizontal and vertical coordinates of other pixels in the rectangular coordinate system established with the center point of pixel block B. Indicates the pixel value at this position in pixel block B.

[0074] S6. Collect the second to sixth consecutive images and repeat steps S1-S5; each time the processing moves back one frame of image until all images are processed.

[0075] In addition, the present invention provides a target centroid detection system for infrared sequence images, comprising:

[0076] The sequence image data preprocessing module is used to perform background matching and channel splicing on the infrared sequence image data, and use the spliced ​​sequence image as the input of the U-Net neural network;

[0077] A sequence image feature extraction module is used to extract image features from the spliced ​​sequence images and obtain a predicted centroid map;

[0078] The centroid calculation module is used to perform centroid correction on the predicted centroid map to obtain the final target centroid position.

[0079] The advantages of the present invention are: (1) Due to the particularity of the input data (sequence images), we can design a very lightweight neural network to accurately detect the target center of mass. The lightweight neural network is easy to port to embedded devices for use. (2) The accuracy of calculating the target center of mass is higher. First, the center of mass map output by the neural network can accurately calculate the target center of mass, and its accuracy is unmatched by traditional center of mass calculation methods (grayscale value center of mass method, etc.); secondly, the center of mass is further corrected using the image blocks around the predicted center of mass position, making the output center of mass position more accurate.

[0080] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0081] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting the centroid of an infrared sequence image, characterized in that: The specific steps include: S1. splitting the infrared sequence image into subsequences, taking five consecutive frames of images as subsequence images, and obtaining a plurality of subsequence images; performing background matching preprocessing on the first subsequence image; specifically comprising the following sub-steps: S101. Collect continuous images from the first frame to the fifth frame in the infrared sequence image, and mark image 1, image 2, image 3, image 4 and image 5; take a pixel block in the central area of ​​image 1, find a pixel block in the central area of ​​image 5 that is most similar to the taken pixel block, and calculate the relative displacement of the pixel block; determine the image of the retained part in image 5 according to the displacement, and paste the retained part of the image in the upper left corner of image 1 to obtain a background matching image; the background matching process of image 2, image 3 and image 4 is the same as that of image 1, and finally obtain 4 background matching images; S102. Splice the 4 background matching images and image 5 by channel; S2. Based on the simplified two-dimensional Gaussian formula and the true position of the target centroid, a label centroid map with the same size as the preprocessed sequence image is calculated; the simplified two-dimensional Gaussian formula is as follows: （1） in, represents the value corresponding to the position in the i-th row and j-th column of the centroid map, x and y represent the horizontal and vertical coordinates of the target centroid, and j and i represent the horizontal and vertical coordinates of each position in the centroid map; S3. Input the preprocessed sequence images into the improved U-Net neural network, perform convolution operation on the preprocessed sequence images, extract image features, and output a predicted centroid map; S4. Inferring the rough position of the target centroid using a simplified two-dimensional Gaussian formula based on the predicted centroid map; S5. Use a centroid correction algorithm to perform centroid correction on the predicted rough position of the target centroid to obtain the final target centroid position; S6. Repeat steps S1-S5 until the processing of all sub-sequence images in the infrared sequence image is completed.

2. The method for detecting target centroid of infrared sequence images according to claim 1, characterized in that: The improved U-Net neural network includes a feature extraction module CBR consisting of a convolution kernel with a size of 3×3, a pooling layer, and a Relu non-linear activation layer.

3. The method for detecting target centroid of infrared sequence images according to claim 2, characterized in that: The convolution operation of step S3 is specifically as follows: the input feature map is sequentially passed through the convolution kernel, the pooling layer, and the Relu nonlinear activation layer to obtain a deeper feature map; the feature map is then downsampled three times and upsampled three times before output.

4. The method for detecting target centroid of infrared sequence images according to claim 3, characterized in that: Downsampling uses a convolution kernel with a stride of 2 and a size of 2×2 to find the maximum value within a 2×2 window on the feature map as the output.

5. The method for detecting target centroid of infrared sequence images according to claim 1, characterized in that: The step S4 specifically includes: finding the pixel with the largest value in the predicted centroid map as the centroid position, expanding three pixels outward from the centroid position to obtain pixel block B; correcting pixel block B according to the centroid correction algorithm to solve the offset C; adding the offset C and the centroid position obtained from the predicted centroid map to obtain the final target centroid position.

6. The method for detecting target centroid of infrared sequence images according to claim 5, characterized in that: The calculation formula of the offset C in the centroid correction algorithm is as follows: （2） （3） （4） （5） （6） Among them, C represents the offset of the final centroid in the xy direction, and x and y represent the horizontal and vertical coordinates of other pixels in the rectangular coordinate system established with the center point of pixel block B. Indicates the pixel value at this position in pixel block B.

7. The method for detecting target centroid of infrared sequence images according to any one of claims 1 to 6, characterized in that: The step size of the subsequence image is 1.

Images