Mask R-CNN-based infrared image segmentation method

By adding an improved attention mechanism and new loss function to the Mask R-CNN network, the problem of insufficient utilization of deep semantic information and low segmentation accuracy in infrared image segmentation is solved, and higher segmentation accuracy and better segmentation are achieved Effect.

CN118918115APending Publication Date: 2024-11-08NANJING UNIV OF SCI & TECH

Patent Information

Application Number
CN202410702824.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-05-31
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing infrared image segmentation methods cannot fully utilize the deep semantic information of the image, especially in complex backgrounds, and the segmentation effect is not good. Due to the low infrared image resolution, deep learning methods are difficult to obtain accurate image detail information, resulting in low segmentation accuracy.

Method used

Using a Mask R-CNN-based method, the generalization ability and segmentation accuracy of the model are improved by adding an improved attention mechanism and designing new regression box loss function and mask loss function. Specific steps include building an infrared image dataset, training an improved Mask R-CNN network, and using improved attention modules and loss functions to improve segmentation performance.

Benefits of technology

It achieves higher infrared image segmentation accuracy and better segmentation effect, improves the stability and convergence speed of the model, and has better segmentation performance compared with other deep learning methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918115A_ABST
    Figure CN118918115A_ABST
Patent Text Reader

Abstract

The invention provides an infrared image segmentation method based on a Mask R-CNN (Region-Convolutional Neural Network). According to the method, a Mask R-CNN network is used as a basic framework, and an improved attention module is added in the Mask R-CNN network to improve the recognition and segmentation capability of the model on an infrared target. And then a new regression frame loss function and a new mask loss function are designed to replace a loss function of a corresponding part of the original network, the problems that the convergence speed of the original network is too low and foreground and background are unbalanced are solved through an optimization mode without increasing parameters, and the stability of the model in the training process is further improved. In addition, the invention also constructs an infrared image data set containing four types of targets of people, vehicles, bicycles and unmanned aerial vehicles, and divides the infrared image data set into a training set and a test set. The training set is used for training the improved model to obtain optimal parameters; the test set is used for verifying the effect of the improved model and completing the recognition and segmentation of the target in the infrared image. Compared with other infrared image segmentation methods based on deep learning, the infrared image segmentation method based on deep learning has higher segmentation precision and better segmentation effect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method for infrared image segmentation based on Mask R-CNN. Background Art

[0002] Infrared image segmentation is an important task in the field of image processing. It is mainly used to separate the target object in the infrared image from the background and realize pixel-level classification. It is widely used in disease diagnosis, agricultural production, intelligent driving and defect detection. Compared with visible light images, infrared images use infrared radiation emitted by objects to form images. They are less affected by lighting conditions and have stronger anti-interference capabilities. Therefore, to a certain extent, infrared images can make up for the problem of performance degradation of visible light images under poor lighting conditions.

[0003] Traditional infrared image segmentation methods include matched filtering, edge detection, threshold segmentation, etc. They can achieve good segmentation results in specific situations, but traditional segmentation algorithms are limited to the surface features of infrared images and cannot make full use of deeper semantic information. In today's increasingly complex image backgrounds, the application effect is often unsatisfactory. With the continuous development of deep learning methods, neural networks have gradually been applied in the field of infrared image segmentation. It can solve the problems of low accuracy and harsh application environment of traditional image segmentation technology. However, due to the low overall resolution of infrared images, it is impossible to obtain very accurate image detail information, so it is challenging to use neural networks for infrared image segmentation. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a method for infrared image segmentation based on Mask R-CNN, which has strong generalization ability, higher segmentation accuracy and better segmentation effect.

[0005] The technical solution adopted by the present invention is: a method for infrared image segmentation based on Mask R-CNN, comprising the following steps:

[0006] Step 1: construct an infrared image dataset containing four types of targets: people, cars, bicycles, and drones, and divide it into a training set and a test set;

[0007] Step 2: Based on the Mask R-CNN network, we add an improved attention mechanism and design new regression box loss function and mask loss function to replace the original network loss function.

[0008] Step 3: Use the training set to train the improved Mask R-CNN network model from scratch to obtain the optimal parameters;

[0009] Step 4: Use the test set to verify the trained network model and obtain the final segmentation result.

[0010] Furthermore, in step 1, an infrared image dataset containing four types of targets, namely people, cars, bicycles and drones, is constructed and divided into a training set and a test set, as follows:

[0011] The images were acquired using the LA6110 high-performance uncooled infrared movement component camera. The initial image data consisted of 921 photos, which were expanded to 2760 photos after data labeling, data enhancement, and data cleaning. They were divided into training and test sets in a ratio of 8:2, with 2208 images in the training set and 552 images in the test set.

[0012] Furthermore, in step 2, the Mask R-CNN network is used as the basic architecture, an improved attention mechanism is added, and a new regression box loss function and mask loss function are designed to replace the original network loss function. The details are as follows:

[0013] The present invention optimizes the original SA attention mechanism by combining the bottleneck structure. The improved attention mechanism mainly includes two parts: the main convolution channel and the attention channel. The attention channel can be structured as Among them, Up(·) is the upsampling function used to restore the size of the output feature map of the attention channel, σ(·) represents the rectified linear unit function, Represents the attention channel F attn (·) output, and F attn (·) is composed of Ω representing the structure of four convolutional layers attn In addition, the average pooling function Apool(·) is used to input feature map X in ∈R C×H×W Downsampling is performed. The overall improved attention mechanism can be formulated as: out =X attn *X res +X attn , where X out ∈R C×H×W Represents the output feature map result, X res is the output of the main convolution channel, X res =F((X in ), F(·) represents the main convolution channel determined by Ω representing the structure of four convolutional layers.

[0014] The Mask R-CNN network model consists of four parts: the backbone network, the region proposal network, the region of interest alignment module, and the output module. The backbone network consists of two parts: the residual network and the feature pyramid network. The improved attention module is added to the backbone network of the Mask R-CNN network model. The feature map output from the residual network will be enhanced by the improved attention module and then input into the feature pyramid network to improve the overall network's segmentation performance for infrared targets.

[0015] In the loss function part, the present invention first designs a new regression box loss function by combining SIoU loss and Focal loss. The original regression box loss uses the cross entropy loss function, which can be expressed as Calculate, where t i is a vector representing the offset between the predicted output detection box and the anchor box. is with t i A vector with consistent dimensions, representing the offset between the anchor box and the ground-truth box.

[0016] However, in fact, the cross entropy loss function simply calculates the numerical difference between the predicted detection box and the real box. When |x|≥1, its first-order derivative with respect to x is a constant, which will affect the decrease of the loss value in the later stage of training, causing the network to fail to achieve better convergence results. In order to solve this problem, the present invention combines SIoU loss and Focal loss to design Focal_SIoU loss to replace the original network regression box loss function. SIoU loss will provide the predicted box with a direction to quickly approach the real box to accelerate the convergence speed of the network model. It uses four penalty terms: intersection-over-combination cost, angle cost, distance cost and shape cost to guide the correct decrease of the loss value. Specifically, the angle cost Λ will prompt the predicted box to move toward the horizontal / vertical distance closest to the real box, and the distance cost Δ is responsible for calculating the distance between the center points of the two boxes, which is greatly affected by the angle cost. When the angle cost decreases, the distance cost will also decrease accordingly, and vice versa. The shape cost Ω promotes the predicted box to be closer to the real box in shape by calculating the difference in width / height and the maximum width / height ratio. The specific calculation formula is as follows: Angle cost: in Represents the distance between the center points of the two boxes. Represents the difference in the vertical coordinates of the center points of the two boxes, Represents the center coordinates of the real frame, Represents the center coordinates of the prediction box. Distance cost: in γ=2-Λ,C x and C yIt is the width and height of the minimum enclosing rectangle of the real box and the predicted box. Shape cost: Where θ = 4, w and w gt are the width of the predicted box and the real box, h and h respectively. gt are the heights of the predicted box and the true box respectively. Combining the four penalty terms, SIoU loss can be formulated as: In addition, due to the imbalance between the foreground and background in the image, the prediction box that is closer to the real box accounts for a small proportion of the overall prediction result, and the prediction box with high regression accuracy should have a greater influence on the gradient during the model training process. Conversely, the loss value weight of the prediction box with poor regression effect should also be suppressed. To this end, the present invention combines the idea of ​​Focal loss to optimize SIoU loss, and finally obtains the optimized regression box loss function Focal_SIoU loss: L Focal-SIoU =IoU γ L SIoU , where γ is responsible for adjusting the degree of suppression of low-quality prediction boxes, and its value is generally 0.5.

[0017] The present invention also combines Dice loss and Lovasz-SoRmax loss to design a new mask loss function. The mask prediction branch in the MaskR-CNN network predicts the segmentation results based on the results of the region of interest, and outputs k mask images of size m*m for each region of interest, where k represents the total number of categories of targets detected on the image, and m represents the size of the mask image, which is generally set to 28. Each mask image only has predicted mask results for targets of the same category. For objects of category k, only the kth mask image and its true mask value need to be used for loss calculation, which effectively avoids the problem of inter-class competition. It can be said that the mask branch in the Mask R-CNN network converts the multi-classification loss calculation problem into multiple binary classification loss calculation problems. Therefore, its mask loss uses the average binary cross entropy loss, and the specific calculation formula is as follows: Where i represents the index of the pixel in the mask image, x i ∈(0, 1), which is obtained by transforming the output mask pixel value through the sigmoid function, y i ∈{0, 1} represents the positive and negative sample values ​​of the current mask pixel. k This means that for targets of category k, only the loss between the kth mask image and the true value is calculated. When the category is k, its value is 1, and when the category is not k, its value is set to 0.

[0018] The average binary cross entropy loss is essentially a pixel-by-pixel loss calculation function, which mainly calculates the loss based on the local information in the image. However, when the foreground and background in the image are extremely unbalanced or the size difference of the segmented target is too large, the average binary cross entropy loss will tend to learn the background or smaller targets and provide incorrect segmentation results. The Dice loss, on the other hand, tends to learn samples with larger targets from a global perspective and is not affected by the ratio of the foreground and background. It forms a complementary relationship with the average binary cross entropy loss, and its formula can be expressed as:

[0019] In addition, since both the average binary cross entropy loss and the Dice loss tend to learn the mask segmentation results of the correct classification, the present invention also adds the Lovasz-Softmax loss to supplement the difference features of the misclassification. The Lovasz-Softmax loss uses the Lovasz extension to expand the generated prediction probability distribution results into ordered subsets belonging to different categories to calculate the loss. Specifically, the definition To predict the label, define is the true label, then the IoU loss between the predicted label and the true label can be formulated as: The Lovasz-Softmax loss function, as a differential alternative to the IoU loss, can be formulated as: in, Represents the error function of the prediction result, c i ∈{0, 1} is the predicted label of pixel i of class k, G(x i ,y i )=Δ(S i ,y)-Δ(S i-1 , y), S i is x i The corresponding ordered set of segmented pixels, x i according to Sorting by i Sort by sorting result and x i The corresponding c i , that is, S i ={c (i=0) , c (i=1) , …, c (i=i)}.

[0020] The optimized mask loss function is referred to as MBCE_Dice_LS loss. In addition, L ls Multiply by a coefficient of 0.1 to ensure the consistency of the three loss values ​​in magnitude. The specific formula is: L MBCE_Dice_LS =L MBCE +L Dice+0.1L LS

[0021] Furthermore, in step 3, the improved Mask R-CNN network model is trained from scratch using the training set to obtain the optimal parameters. The specific method is: the training set has a total of 2208 images, the input image size is 640*512, the batch setting is 8, and the NVIDIA GeForce RTX 3090 Ti (arch=8.6) GPU is used for acceleration. The software environment is set to Python 3.8, PyTorch 1.10, and the learning rate is set to 0.001. The warm-up method is used to adjust the initial learning rate, and the Adam optimizer is used to optimize the model. A total of 100 rounds of training are performed to avoid poor generalization ability caused by underfitting or overfitting.

[0022] Furthermore, in step 4, the trained network model is verified using the test set to obtain the final segmentation result. Specifically, the images in the test set are input into the improved Mask R-CNN model that has been loaded with the optimal weights, and the subjective segmentation results and objective performance indicators of the images are finally output. The subjective segmentation results are displayed by visualizing the images, and the objective performance indicators use the COCO dataset evaluation indicators to judge the effect of the algorithm model based on the results of mAP and recall rate.

[0023] For the network model, the recall rate indicates the proportion of samples predicted correctly in the real samples, and the precision indicates the proportion of correct prediction results in all prediction results. AP is obtained by calculating the area of ​​the curve with the horizontal axis being the recall rate and the vertical axis being the precision. The mAP is the average of the AP of each category in different categories. The larger the mAP value, the better the model accuracy is considered to be.

[0024] Compared with the prior art, the beneficial effects of the present invention are: taking the Mask R-CNN network as the basic architecture, an improved attention module is added to it to improve the model's recognition and segmentation capabilities for infrared targets; a new regression box loss function and mask loss function are designed to replace the loss function of the corresponding part of the original network, further improving the stability and convergence speed of the model during training. Compared with other infrared image segmentation methods based on deep learning, the present invention has higher segmentation accuracy and better segmentation effect.

[0025] The present invention will be further described below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the overall process implementation of the present invention.

[0027] Figure 2 It is a schematic diagram of the network structure implementation process of the present invention.

[0028] Figure 3 Schematic diagram of the improved attention module structure in the present invention.

[0029] Figure 4 Schematic diagram of the improved Mask R-CNN backbone network structure in the present invention.

[0030] Figure 5 This is a diagram showing the model segmentation effect in the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described below in conjunction with the accompanying drawings.

[0032] The present invention provides a method for infrared image segmentation based on Mask R-CNN, comprising the following steps:

[0033] Step 1: Build an infrared image dataset containing four types of targets: people, cars, bicycles, and drones, and divide it into a training set and a test set. The initial image data taken by the infrared camera has a total of 921 photos, which are expanded to 2760 after data annotation, data enhancement, and data cleaning. They are divided into a training set and a test set in a ratio of 8:2. The training set has 2208 images and the test set has 552 images.

[0034] Step 2: Based on the Mask R-CNN network, we add an improved attention mechanism and design new regression box loss function and mask loss function to replace the original network loss function.

[0035] Figure 1 It is an overall flow chart of the present invention. After constructing the data set, the present invention uses the training set to train the improved Mask R-CNN network and uses the test set to test the model performance.

[0036] Figure 2 It is a schematic diagram of the overall network flow for implementing the method of the present invention. A new attention mechanism is added to the original Mask R-CNN backbone network, and the loss functions of the regression frame module and the mask module of the network output part are modified. The feature map output after the input image is extracted by the residual network needs to be enhanced by the attention module, and then enter the feature pyramid network for feature fusion. The feature map after feature fusion is sent to the region proposal network to extract the region of interest. Subsequently, the region of interest alignment module will process all the obtained regions of interest to a uniform size and send the results to the final network output module. The network output module classifies and identifies the obtained regions of interest, performs regression frame correction and mask extraction, and comprehensively generates the final segmentation result.

[0037] Figure 3This is a schematic diagram of the improved attention module structure in the present invention. The present invention optimizes the original SA attention module by combining the bottleneck structure, which mainly includes two parts: the main convolution channel and the attention channel. The attention channel can be structured as Among them, Up(·) is the up-sampling function used to restore the size of the output feature map of the attention channel, σ(·) represents the ReLU activation function, Represents the attention channel F attn (·) output, and F attn (·) is composed of Ω representing the structure of four convolutional layers attn In addition, the average pooling function Apool(·) is used to input feature map X n ∈R C×H×W Downsampling is performed. The overall improved attention module can be formulated as: out =X attn *X res +X attn , where X out ∈R C×H×W Represents the output feature map result, X res is the output of the main convolution channel, X res =F((X in ), F(·) represents the main convolution channel determined by Ω representing the structure of four convolutional layers.

[0038] Figure 4 This is a schematic diagram of the improved Mask R-CNN backbone network structure in the present invention. The attention mechanism designed by the present invention has a significant feature enhancement effect on the feature map extracted by the residual network, so it is added after the residual network in the Mask R-CNN backbone network. In addition, in order to avoid the influence of too scattered feature information of shallow feature maps and excessive convolution of deep feature maps, the attention module is inserted after the C3 and C4 feature maps of the residual network to enhance the overall network's ability to segment infrared targets.

[0039] In addition, the present invention designs new regression box loss functions and mask loss functions to optimize training. In terms of regression box loss, since the original regression box loss function simply calculates the numerical difference between the predicted box and the true box, it cannot achieve a good convergence effect in the later stage of network training. Therefore, combined with SIoU loss and Focal loss, the model training results are optimized by quickly converging the prediction box and reducing the proportion of incorrect prediction results. The newly designed regression box loss function is named Focal_SIoU loss, specifically: SIoU Loss: Focal_SIoU loss: L Focal-SIoU=IoU γ L SIoU

[0040] In terms of mask loss, in order to avoid the erroneous learning results of the original average binary cross entropy mask loss function when the foreground and background are extremely unbalanced or the size difference of the segmented target is too large, Dice loss and Lovasz-Softmax loss are used as supplements to optimize the loss function of the segmentation part. The weighted fusion mask loss function is named MBCE_Dice_LS loss, which is: Average Binary Cross Entropy Mask Loss Function: Dice loss: Lovasz-Softmax Loss: MBCE_Dice_LS loss: L MBCE_Dice_LS =L MBCE +L Dice +0.1L LS

[0041] Step 3. Use the training set to train the improved Mask R-CNN network model from scratch to obtain the optimal parameters. The specific method is: the training set has a total of 2208 pictures, the input image size is 640*512, the batch setting is 8, and the NVIDIA GeForce RTX 3090 Ti (arch=8.6) GPU is used for acceleration. The software environment is set to Python 3.8, PyTorch1.10, and the learning rate is set to 0.001. The warm-up method is used to adjust the initial learning rate, and the Adam optimizer is used to optimize the model. A total of 100 rounds of training are performed to avoid poor generalization ability caused by underfitting or overfitting.

[0042] Step 4: Use the test set to verify the trained network model and obtain the final segmentation result. Specifically, the images in the test set are input into the improved Mask R-CNN model that has been loaded with the optimal weights, and the subjective segmentation results and objective performance indicators of the images are finally output. The subjective segmentation results are displayed by visualizing the images, and the objective performance indicators use the COCO dataset evaluation indicators to judge the effect of the algorithm model based on the results of mAP and recall rate.

[0043] Figure 5 This is a diagram showing the model segmentation effect in the present invention.

[0044] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. A person with ordinary knowledge in the technical field to which the present invention belongs may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the definition of the claims.

Claims

1. A method for infrared image segmentation based on Mask R-CNN, characterized in that: Here are the steps: Step 1: construct an infrared image dataset containing four types of targets: people, cars, bicycles, and drones, and divide it into a training set and a test set; Step 2: Based on the Mask R-CNN network, we add an improved attention mechanism and design new regression box loss function and mask loss function to replace the original network loss function. Step 3: Use the training set to train the improved Mask R-CNN network model from scratch to obtain the optimal parameters; Step 4: Use the test set to verify the trained network model and obtain the final segmentation result.

2. The method for infrared image segmentation based on Mask R-CNN according to claim 1, characterized in that: The specific process of step 1 is: 921 images are obtained by taking pictures with an infrared camera, which are expanded to 2760 images after data annotation, data enhancement and data cleaning, and divided into training set and test set in a ratio of 8:

2. The training set has 2208 images and the test set has 552 images.

3. The method for infrared image segmentation based on Mask R-CNN according to claim 1, characterized in that: The specific process of step 2 is: Step 1: The present invention optimizes the original SA attention module by combining the bottleneck structure. A 1*1 convolution module is added before and after the main convolution channel and the attention convolution channel in the original SA attention module, and the number of channels is scaled to 1 / 4 of the original convolution layer to reduce the amount of calculation parameters. The optimized attention module is added to the Mask R-CNN backbone network to improve the infrared image segmentation effect; Step 2: Construct a new regression box loss function to optimize training. Since the original regression box loss function simply calculates the numerical difference between the predicted box and the true box, it will affect the decrease of the loss value in the later stage of network training. Therefore, SIoU loss and Focal loss are combined to optimize model training by promoting rapid convergence of the prediction box and reducing the proportion of incorrect prediction results. The newly designed regression box loss function is named Focal_SIoU loss, specifically: SIoU Loss: Δ=∑ t=x,y (1-e -γρt ),Ω=∑ t=w,h (1-e -ωt ) θ Focal_SIoU loss: L Focal-SIoU =IoU γ L SIoU Step 3: Construct a new mask loss function to optimize the segmentation results. In order to avoid the incorrect learning results of the original average binary cross entropy mask loss function when the foreground and background are extremely unbalanced or the size difference of the segmented target is too large, Dice loss and Lovasz-Softmax loss are used as supplements to optimize the loss function of the segmentation part. The weighted fusion mask loss function is named MBCE_Dice_LS loss, which is: Average Binary Cross Entropy Mask Loss Function: Dice loss: Lovasz-Softmax Loss: MBCE_Dice_LS loss: L MBCE_Dice_LS =L MBCE +L Dice +0.1L LS 4. The method for infrared image segmentation based on Mask R-CNN according to claim 1, characterized in that: The specific process of step 3 is: There are 2208 training sets in total, the image size is 640*512, the batch size is set to 8, and the NVIDIA GeForceRTX3090 Ti (arch=8.6) GPU is used for acceleration. The software environment is set to Python 3.8 and PyTorch 1.

10. The learning rate is set to 0.001, and the warm-up method is used to adjust the initial learning rate. The Adam optimizer is used to optimize the model. A total of 100 rounds of training are performed to avoid the problem of poor generalization ability caused by underfitting or overfitting.

5. The method for infrared image segmentation based on Mask R-CNN according to claim 1, characterized in that: The specific process of step 4 is: The images in the test set are input into the improved Mask R-CNN model that has been loaded with the optimal weights, and the subjective segmentation results and objective performance indicators of the images are finally output. The subjective segmentation results are displayed by visualizing the images, and the objective performance indicators use the COCO dataset evaluation indicators to judge the effect of the algorithm model based on the results of mAP and recall rate.

Citation Information

Patent Citations

  • Faster R-CNN-based bridge crack instance segmentation method

    CN111861978A

  • Improved Mask R-CNN target segmentation method

    CN114445620A

  • Flower identification and detection method based on improved Mask R-CNN

    CN117975142A

Cited By

  • Instance segmentation method and system for infrared image of power equipment

    CN119762789A