Three-channel image instance segmentation method based on thermodynamic diagram guidance
Through the three-channel image instance segmentation method guided by the thermal map, combining low-pass filtering and high-pass filtering channels, combined with spatial attention mechanism, the problem of insufficient robustness and accuracy of the existing technology in complex situations is solved, and the complexity is achieved The efficient and accurate segmentation of the targets in the scene improves the robustness and adaptability of image instance segmentation.
Patent Information
- Application Number
- CN202410113620.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-26
- Publication Date
- 2024-11-29
AI Technical Summary
The existing image instance segmentation method shows limited robustness and accuracy in complex situations, such as overlap between targets, non-uniform lighting and noise, and it is difficult to meet the precise segmentation needs of diversified targets.
The three-channel image instance segmentation method based on thermal map guidance is adopted to filter out noise and capture high-frequency information through low-pass filtering channels. Combined with the spatial attention mechanism, the model detects the target accuracy and stability.
In complex scenarios, the robustness and adaptability of image instance segmentation are significantly improved, the ability to capture target edges and subtle features is enhanced, and the model's understanding and generalization of target structure and context features is improved.
Smart Images

Figure CN119048533A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of image processing, and in particular to a three-channel image instance segmentation method based on heat map guidance. Background Art
[0002] In the field of image processing, image instance segmentation, as a key technology, aims to accurately identify individual targets in images and provide detailed semantic information for computer vision systems. Currently, with the continuous development of image instance segmentation technology, its application in medical image analysis, autonomous driving, intelligent monitoring and other fields is becoming more and more extensive. However, existing image instance segmentation methods often show limited robustness and accuracy when faced with certain complex situations, such as overlap between targets, non-uniform illumination and noise in images. In addition, traditional methods are usually sensitive to the spatial scale and shape changes of targets, and it is difficult to meet the needs of accurate segmentation of diverse targets in practical applications.
[0003] To meet the above challenges, the present invention proposes a three-channel image instance segmentation method based on heat map guidance. This method innovatively uses heat map information to guide the low-pass filter channel, learns and guides the heat map of the target in the image to filter out the noise in the image, and captures the high-frequency information in the image through the high-pass filter channel, thereby achieving efficient and accurate segmentation of individual targets in complex scenes. Compared with traditional methods, the method of the present invention shows stronger robustness and adaptability in dealing with complex situations such as overlap between targets, large-scale changes and illumination changes. Therefore, the present invention aims to bring an innovative and efficient solution to the field of image instance segmentation, so as to enhance the image interpretation and understanding capabilities of computer vision systems in complex scenes. Through this method, we hope to provide more reliable, accurate and robust image instance segmentation algorithms for fields such as intelligent transportation systems, visual guidance and medical images, and promote the widespread implementation of image instance segmentation technology in more practical applications. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a three-channel image instance segmentation method based on heat map guidance, which can fully utilize the heat map information to guide the low-pass filter channel to filter out the background noise in the image, and capture the high-frequency information of the image through the high-pass filter channel. The three channels can capture the texture features and context features in the image more comprehensively and accurately, improve the accuracy and stability of target detection, and improve the model's understanding and generalization ability of target and context features.
[0005] The technical solution of the present invention is:
[0006] A three-channel image instance segmentation method based on heat map guidance specifically includes the following steps:
[0007] (1) Perform image preprocessing and then use the convolution module in the backbone network to extract features from the image to obtain input features;
[0008] (2) Using the heat map guidance channel to calculate the adaptive heat map guidance parameter vector;
[0009] (3) After low-pass filtering the feature extraction image using a low-pass filter channel, adaptive feature fusion is performed with the adaptive heat map guidance parameter vector to obtain the features guided by the heat map;
[0010] (4) The input feature map and the guided feature map are input into the high-pass filter guidance channel as a difference, and the spatial attention mechanism is used to make the model more focused on the target area in the image, so as to more effectively capture the important information of the target or task, while suppressing irrelevant texture information;
[0011] (5) The output features of the low-pass filter channel guided by the heat map are added to the output features of the high-pass filter, and then the category, border position and mask are obtained through the segmentation head, and the instance segmentation result is obtained by superposition.
[0012] The input image to be detected is defined as input image P; the image preprocessing is to perform multiple convolution processing on all training images through the backbone network; by analyzing related tasks and training based on existing data sets, a trained network is obtained, and the input image P is input to the trained network for instance segmentation.
[0013] The specific steps of the three-channel image instance segmentation method based on heat map guidance are:
[0014] (1) Use the commonly used backbone network module to perform a convolution operation on the input image P to extract features, and then obtain the input feature X after another convolution operation;
[0015] (2) Input feature X into the heat map guidance channel to obtain the heat map M(i,j) of the input feature:
[0016]
[0017] Where i, j, and n are the row index, column index, and number of channels of the feature, respectively.
[0018] (3) Decompose the heat map into background area S b 、Target high heat area S th and target low heat area S tl When M=S b +S th +S tl , in order to achieve better instance segmentation effect, make S bThe regional high heat is small enough, while ensuring S th and S tl It can fully cover the target area, thereby guiding the characteristics of the low-pass filter path output Corresponding heat map Satisfies the following objective function:
[0019]
[0020] (4) The obtained heat map is decomposed into s×s different local regions through the Softmax function. Under the guidance of the objective function, through the Softmax operation, we obtain a vector U with the highest probability of belonging to the target to represent the region. Let c = s×s, M = {M1, M2, …, M c},Right now:
[0021]
[0022] (5) For the low-pass filter path, the entire input is feature extracted through convolution operation, and then the spatial resolution of the feature map is reduced through pooling operation, which retains the main feature information and ensures that the model can better focus on the structure of the overall image. Finally, the upsampling operation is used to restore the spatial resolution of the feature map so that the size of the output feature map matches the original heat map.
[0023]
[0024] (6) Combine the feature map obtained in the low-pass filter path We can do this by Perform element-wise multiplication with U to obtain the result of low-pass filtering guided by the heat map As shown in formula 5:
[0025]
[0026] (7) The input to the high-pass filter path is the original feature X and the output of the low-pass filter path. Subtract the result. As shown in formula 6:
[0027]
[0028] (8) In the high-pass filter path, a spatial attention mechanism is introduced, which emphasizes important areas in the image by weighting the target features while suppressing irrelevant texture information. Specifically, we perform mean and maximum operations to obtain high-pass features weighted by spatial attention. This operation enables the model to focus more specifically on the locations in the image that contain target information while weakening the influence of areas that only contain texture information. In this way, we achieve fine-tuning of the high-pass filter output, making it more focused on the key areas containing the target edges. As shown in Formula 7:
[0029]
[0030] Among them, σ(·) is the sigmiod activation function, is the concat operation, F is a 3×3 convolution, F Max (·) is the maximum value of the same position on the feature map in the channel dimension, F Mean (·) Take the average of the same position on the feature map in the channel dimension.
[0031] (9) The final output of the high-pass filter channel is as shown in Formula 8
[0032]
[0033] Among them, ⊙ is the point multiplication operation and F is the 3×3 convolution.
[0034] (10) Add the high-pass feature after spatial attention weighting to the output of the low-pass filter path after the heat map guidance path. As shown in Formula 9:
[0035]
[0036] Finally, the obtained feature Y is input into the segmentation head to obtain the category Class of the target object, the location bounding box Box of the target object and the mask Mask of the target, and load them on the image to be segmented, finally completing the instance segmentation task.
[0037] Advantages of the present invention:
[0038] (1) The present invention uses a heat map to guide the low-pass filter channel to filter out background noise in the image, realizing an adaptive filtering mechanism, which makes the model more inclined to focus on areas with high semantic strength. This helps to reduce background interference, allowing the model to focus more on the central part of the target, while also allowing the model to pay more attention to the overall contextual features, which can effectively solve the instance segmentation accuracy under complex background conditions;
[0039] (2) The present invention captures high-frequency information of the image, especially the edges and subtle geometric features of the target, through a high-pass filter channel. The model's attention to different areas of the image is adjusted by combining the spatial attention mechanism. This combination enables the model to more specifically enhance the perception of the target boundary, thereby improving the ability to capture detailed information, helping the model to better understand the geometric shape and structure of the target, and further effectively improving the instance segmentation ability under large-scale changes, making the model more robust;
[0040] (3) The present invention forms an overall instance segmentation network framework by combining three channels, namely, the heat map guidance channel, the low-pass filter channel and the high-pass filter channel. By integrating information of different frequencies, this method can capture the target features in the image more comprehensively and accurately, and provides an effective feature extraction method for visual tasks. This design can improve the model's understanding and generalization ability of target structure and context features in practical applications, and can effectively improve the instance segmentation effect in scenarios with complex background information, different lighting intensity, and large-scale changes. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] (1) Figure 1 It is a flow chart of the present invention.
[0042] (2) Figure 2 It is a comparison chart of the target detection method of the present invention and the Mask R-CNN and YOLOv8-seg models at mAP@0.5 in the embodiment.
[0043] (3) Figure 3 It is a comparison chart of the target detection method of the present invention and the Mask R-CNN and YOLOv8-seg models at mAP@0.5:0.95 in the embodiment. DETAILED DESCRIPTION
[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0045] See Figure 1 , a three-channel image instance segmentation method based on heat map guidance, specifically including the following steps:
[0046] (1) The input image to be detected is defined as an input image P; the image preprocessing is to perform multiple convolution processing on all training images through the backbone network; by analyzing related tasks and training based on existing data sets, a trained network is obtained, and the input image P is input into the trained network for instance segmentation.
[0047] (2) The specific steps of the thermal map guiding channel are:
[0048] S2.1, using the commonly used backbone network module to perform convolution operation on the input image P to extract features, and then after one convolution operation, the input feature X is obtained;
[0049] S2.2. Input feature X into the heat map guidance channel to obtain the heat map M(i,j) of the input feature:
[0050]
[0051] Where i, j, and n are the row index, column index, and number of channels of the feature, respectively.
[0052] S2.3. Decompose the heat map into background area S b 、Target high heat area S th and target low heat area S tl When M=S b +S th +S tl , in order to achieve better instance segmentation effect, make S b The regional high heat is small enough, while ensuring S th and S tl It can fully cover the target area, thereby guiding the characteristics of the low-pass filter path output Corresponding heat map Satisfies the following objective function:
[0053]
[0054] S2.4. Decompose the obtained heat map into s×s different local areas through the Softmax function. Under the guidance of the objective function, we obtain a vector U with the highest probability of belonging to the target to represent the area through the Softmax operation. Let c = s×s, M = {M1, M2, …, M c},Right now:
[0055]
[0056] (3) The specific steps of the low-pass filtering channel are:
[0057] S3.1. For the low-pass filter path, the entire input is extracted through convolution operation, and then the spatial resolution of the feature map is reduced through pooling operation, which retains the main feature information and ensures that the model can better focus on the structure of the overall image. Finally, the upsampling operation is used to restore the spatial resolution of the feature map so that the size of the output feature map matches the original heat map.
[0058]
[0059] S3.2. Combine the feature maps obtained in the low-pass filter path We can do this by Perform element-wise multiplication with U to obtain the result of low-pass filtering guided by the heat map As shown in formula 5:
[0060]
[0061] (4) The specific steps of the high-pass filter channel are:
[0062] S4.1. The input to the high-pass filter path is the original feature X and the output of the low-pass filter path. Subtract the result. As shown in formula 6:
[0063]
[0064] S4.2. In the high-pass filter path, a spatial attention mechanism is introduced, which emphasizes important areas in the image by weighting the target features while suppressing irrelevant texture information. Specifically, we perform mean and maximum operations to obtain high-pass features weighted by spatial attention. This operation enables the model to focus more specifically on the locations in the image that contain target information, while reducing the influence of areas that only contain texture information. In this way, we achieve fine-tuning of the high-pass filter output, making it more focused on key areas containing target edges. As shown in Formula 7:
[0065]
[0066] Among them, σ(·) is the sigmiod activation function, is the concat operation, F is a 3×3 convolution, F Max (·) is the maximum value of the same position on the feature map in the channel dimension, F Mean (·) Take the average of the same position on the feature map in the channel dimension.
[0067] S4.3, the final output of the high-pass filter channel is as shown in Formula 8
[0068]
[0069] Among them, ⊙ is the point multiplication operation and F is the 3×3 convolution.
[0070] (5) The high-pass feature after the spatial attention weighting is added to the result output by the low-pass filter path after the heat map guidance path. As shown in Formula 9:
[0071]
[0072] (6) The feature Y obtained after the three-channel processing is input into the segmentation head to obtain the category Class of the target object, the location bounding box Box of the target object and the mask Mask of the target, and load them on the image to be instance segmented, finally completing the instance segmentation task.
[0073] The detection accuracy of the instance segmentation method of the present invention (solid line) is compared with that of the Mask R-CNN model (dotted line) and the YOLOv8-seg model (dashed line), and the mAP@0.5 comparison chart is obtained ( Figure 2 ) and mAP@0.5:0.95 comparison chart ( Figure 3 );Depend on Figure 2 It can be seen that the mAP@0.5 of the present invention is 96.6% at 50 Epochs, the Mask R-CNN model is 90.4% at 50 Epochs, and the YOLOv8-seg model is 94.7% at 50 Epochs. Compared with Mask R-CNN and YOLOv8-seg, the accuracy of the present invention is improved by 6.2% and 1.9% respectively. Figure 3 It can be seen that the mAP@0.5:0.95 of the present invention is 69.3% at 50 Epochs, the MaskR-CNN model is 58.4% at 50 Epochs, and the YOLOv8-seg model is 64.9% at 50 Epochs. Compared with MaskR-CNN and YOLOv8-seg, the accuracy of the present invention is improved by 10.9% and 4.4%, respectively.
[0074] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A three-channel image instance segmentation method based on heat map guidance, characterized by: The specific steps include: (1) Perform image preprocessing and then use the convolution module in the backbone network to extract features from the image to obtain input features; (2) Using the heat map guidance channel to calculate the adaptive heat map guidance parameter vector; (3) After low-pass filtering the feature extraction image using a low-pass filter channel, adaptive feature fusion is performed with the adaptive heat map guidance parameter vector to obtain the features guided by the heat map; (4) The input feature map and the guided feature map are input into the high-pass filter guidance channel as a difference, and the spatial attention mechanism is used to make the model more focused on the target area in the image, so as to more effectively capture the important information of the target or task, while suppressing irrelevant texture information; (5) The output features of the low-pass filter channel guided by the heat map are added to the output features of the high-pass filter, and then the category, border position and mask are obtained through the segmentation head, and the instance segmentation result is obtained by superposition.
2. The method for three-channel image instance segmentation based on heat map guidance according to claim 1, characterized in that: The input image to be detected is defined as input image P; the image preprocessing is to perform multiple convolution processing on all training images through the backbone network; by analyzing related tasks and training based on existing data sets, a trained network is obtained, and the input image P is input to the trained network for instance segmentation.
3. The method for three-channel image instance segmentation based on heat map guidance according to claim 2, characterized in that: The specific steps of the heat map guiding channel are: S2.1, using the commonly used backbone network module to perform convolution operation on the input image P to extract features, and then after one convolution operation, the input feature X is obtained; S2.
2. Input feature X into the heat map guidance channel to obtain the heat map M(i,j) of the input feature: Where i, j, and n are the row index, column index, and number of channels of the feature, respectively. S2.
3. Decompose the heat map into background area S b 、Target high heat area S th and target low heat area S tl When M=S b +S th +S tl , in order to achieve better instance segmentation effect, make S b The regional high heat is small enough, while ensuring S th and S tl It can fully cover the target area, thereby guiding the characteristics of the low-pass filter path output Corresponding heat map Satisfies the following objective function: S2.
4. Decompose the obtained heat map into s×s different local areas through the Softmax function. Under the guidance of the objective function, we obtain a vector U with the highest probability of belonging to the target to represent the area through the Softmax operation. Let c = s×s, M = {M1, M2, …, M c },Right now:
4. The method for three-channel image instance segmentation based on heat map guidance according to claim 3, characterized in that: The specific steps of the low-pass filtering channel are: S3.
1. For the low-pass filter path, the entire input is extracted through convolution operation, and then the spatial resolution of the feature map is reduced through pooling operation, which retains the main feature information and ensures that the model can better focus on the structure of the overall image. Finally, the upsampling operation is used to restore the spatial resolution of the feature map so that the size of the output feature map matches the original heat map. S3.
2. Combine the feature maps obtained in the low-pass filter path We can do this by Perform element-wise multiplication with U to obtain the result of low-pass filtering guided by the heat map , as shown in Formula 5:
5. The method for three-channel image instance segmentation based on heat map guidance according to claim 4, characterized in that: The specific steps of the high-pass filtering channel are: S4.
1. The input to the high-pass filter path is the original feature X and the output of the low-pass filter path. Subtract the result. As shown in formula 6: S4.
2. In the high-pass filter path, a spatial attention mechanism is introduced, which emphasizes important areas in the image by weighting the target features while suppressing irrelevant texture information. Specifically, we perform mean and maximum operations to obtain high-pass features weighted by spatial attention. This operation enables the model to focus more specifically on the locations in the image that contain target information, while reducing the influence of areas that only contain texture information. In this way, we achieve fine-tuning of the high-pass filter output, making it more focused on key areas containing target edges. As shown in Formula 7: Among them, σ(·) is the sigmiod activation function, is the concat operation, F is a 3×3 convolution, F Max (·) is the maximum value of the same position on the feature map in the channel dimension, F Mean (·) Take the average of the same position on the feature map in the channel dimension. S4.3, the final output of the high-pass filter channel is as shown in Formula 8 Among them, ⊙ is the point multiplication operation and F is the 3×3 convolution.
6. The method for three-channel image instance segmentation based on heat map guidance according to claim 5, characterized in that: The high-pass feature after the spatial attention weighting is added to the result output by the low-pass filter path after the heat map guidance path. As shown in Formula 9:
7. The method for three-channel image instance segmentation based on heat map guidance according to claim 1, characterized in that: The feature Y obtained after the three-channel processing is input into the segmentation head to obtain the category Class of the target object, the location bounding box Box of the target object and the mask Mask of the target, and load them on the image to be instance segmented, finally completing the instance segmentation task.
Citation Information
Patent Citations
Landslide remote sensing information extraction method based on convolutional neural network and category thermodynamic diagram
CN113408462A
Abdomen multi-organ CT image segmentation method based on eye tracker
CN116433697A
Image sorting apparatus
US20110069896A1
Image segmentation method and apparatus for panoramic image, and computer device and storage medium
WO2023066143A1
