Image semantic segmentation method based on cross-dimension multispectral edge fusion

Abstract

The invention discloses an image semantic segmentation method based on cross-dimension multispectral edge fusion. The method comprises the following steps: firstly, constructing a bilateral multi-modal fusion encoder based on hierarchical attention, and extracting low-level and high-level features in an RGB image and a thermal infrared image; secondly, introducing an edge detection branch (namely a jumping edge guide structure), and enhancing the utilization of global semantic information; and finally, designing a deep supervision mechanism oriented to a plurality of loss functions, and supervising the network with the plurality of loss functions. According to the method provided by the invention, the RGB image and the thermal infrared image are used as input, various attributes and interaction between low-level and high-level features in a dynamic traffic scene are comprehensively considered, and the problems of information loss and the like are solved by fusing semantic and edge features, so that accurate and robust semantic segmentation of the RGB-thermal infrared image under a complex illumination condition is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent connected vehicle environment perception, and in particular relates to an image semantic segmentation method based on cross-dimensional multi-spectral edge fusion. Background Art

[0002] Traffic safety is a key topic in social research and has attracted widespread attention. In order to alleviate the pressure and adverse effects of traffic congestion, intelligent vehicles have become the focus. In complex intelligent driving systems, acquiring surrounding information and achieving scene understanding are the basic foundations for realizing various advanced functions, such as semantic mapping, decision-making, and safe and efficient navigation. Semantic segmentation classifies each pixel in an image and has become a crucial computer vision task, allowing for a deeper understanding of the driving environment.

[0003] Convolutional neural networks have attracted widespread attention due to their outstanding achievements in semantic segmentation and are widely used in intelligent vehicles. Most of these convolutional neural network-driven techniques rely primarily on visual cues. However, despite their significant progress in visible light images, unimodal sensory approaches have been shown to be insufficient to understand scenes captured under different lighting and weather conditions or in complex backgrounds. Specifically, images captured in the dark, with glare, or in shadows are usually of poor quality, resulting in decreased semantic segmentation accuracy. In addition, low visibility and poor lighting conditions at night hinder the ability of drivers and pedestrians to accurately observe road conditions.

[0004] To this end, depth data is used to provide three-dimensional geometric information for two-dimensional RGB images, which improves the image segmentation accuracy to a certain extent. However, depth information may be insufficient in some scenarios. First, the depth data provided by the lidar sensor is sparse and uneven, and cannot be accurately aligned pixel by pixel with the dense semantic information of the image, thus limiting the utilization of the rich content of the image; second, the depth data captured by the depth camera may become blurred during high-speed motion, and its time-of-flight measurement may encounter challenges when processing multiple reflections. Therefore, the depth information lacks robustness and reliability in the case of insufficient lighting or cluttered background.

[0005] Thermal infrared cameras offer a unique way to overcome depth mapping constraints by capturing the infrared heat energy emitted by objects, eliminating the reliance on external lighting. Such cameras excel at imaging objects in challenging lighting conditions, from nighttime scenes to bright lights and shadows. They are particularly good at observing objects whose temperature exceeds that of their surroundings, such as vehicles and pedestrians. However, compared to color images, thermal infrared images typically lack color and texture details, have low contrast, and are subject to cluttered noise.

[0006] At present, there are papers and patents focusing on the semantic segmentation technology of RGB-thermal infrared images. However, the contributions of features of different modalities to semantic segmentation are not the same. How to give full play to the advantages of RGB image and thermal image features and effectively fuse cross-modal features is one of the key issues that need to be urgently addressed in the field of environmental perception of intelligent connected vehicles. In general, the existing semantic segmentation methods of RGB-thermal infrared images do not fully consider the various attributes and interactions between low-level and high-level features in the image feature fusion stage, and it is difficult to effectively solve the problem of poor semantic segmentation performance caused by the loss of image feature information under complex lighting conditions. Summary of the invention

[0007] In order to solve the problem of poor semantic segmentation performance caused by the loss of image feature information under complex lighting conditions, the present invention discloses an image semantic segmentation method based on cross-dimensional multi-spectral edge fusion.

[0008] In order to achieve the above-mentioned purpose of the present invention, the present invention provides an image semantic segmentation method based on cross-dimensional multispectral edge fusion. First, a bilateral multimodal fusion encoder based on hierarchical attention is constructed to extract low-level and high-level features in the "RGB-thermal infrared" modality. Secondly, a bilateral multimodal fusion encoder based on hierarchical attention is constructed to extract low-level and high-level features in RGB images and thermal infrared images; secondly, an edge detection branch is introduced, and the constructed jump edge guidance structure is used to enhance the utilization of global semantic information; finally, a deep supervision mechanism for multiple loss functions is designed to supervise the network with multiple loss functions; it is characterized in that it includes the following steps:

[0009] Step 1: Construct a bilateral multimodal fusion encoder;

[0010] First, RGB image features at four different scales are taken as input, and a separate convolution module is used to calculate the weights associated with the features at each scale; it is worth noting that the shape of the resulting weights is the same as that of the original features; subsequently, these weights are assigned to the corresponding features by element-wise multiplication; this strategic combination of attention-based feature fusion aims to enhance the adaptability and discrimination of the model by allowing the model to selectively emphasize or de-emphasize specific features at different scales, thereby optimizing the fusion process; the process can be expressed as:

[0011]

[0012] Among them, F' ci Represents the RGB image features of the i-th scale after feature fusion;

[0013] F ci Represents the RGB image features of the i-th scale;

[0014] f() represents the attention-based feature fusion function;

[0015] ConvA(·) represents a convolution operation with activation;

[0016] ⊙ represents element-by-element multiplication;

[0017] σ() represents the sigmoid function;

[0018] After that, the RGB image feature F' selected after feature fusion ci Added to thermal image feature F ti Get the fusion feature F fi As input to the next layer;

[0019] Finally, the four-scale RGB image features F' are integrated by fusion skip connection c Connect to the last decoder (layer); thus achieving semantic segmentation of the image. In this way, we can propagate the error gradient directly back to the feature fusion module to achieve direct supervision;

[0020] Step 2: Introduce the edge detection branch, which is also part of the encoder;

[0021] The thermal image is input into the semantic branch and the edge detection branch respectively. First, semantics are extracted by the first-level semantic branch and edge is extracted by the first-level edge detection branch. Then, the semantic and edge features are fused by the dual attention feature enhancement module.

[0022] Then, semantics are extracted through the secondary semantic branch, and edge extraction is performed through the secondary edge detection branch; semantic and edge features are fused through the dual attention feature enhancement module;

[0023] Then, semantics is extracted through the three-level semantic branch, and edge is extracted through the three-level edge detection branch; semantic and edge features are fused through the dual attention feature enhancement module;

[0024] The introduced edge detection branch is a skip edge guided structure, which fuses the global edge information through multi-level skip connections, which not only completes the global information but also enhances the network's attention to the target edge information.

[0025] The fusion of semantic and edge features through the dual attention feature enhancement module includes:

[0026] The intermediate feature F of the semantic branch s , the intermediate feature F of the edge branch e and input data F inInput the dual attention feature enhancement module; when the first-level semantic branch performs semantic extraction and the first-level edge detection branch performs edge extraction, input data F in is a thermal image; when the secondary semantic branch performs semantic extraction and the secondary edge detection branch performs edge extraction; input data F in is the output of the first dual attention feature enhancement module; when the three-level semantic branch performs semantic extraction and the three-level edge detection branch performs edge extraction; the input data F in is the output of the second dual attention feature enhancement module;

[0027] First, F s and F e Reshape the size to get the feature and Features Respectively represent F s and F e An estimated value of

[0028] Then, we fuse the multi-head cross attention (MHCA) and layer normalization (LN) and To generate feature F CA1 ; F in Reshape the size to get Indicates F in An estimated value of

[0029] Then, the multi-head cross attention (MHCA) and layer normalization (LN) are fused. and F CA1 To generate feature F CA2 ; F CA2 The fused feature F is obtained by processing and resizing it with a multi-layer perceptron (MLP) and layer normalization (LN). DAFEB ;

[0030] The formula expression is as follows:

[0031]

[0032] Among them, F DAFEB It is the input of the two branches of the third layer and the second DAFEB.

[0033] R() is reshape size;

[0034] LN() is layer normalization;

[0035] MLP() is a multi-layer perceptron;

[0036] MHCA() is the long cross attention;

[0037] F sis the intermediate feature from the semantic branch;

[0038] F e are intermediate features from marginal branches;

[0039] F in is the input data;

[0040] The dual attention feature enhancement module can not only fuse semantic and edge features, but also enhance features by introducing information before the pooling operation; the fusion process follows the direction from low resolution to high resolution, thereby supplementing the corresponding details; these features cover different scales and different semantic levels; it not only enhances the network's attention to the edges of objects, but also supplements the useful information of global edge features to objects; in this way, edges with semantic information can improve the ability to distinguish between categories, especially on similar external features of adjacent objects, and can strengthen the optimization of semantic features;

[0041] Step 3: Design a deep supervision mechanism for multiple loss functions; supervise the output of all decoded features and the fusion output by constructing multiple loss functions;

[0042] First, considering the large pixel differences of the data set used in training the network, the present invention uses a weighted cross entropy function to train the entire network. The calculation formula of the weighted loss function can be expressed as:

[0043]

[0044] Among them, C represents the total number of categories;

[0045] y i,c and x i,c Represents the target label and predicted probability of category c and the i-th pixel;

[0046] N represents the number of pixels of samples sent to the network for training each time;

[0047] w c represents the weight of category c;

[0048] The loss function L of the fusion output fuse Calculated as:

[0049]

[0050] Among them, d i represents the decoding feature;

[0051] w fuse represents the fusion output weight;

[0052] The overall loss function of the network is:

[0053]

[0054] Among them, x m represents the predicted output of each decoder output layer;

[0055] M represents the number of decoding stages, and in the present invention, M=4.

[0056] Compared with general methods, the semantic segmentation method proposed in this paper takes RGB images and thermal infrared images as input, comprehensively considers various properties and interactions between low-level and high-level features in dynamic traffic scenes, and realizes accurate and robust semantic segmentation of RGB-thermal infrared images under complex lighting conditions by fusing semantic and edge features and solving problems such as information loss. Specifically, it is embodied in:

[0057] (1) The method proposed in this paper takes RGB images and thermal infrared images as input, comprehensively considers various properties and interactions between low-level and high-level features, and effectively performs multi-level feature fusion and multi-type feature aggregation, thus realizing robust semantic segmentation in dynamic traffic scenarios.

[0058] (2) The bilateral multimodal fusion encoder based on hierarchical attention constructed in the present invention can dynamically reweight and select effective features of different scales, thereby optimizing the fusion process and realizing the comprehensive extraction of multi-scale multimodal features.

[0059] (3) The skip edge guided structure based on multi-head cross attention constructed by the present invention not only integrates semantic and edge features, but also solves the problem of information loss, further improving the accuracy of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a schematic diagram of the technical route of the present invention;

[0061] Figure 2 It is a bilateral multi-modal fusion encoder structure designed by the present invention;

[0062] Figure 3 It is a hierarchical attention-based feature fusion module structure constructed by the present invention;

[0063] Figure 4 It is a jumping edge guiding structure constructed by the present invention;

[0064] Figure 5 It is the structure of the dual attention feature enhancement block constructed by the present invention. DETAILED DESCRIPTION

[0065] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0066] The present invention discloses an image semantic segmentation method based on cross-dimensional multi-spectral edge fusion, such as Figure 1 As shown. First, a bilateral multimodal fusion encoder based on hierarchical attention is constructed to extract low-level and high-level features in RGB images and thermal infrared images; secondly, an edge detection branch (i.e., a jump edge guided structure) is introduced to strengthen the use of global semantic information, which is also used in the feature extraction part; finally, a deep supervision mechanism for multiple loss functions is designed to supervise the network with multiple loss functions; it is characterized by comprising the following steps:

[0067] Step 1: Build a bilateral multimodal fusion encoder

[0068] In the encoding stage, two identical improved ResNet modules are used as the feature extraction backbone networks for RGB images and thermal images, namely, RGB encoder and thermal image encoder. Specifically, in order to reduce the loss of information features and the number of network parameters, the ResNet18 variant structure is used and its last fully connected layer is deleted; Figure 2 As shown;

[0069] Considering the inherent sensitivity of RGB images to environmental conditions, if the features of RGB images are directly fused, it may have a potential impact on the overall segmentation performance of the network;

[0070] To this end, the present invention constructs a bilateral multimodal fusion method to selectively fuse the effective features in the RGB image with the effective features in the thermal image; considering that the effects of RGB image features on different regions and channels are different, in order to optimize the feature fusion of RGB images and thermal images and improve the performance of semantic segmentation, the present invention adopts a pixel-level evaluation method to identify its subtle effects on the results;

[0071] In order to achieve dynamic re-weighting and effective feature selection, this paper introduces a feature fusion module based on hierarchical attention, such as Figure 3As shown in the figure, it takes RGB image features of four different scales (e.g., resolution) as input and uses a separate convolution module to calculate the weights associated with the features of each scale. It is worth noting that the shape of the resulting weights is the same as that of the original features. Subsequently, these weights are assigned to the corresponding features by element-by-element multiplication. This attention-based feature fusion strategy aims to enhance the adaptability and discrimination of the model by allowing the model to selectively emphasize or de-emphasize specific features at different scales, thereby optimizing the fusion process. The process can be expressed as:

[0072]

[0073] Among them, F' ci Represents the RGB image features of the i-th scale after feature fusion;

[0074] F ci Represents the RGB image features of the i-th scale;

[0075] f() represents the attention-based feature fusion function;

[0076] ConvA(·) represents a convolution operation with activation;

[0077] ⊙ represents element-by-element multiplication;

[0078] σ() represents the sigmoid function;

[0079] After that, the RGB image feature F' selected after feature fusion ci Add to (add) thermal image feature F ti Get the fusion feature F fi As input to the next layer;

[0080] Finally, the four-scale RGB image features F' are integrated by fusion skip connection c Connect to the last decoder (layer); in this way, we can make the error gradient propagate directly back to the feature fusion module to achieve direct supervision;

[0081] Step 2: Introduce edge detection branch and build a skip edge guidance structure (Skip Edge Guidance Structure)

[0082] Semantic segmentation generally consists of two parts: encoder and decoder. This is a part of the encoder.

[0083] Redundant information of the background will introduce noise in the feature re-aggregation process; fortunately, edge information can handle this noise and make up for the information loss caused by downsampling; considering this, we propose an edge detection branch that integrates global edge information into the semantic segmentation branch to ensure that relevant contextual features are fully explored and utilized while preserving edge details, thereby alleviating noise and optimizing object extraction in the context of semantic segmentation;

[0084] However, the traditional edge enhancement module mainly focuses on aggregating features of the same scale; although such methods are effective, they may lack comprehensiveness and limit the space and flexibility of feature aggregation; in addition, in the traditional edge enhancement module, repeated pooling operations may lead to the loss of spatial information, which is more serious when detecting edges; the lost information cannot be easily restored by skip connections; in order to enhance the learning ability of global information, the present invention introduces an edge detection branch, that is, a Skip Edge Guidance (SEG) structure is designed, so Figure 4 As shown in the figure, it fuses global edge information through multi-level skip connections, which not only completes the global information but also enhances the network's attention to the target edge information.

[0085] While taking into account semantic segmentation performance and efficiency, the present invention designs Figure 4 The three dual attention feature enhancement blocks (DAFEBs) shown in Figure 2 are shown in Figure 2; DAFEBs can not only fuse semantic and edge features, but also enhance features by introducing information before the pooling operation; it should be emphasized that the three DAFEBs have the same structural design but do not share parameters; they have inputs and outputs of different sizes; here, we take the first DAFEB (such as Figure 5 The process is described as follows:

[0086]

[0087] Among them, F DAFEB It is the input of the two branches of the third layer and the second DAFEB.

[0088] R() is reshape size;

[0089] LN() is layer normalization;

[0090] MLP() is a multi-layer perceptron;

[0091] MHCA() is the long cross attention;

[0092] F s is the intermediate feature from the semantic branch;

[0093] F e are intermediate features from marginal branches;

[0094] F in is the input data;

[0095] The input is the intermediate feature F of the semantic branch s , the intermediate feature F of the edge branch e and input data F in . F s and F e The size of and is resized to Features and Features Respectively represent F s and F e Estimation of; fused by multi-head cross attention (MHCA) and layer normalization (LN) and To generate feature F CA1 ; Set the F of size [w×h×1] in is reshaped to size of Indicates F in The estimated value of is then fused through multi-head cross attention (MHCA) and layer normalization (LN) and F CA1 To generate feature F CA2 ; F CA2 Processed by a multi-layer perceptron (MLP) and layer normalization (LN), and reshaped to F DAFEB ; F DAFEB It is the input of the two branches of the third layer and the second DAFEB (F in );

[0096] The fusion process follows the direction from low resolution to high resolution, thus supplementing the corresponding details; these features cover different scales and different semantic levels; it not only enhances the network's attention to the edges of objects, but also supplements the useful information of global edge features to objects; in this way, edges with semantic information can improve the ability to distinguish between categories, especially on similar external features of adjacent objects, and can strengthen the optimization of semantic features;

[0097] Step 3: Design a deep supervision mechanism for multiple loss functions

[0098] In the decoding process, there are 4 decoded feature outputs; the traditional encoder-based method mainly uses the last feature for prediction, which is difficult to meet the requirements of expected multi-feature output and cannot adapt to segmentation targets of different scales; considering this, we use all features in the decoding process and add an output layer after each decoder output to obtain the segmentation output of the network; specifically,

[0099] First, in order to restore the resolution to the image input size, the decoded features of each layer are interpolated; second, the number of output channels is adjusted using 1×1 convolution; third, all decoded outputs are fused together and refined through 3×3 convolution;

[0100] In order to effectively supervise the entire network, the present invention designs a deep fusion feature supervision method; specifically, multiple loss functions are constructed to supervise the output of all decoded features and the fusion output; considering that the pixel differences of the data set used in training the network are large, the present invention uses a weighted cross entropy function to train the entire network, and the calculation formula of the weighted loss function can be expressed as:

[0101]

[0102] Among them, C represents the total number of categories;

[0103] y i,c and x i,c Represents the target label and predicted probability of category c and the i-th pixel;

[0104] N represents the number of pixels of samples sent to the network for training each time;

[0105] w c represents the weight of category c;

[0106] The loss function L of the fusion output fuse Calculated as:

[0107]

[0108] Among them, d i represents the decoding feature;

[0109] w fuse represents the fusion output weight;

[0110] The overall loss function of the network is:

[0111]

[0112] Among them, x m represents the predicted output of each decoder output layer;

[0113] M represents the number of decoding stages, and in the present invention, M=4.

Claims

1. The present invention discloses an image semantic segmentation method based on cross-dimensional multispectral edge fusion; firstly, a bilateral multimodal fusion encoder based on hierarchical attention is constructed to extract features from RGB images and thermal infrared images; Secondly, an edge detection branch is introduced to enhance the use of global semantic information by using the constructed jump edge guidance structure; finally, a deep supervision mechanism for multiple loss functions is designed to supervise the network with multiple loss functions; its characteristics are as follows: The following steps are involved: Step 1: Construct a bilateral multimodal fusion encoder; First, RGB image features at four different scales are taken as input, and separate convolution modules are used to calculate the weights associated with the features at each scale; then, these weights are assigned to the corresponding features through element-wise multiplication; The process is expressed as: Among them, F' ci Represents the RGB image features of the i-th scale after feature fusion; F ci Represents the RGB image features of the i-th scale; f() represents the attention-based feature fusion function; ConvA(·) represents a convolution operation with activation; ⊙ represents element-by-element multiplication; σ() represents the sigmoid function; After that, the selected RGB image feature F' after feature fusion ci Added to thermal image feature F ti Get the fusion feature F fi As input to the next layer; Finally, the four-scale RGB image features F' are integrated by fusion skip connection c Connect to the last decoder; Step 2: Introduce edge detection branch; The thermal image is input into the semantic branch and the edge detection branch respectively. First, semantics are extracted by the first-level semantic branch and edge is extracted by the first-level edge detection branch. Then, the semantic and edge features are fused by the dual attention feature enhancement module. Then, semantics are extracted through the secondary semantic branch, and edge extraction is performed through the secondary edge detection branch; semantic and edge features are fused through the dual attention feature enhancement module; Then, semantics is extracted through the three-level semantic branch, and edge is extracted through the three-level edge detection branch; semantic and edge features are fused through the dual attention feature enhancement module; The fusion of semantic and edge features through the dual attention feature enhancement module includes: The intermediate feature F of the semantic branch s , the intermediate feature F of the edge branch e and input data F in Input the dual attention feature enhancement module; when the first-level semantic branch performs semantic extraction and the first-level edge detection branch performs edge extraction, input data F in is a thermal image; when the secondary semantic branch performs semantic extraction and the secondary edge detection branch performs edge extraction; input data F in is the output of the first dual attention feature enhancement module; when the three-level semantic branch performs semantic extraction and the three-level edge detection branch performs edge extraction; the input data F in is the output of the second dual attention feature enhancement module; First, F s and F e Reshape the size to get the feature and Features Respectively represent F s and F e An estimated value of Then, through multi-head cross attention and layer normalization fusion and To generate feature F CA1 ; F in Reshape the size to get Indicates F in An estimated value of Then, multi-head cross attention and layer normalization fusion and F CA1 To generate feature F CA2 ; F CA2 The fused feature F is obtained by multi-layer perceptron and layer normalization and resizing. DAFEB ; Step 3: Design a deep supervision mechanism for multiple loss functions; First, considering the large pixel differences of the data set used in training the network, the present invention uses a weighted cross entropy function to train the entire network. The calculation formula of the weighted loss function can be expressed as: Among them, C represents the total number of categories; y i,c and x i,c Represents the target label and predicted probability of category c and the i-th pixel; N represents the number of pixels of samples sent to the network for training each time; w c represents the weight of category c; The loss function L of the fusion output fuse Calculated as: Among them, d i represents the decoding feature; w fuse represents the fusion output weight; The overall loss function of the network is: Among them, x m represents the predicted output of each decoder output layer; M represents the number of decoding stages.

Images