A multi-modal drivable road segmentation method and system based on boundary perception

Abstract

The application discloses a kind of based on boundary perception multi-modal feasible road segmentation method and system, method includes: by feasible road image extraction output feature input feature pyramid transmission module different branch, processing and transmission;By cross-modal fusion module, the rough prediction area map and rough prediction boundary map of road image are obtained;Input feature is input into boundary refinement perception module, and road image segmentation model is established;Image obtained by cross-modal fusion module and boundary refinement perception module is sampled to original image size, and loss function is constructed to calculate error respectively, and the road image segmentation model is updated;Feasible road image is input to the updated model, and the road image segmentation prediction map is output.The application expands receptive field and enhances the semantic consistency between different levels, enhances the capture ability of multi-scale target, effectively reduces the inaccurate boundary of image due to the influence of different scales and perspective difference, significantly improves the precision of image segmentation.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and digital image processing, and particularly to a multimodal feasible road segmentation method and system based on boundary awareness. Background Technology

[0002] Image segmentation technology classifies pixels within an image, breaking it down into different regions with various characteristics, and then extracts the target of interest from these regions. This technology has wide applications in various fields such as traffic control systems, facial recognition, and satellite image analysis.

[0003] Semantic segmentation of road images is a key task in computer vision, aiming to segment road images into multiple meaningful regions, such as roads, vehicles, and pedestrians. It has wide applications in autonomous driving, traffic monitoring, and intelligent transportation systems. However, several significant challenges exist in road image semantic segmentation research. First, there is the class imbalance problem: the number of pixels of different classes in road images is often unbalanced, affecting the performance of the trained model. Second, there is the segmentation of small objects and details. Small objects and details have fewer pixels in images, which may lead to them being ignored or inaccurately identified in low-resolution images. Furthermore, multi-scale and multi-viewpoint issues are also significant challenges. Road images typically contain objects at multiple scales and from multiple viewpoints, and this scale variation also impacts the performance of semantic segmentation models. Existing neural networks learn to segment images by training on existing images. Methods that fuse boundary features are widely used in segmentation tasks, but directly applying them to feasible road segmentation is problematic. Due to the complexity of object types in the scene and the significant differences in scale and viewpoint, inaccurate boundaries can easily lead to the model segmenting incorrect regions. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides a boundary-aware multimodal feasible road segmentation method to address the problem of incorrect region segmentation caused by significant differences in image scale and viewpoint.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a boundary-aware multimodal feasible road segmentation method, comprising:

[0008] Output features are extracted from feasible road images, and these output features are input into different branches of the feature pyramid transfer module for processing and transfer at different levels.

[0009] By fusing the outputs from the processing layer and the feature pyramid transfer module through the cross-modal fusion module, a coarse prediction region map and a coarse prediction boundary map of the road image are obtained.

[0010] The outputs of the cross-modal fusion module and the feature pyramid transfer module are used as input features to the boundary refinement perception module to establish a road image segmentation model;

[0011] The images obtained from the cross-modal fusion module and the boundary refinement perception module are sampled to the original image size, and loss functions are constructed to calculate the errors respectively. The obtained errors are then used to update the road image segmentation model in reverse.

[0012] Input feasible road images into the updated model, obtain and output the corresponding road image segmentation prediction map.

[0013] As a preferred embodiment of the boundary-aware multimodal feasible road segmentation method described in this invention, the method involves: extracting output features from feasible road images; these output features are then input into different branches of a feature pyramid transfer module for processing and transfer at different levels, including...

[0014] Remove the last fully connected layer in the Swin-B_224 backbone network, and pass the RGB image f0 through the Swin-B_224 backbone network to obtain the output features of each layer.

[0015] Remove the last fully connected layer from the ResNet-18 backbone network, and pass the RGB image f0 through the ResNet-18 backbone network to obtain the output features of each layer.

[0016] As a preferred embodiment of the boundary-aware multimodal feasible road segmentation method described in this invention, it further includes:

[0017] The output of the fourth layer of the ResNet-18 backbone network In the first branch of the input feature pyramid transfer module, the output of the first branch is obtained through a multi-head self-attention module that applies linear transformation and attention calculation, followed by concatenation, compression, and recalibration.

[0018] The output of the third layer of the ResNet-18 backbone network In the second branch of the input feature pyramid transfer module, the output of the second branch is obtained through a multi-head self-attention module that applies linear transformation and attention calculation, followed by concatenation, compression, and recalibration.

[0019] The output of the second layer of the ResNet-18 backbone network In the third branch of the input feature pyramid transfer module, the output of the third branch is obtained through a multi-head self-attention module that applies linear transformation and attention calculation, followed by concatenation, compression, and recalibration.

[0020] The output of the first layer of the ResNet-18 backbone network In the fourth branch of the input feature pyramid transfer module, the output of the fourth branch is obtained through a multi-head self-attention module that applies linear transformation and attention calculation, followed by concatenation, compression, and recalibration.

[0021] As a preferred embodiment of the boundary-aware multimodal feasible road segmentation method described in this invention, the method involves fusing the outputs from the processing layer and the feature pyramid transfer module via a cross-modal fusion module to obtain a coarse predicted region map and a coarse predicted boundary map of the road image, including:

[0022] The output of the Swin-B_224 backbone network and Input to the first preprocessing layer, and The output is obtained by concatenating convolutions, batch normalization, and ReLU activation.

[0023] Output the second layer of the same-sized Swin-B_224 backbone network. With the output of the fourth layer of the ResNet-18 backbone network Input to the second preprocessing layer, the With the The output is obtained by concatenating convolutions, batch normalization, and ReLU activation.

[0024] The output and output Input to the third preprocessing layer, the With the The output f is obtained by concatenating convolutions, batch normalization, and ReLU activation. CF ;

[0025] Output the second branch of the feature pyramid transfer module and the first layer output of the Swin-B_224 backbone network Input cross-modal fusion module, and After each element is convolved, the corresponding elements are summed, and then the result is fed into a hierarchical split convolutional layer to obtain the output.

[0026] The and the f CF Input region prediction branch concatenation convolution obtains a coarse prediction region map The Input boundary prediction branch convolution obtains a coarse prediction boundary map

[0027] As a preferred embodiment of the boundary-aware multimodal feasible road segmentation method described in this invention, the output of the cross-modal fusion module and the output of the feature pyramid transfer module are used as input features to the boundary refinement perception module to establish a road image segmentation model, including:

[0028] Roughly predicted area map Roughly predict the boundary map and the third branch output of the feature pyramid transfer module The first module of the input boundary refinement perception is used to obtain the first region map and the first boundary map.

[0029] The process expression is:

[0030]

[0031]

[0032]

[0033]

[0034]

[0035]

[0036] Here, Up(·,·) represents upsampling the first input to the same size as the second input, Conv() represents convolving the data, CatConv(·,·) represents concatenating the two inputs along the channel dimension and then convolving them, and RA(·,·) represents performing inverse attention operations on the input. For a rough prediction of the area map, This is the output of the third branch of the feature pyramid transfer module. for and The values ​​are obtained through sampling, convolution, anti-attention operations, and concatenation of convolution processes. for and By concatenating the values ​​from the convolution process for and By concatenating the values ​​from the convolution process for and splicing convolution values This is the first region map. This is the first boundary diagram;

[0037] The first region map of the first module is refined to perceive the boundary. First boundary diagram and the fourth branch output of the feature pyramid transfer module The second module of the input boundary refinement perception is used to obtain the second region map and the second boundary map.

[0038] The process expression is:

[0039]

[0040]

[0041]

[0042]

[0043]

[0044]

[0045] Here, Up(·,·) represents upsampling the first input to the same size as the second input, Conv() represents convolving the data, CatConv(·,·) represents concatenating the two inputs along the channel dimension and then convolving them, and RA(·,·) represents performing inverse attention operations on the input. This is the first region map. This is the output of the fourth branch of the feature pyramid transfer module. for and The values ​​are obtained through sampling, convolution, anti-attention operations, and concatenation of the convolution process. for and By concatenating the values ​​from the convolution process for and By concatenating the values ​​from the convolution process for and splicing convolution values This is the second region map. This is the second boundary diagram.

[0046] As a preferred embodiment of the boundary-aware multimodal feasible road segmentation method described in this invention, the method includes: sampling the images obtained from the cross-modal fusion module and the boundary refinement perception module to the original image size, constructing loss functions to calculate the errors respectively, and using the obtained errors to back-update the road image segmentation model, including:

[0047] The expression for sampling to the original image size is:

[0048]

[0049]

[0050] in, This represents the predicted region map of the i-th layer after upsampling. Let f0 represent the predicted boundary map of the i-th layer after upsampling, CF represent the prediction result from the cross-modal fusion module, BA1 represent the prediction result from the first boundary refinement perception module, BA2 represent the prediction result from the second boundary refinement perception module, and f0 represent the RGB image.

[0051] The loss function expression is:

[0052] L bc =l bce +l iou

[0053] Among them, l bce Let l represent the binary cross-entropy function. iou L represents the cross-joint function. bce+iou The error between the predicted region map and the ground truth map, and the error between the predicted boundary map and the boundary map of the ground truth map;

[0054] The errors of the coarse predicted region map and coarse predicted boundary map, the first region map and the first boundary map, the second region map and the second boundary map obtained by the loss function and the actual labeled map are weighted and summed, and the resulting errors are used to update the road image segmentation model in reverse.

[0055] As a preferred embodiment of the boundary-aware multimodal feasible road segmentation method described in this invention, the method includes: inputting a feasible road image into the updated model, obtaining and outputting the corresponding road image segmentation prediction map, including:

[0056] The coarse prediction region map, the first region map, and the second region map are activated by a Sigmoid layer and then output to obtain the feasible road image segmentation prediction map.

[0057] Secondly, this invention provides a system for boundary-aware multimodal feasible road segmentation, comprising,

[0058] The feature extraction module is used to extract output features from feasible road images. The output features are input into different branches of the feature pyramid transfer module for processing and transfer at different levels.

[0059] The cross-modal fusion module is used to fuse the outputs from the processing layer and the feature pyramid transfer module to obtain a coarse prediction region map and a coarse prediction boundary map of the road image.

[0060] The boundary refinement perception module is used to input the output of the cross-modal fusion module and the output of the feature pyramid transfer module as input features to the boundary refinement perception module to establish a road image segmentation model.

[0061] The prediction module is used to sample the coarse prediction region map and coarse prediction boundary map, the first region map and the first boundary map, the second region map and the second boundary map to the original image size, construct a loss function to calculate the error respectively, use the obtained error to update the road image segmentation model in reverse, input the feasible road image into the updated model, and obtain and output the corresponding road image segmentation prediction map.

[0062] Thirdly, the present invention provides a computing device, comprising:

[0063] Memory and processor;

[0064] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the boundary-aware multimodal feasible road segmentation method.

[0065] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the boundary-aware multimodal feasible road segmentation method.

[0066] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention expands the receptive field and enhances the semantic consistency between different layers, improves the problem of semantic information inconsistency between different layers in the ResNet backbone network caused by feature size compression, and enhances the ability to capture multi-scale targets; in the face of complex scenes, it effectively reduces the problem of the model segmenting incorrect regions due to inaccurate boundaries obtained by the image due to the influence of different scales and viewpoints, and significantly improves the accuracy of image segmentation. Attached Figure Description

[0067] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0068] Figure 1 This is a schematic diagram of the general flow of a boundary-aware multimodal feasible road segmentation method according to an embodiment of the present invention;

[0069] Figure 2 This is a schematic diagram of the overall process of a boundary-aware multimodal feasible road segmentation method according to an embodiment of the present invention. Detailed Implementation

[0070] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0071] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0072] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0073] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0074] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the system or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0075] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0076] Example 1

[0077] Reference Figure 1 As an embodiment of the present invention, a boundary-aware multimodal feasible road segmentation method is provided, comprising:

[0078] S1: Extract output features from feasible road images, and input the output features into different branches of the feature pyramid transfer module for processing and transfer at different levels;

[0079] Furthermore, by removing the last fully connected layer in the Swin-B_224 backbone network, and passing the RGB image f0 through the Swin-B_224 backbone network, the output features of each layer are obtained.

[0080] Remove the last fully connected layer from the ResNet-18 backbone network, and pass the RGB image f0 through the ResNet-18 backbone network to obtain the output features of each layer.

[0081] The output of the fourth layer of the ResNet-18 backbone network In the first branch of the input feature pyramid transfer module, the output of the first branch is obtained through a multi-head self-attention module that applies linear transformation and attention calculation, followed by concatenation, compression, and recalibration.

[0082] Preferably, the process can be expressed as a formula:

[0083]

[0084]

[0085] Where MHSA(·) represents Multi-headed Self-attention, UpCat(·,·) represents upsampling the second input to the same size as the first input and concatenating it along the channel dimension of the first input; DASPP(·) represents Dense Atrous Spatial Pyramid Pooling, and SE(·) represents a compressed activation layer. for By applying linear transformations and attention calculations through a multi-head self-attention module, the values ​​are then concatenated, convolved, compressed, and recalibrated. This is the output of the first branch of the feature pyramid transfer module;

[0086] The output of the third layer of the ResNet-18 backbone network In the second branch of the input feature pyramid transfer module, the output of the second branch is obtained through a multi-head self-attention module that applies linear transformation and attention calculation, followed by concatenation, compression, and recalibration.

[0087] Preferably, the process can be expressed as a formula:

[0088]

[0089]

[0090] Where MHSA(·) represents Multi-headed Self-attention, UpCat(·,·) represents upsampling the second input to the same size as the first input and concatenating it along the channel dimension of the first input; DASPP(·) represents Dense Atrous Spatial Pyramid Pooling, and SE(·) represents a compressed activation layer. for By applying linear transformations and attention calculations through a multi-head self-attention module, the values ​​are then concatenated, convolved, compressed, and recalibrated. This is the output of the second branch of the feature pyramid transfer module;

[0091] The output of the second layer of the ResNet-18 backbone network In the third branch of the input feature pyramid transfer module, the output of the third branch is obtained through a multi-head self-attention module that applies linear transformation and attention calculation, followed by concatenation, compression, and recalibration.

[0092] Preferably, the process can be expressed as a formula:

[0093]

[0094]

[0095] In this context, MHSA(·) represents Multi-headed Self-attention, UpCat(·,·) represents upsampling the second input to the same size as the first input and concatenating it along the channel dimension; DASPP(·) represents Dense Atrous Spatial Pyramid Pooling, Conv() represents convolution of the data, and SE(·) represents Squeeze-and-Excitation. for By applying linear transformations and attention calculations through a multi-head self-attention module, the values ​​are then concatenated, convolved, compressed, and recalibrated. This is the output of the third branch of the feature pyramid transfer module;

[0096] The output of the first layer of the ResNet-18 backbone network In the fourth branch of the input feature pyramid transfer module, the output of the fourth branch is obtained through a multi-head self-attention module that applies linear transformation and attention calculation, followed by concatenation, compression, and recalibration.

[0097] Preferably, the process can be expressed as a formula:

[0098]

[0099]

[0100] In this context, MHSA(·) represents Multi-headed Self-attention, UpCat(·,·) represents upsampling the second input to the same size as the first input and concatenating it along the channel dimension; DASPP(·) represents Dense Atrous Spatial Pyramid Pooling, Conv() represents convolution of the data, and SE(·) represents Squeeze-and-Excitation. for By applying linear transformations and attention calculations through a multi-head self-attention module, the values ​​are then concatenated, convolved, compressed, and recalibrated. This is the output of the fourth branch of the feature pyramid transfer module;

[0101] It should be noted that the multi-head self-attention mechanism allows the model to simultaneously focus on different parts of the input and extract diverse features by dividing the input into multiple heads, independently applying linear transformations and attention calculations, and then merging these results. It captures contextual information at different scales by running multiple convolutional layers in parallel at different dilation rates and integrates these features to improve the accuracy of semantic segmentation. First, the feature channels are compressed through global average pooling, then the relationships between channels are learned and activated through fully connected layers, and finally, the original feature channels are recalibrated through weighted averaging to enhance the model's attention to important features.

[0102] S2: By fusing the outputs from the processing layer and the feature pyramid transfer module through the cross-modal fusion module, a coarse prediction region map and a coarse prediction boundary map of the road image are obtained;

[0103] Furthermore, the output of the Swin-B_224 backbone network... and Input to the first preprocessing layer, and The output is obtained by concatenating convolutions, batch normalization, and ReLU activation.

[0104] The preferred option, expressed by the formula, is:

[0105]

[0106] Here, ReLU(·) indicates that the features are activated using the ReLU function, BN(·) indicates that the input is batch normalized, and CatConv(·,·) indicates that the two inputs are concatenated along the channel dimension and then convolved. for and The first preprocessed layer's computed values ​​are obtained by concatenating convolutions, batch normalization, and ReLU activation. This is the output of the fourth layer of the Swin-B_224 backbone network. This is the output of the third layer of the Swin-B_224 backbone network;

[0107] Output the second layer of the same-sized Swin-B_224 backbone network. With the output of the fourth layer of the ResNet-18 backbone network Input to the second preprocessing layer, the With the The output is obtained by concatenating convolutions, batch normalization, and ReLU activation.

[0108] The preferred option, expressed by the formula, is:

[0109]

[0110] Here, ReLU(·) indicates that the features are activated using the ReLU function, BN(·) indicates that the input is batch normalized, and CatConv(·,·) indicates that the two inputs are concatenated along the channel dimension and then convolved. for and The computational values ​​of the second preprocessed layer obtained by concatenating convolutions, batch normalization, and ReLU activation are used. This is the output of the fourth layer of the ResNet-18 backbone network. This is the output of the second layer of the Swin-B_224 backbone network;

[0111] The output and output Input to the third preprocessing layer, the With the The output f is obtained by concatenating convolutions, batch normalization, and ReLU activation. CF ,

[0112] The preferred option, expressed by the formula, is:

[0113]

[0114] Where ReLU(·) represents activating features using the ReLU function, BN(·) represents batch normalization of the input, and CatConv(·,·) represents concatenating the two inputs along the channel dimension before performing convolution. CF for and The computational values ​​of the third preprocessing layer are obtained by concatenating convolutions, batch normalization, and ReLU activation. This is the value calculated for the second preprocessing layer. This is the value calculated for the first preprocessing layer;

[0115] Output the second branch of the feature pyramid transfer module and the first layer output of the Swin-B_224 backbone network Input cross-modal fusion module, and After each element is convolved, the corresponding elements are summed, and then the result is fed into a hierarchical split convolutional layer to obtain the output.

[0116] The preferred option, expressed by the formula, is:

[0117]

[0118] Wherein, HSC(·) represents a Hierarchical Split Convolution layer. This is the output of the first layer of the Swin-B_224 backbone network. This is the output of the second branch of the feature pyramid transfer module. The output of the hierarchical split convolutional layer is...

[0119] The and the f CF Input region prediction branch concatenation convolution obtains a coarse prediction region map The Input boundary prediction branch convolution obtains a coarse prediction boundary map

[0120] The preferred option, expressed by the formula, is:

[0121]

[0122]

[0123] in, For a rough prediction of the region map, To roughly predict the boundary map, For the output of a hierarchical split convolutional layer, f CF The values ​​are calculated for the third preprocessing layer. Conv() indicates that the data is convolved.

[0124] It should be noted that by concatenating the outputs of the Swin-B_224 backbone network and the ResNet-18 backbone network, and through convolution, batch normalization, and ReLU activation, the advantages of the two different backbone networks are fully integrated, thereby improving the accuracy of object detection. Through multi-layer preprocessing and concatenation, the output features are made richer and more capable of capturing complex features in the image, thus improving the accuracy of object detection. By employing hierarchical split convolutional layers, the input feature map is divided into multiple subsets, convolution is applied independently to each subset, and then these results are merged at different levels, thereby enhancing the network's representational power while reducing computational complexity.

[0125] S3: Using the output of the cross-modal fusion module and the output of the feature pyramid transfer module as input features, the boundary refinement perception module is used to establish a road image segmentation model;

[0126] Furthermore, a rough prediction of the region map will be made. Roughly predict the boundary map and the third branch output of the feature pyramid transfer module The first module of the input boundary refinement perception is used to obtain the first region map and the first boundary map.

[0127] The process expression is:

[0128]

[0129]

[0130]

[0131]

[0132]

[0133]

[0134] Here, Up(·,·) represents upsampling the first input to the same size as the second input, Conv() represents convolving the data, CatConv(·,·) represents concatenating the two inputs along the channel dimension and then convolving them, and RA(·,·) represents performing inverse attention operations on the input. For a rough prediction of the region map, This is the output of the third branch of the feature pyramid transfer module. for and The values ​​are obtained through sampling, convolution, anti-attention operations, and concatenation of convolution processes. for and By concatenating the values ​​from the convolution process for and By concatenating the values ​​from the convolution process for and splicing convolution values This is the first region map. This is the first boundary diagram;

[0135] The first region map of the first module is refined to perceive the boundary. First boundary diagram and the fourth branch output of the feature pyramid transfer module The second module of the input boundary refinement perception is used to obtain the second region map and the second boundary map.

[0136] The process expression is:

[0137]

[0138]

[0139]

[0140]

[0141]

[0142]

[0143] Here, Up(·,·) represents upsampling the first input to the same size as the second input, Conv() represents convolving the data, CatConv(·,·) represents concatenating the two inputs along the channel dimension and then convolving them, and RA(·,·) represents performing inverse attention operations on the input. This is the first region map. This is the output of the fourth branch of the feature pyramid transfer module. for and The values ​​are obtained through sampling, convolution, anti-attention operations, and concatenation of convolution processes. for and By concatenating the values ​​from the convolution process for and By concatenating the values ​​from the convolution process for and splicing convolution values This is the second region map. This is the second boundary diagram;

[0144] It should be noted that the boundary refinement perception module can better capture local details and contextual information in the image, thereby improving the accuracy of object detection. Because the boundary refinement perception module can better locate the boundary of the target, it can reduce the false detection rate in object detection. By performing operations such as concatenation and convolution of the outputs of different modules, computation is made more efficient while reducing memory usage. Since this technical solution adopts a hierarchical structure, the structure and depth of the model can be easily expanded to adapt to the needs of different tasks and datasets. It employs various input features, including the output of the cross-modal fusion module and the output of the feature pyramid transfer module, which have different characteristics and advantages, improving the model's generalization ability.

[0145] S4: Sample the images obtained from the cross-modal fusion module and the boundary refinement perception module to the original image size, construct loss functions to calculate the errors respectively, and use the obtained errors to update the road image segmentation model in reverse;

[0146] Furthermore, the expression for sampling to the original image size is:

[0147]

[0148]

[0149] in, This represents the predicted region map of the i-th layer after upsampling. Let f0 represent the predicted boundary map of the i-th layer after upsampling, CF represent the prediction result from the cross-modal fusion module, BA1 represent the prediction result from the first boundary refinement perception module, BA2 represent the prediction result from the second boundary refinement perception module, and f0 represent the RGB image.

[0150] The loss function expression is:

[0151] L bc =l bce +l iou

[0152] Among them, l bce Let l represent the binary cross-entropy function. iou L represents the cross-joint function. bce+iou The error between the predicted region map and the ground truth map, and the error between the predicted boundary map and the boundary map of the ground truth map;

[0153] The errors of the coarse predicted region map and coarse predicted boundary map, the first region map and the first boundary map, the second region map and the second boundary map obtained by the loss function calculation and the actual labeled map are weighted and summed, and the resulting errors are used to update the road image segmentation model in reverse.

[0154] It should be noted that by sampling the images obtained from the cross-modal fusion module and the boundary refinement perception module to the original image size and constructing a loss function to calculate the error, the model can better capture the detailed information in the image, thereby improving the accuracy of road image segmentation.

[0155] S5: Input the feasible road images into the updated model, obtain and output the corresponding road image segmentation prediction map;

[0156] Furthermore, the coarse prediction region map, the first region map, and the second region map are activated by a Sigmoid layer and then output to obtain the feasible road image segmentation prediction map.

[0157] Preferably, the Sigmoid layer is a common activation function in neural networks. The Sigmoid layer is typically used to convert the input signal into a probability value between 0 and 1.

[0158] It should be noted that by using multi-layered region prediction and boundary refinement perception modules, and by performing operations such as concatenating and convolving the outputs of different modules, more accurate region prediction results can be obtained, which can better locate the boundaries and regions of the target, thereby reducing the false detection rate in target detection. Furthermore, after activating the Sigmoid layer, the amplitude and range of the prediction results can be further adjusted to better match the segmentation of actual road images, thereby improving the accuracy of road image segmentation. This further reduces the false detection rate, avoids unnecessary computation and memory consumption, and improves computational efficiency.

[0159] The above is an illustrative scheme of a boundary-aware multimodal feasible road segmentation method according to this embodiment. It should be noted that the technical solution of this boundary-aware multimodal feasible road segmentation system belongs to the same concept as the technical solution of the boundary-aware multimodal feasible road segmentation method described above. Details not described in detail in the technical solution of the boundary-aware multimodal feasible road segmentation system in this embodiment can be found in the description of the technical solution of the boundary-aware multimodal feasible road segmentation method described above.

[0160] This embodiment of the boundary-aware multimodal feasible road segmentation system includes:

[0161] The feature extraction module is used to extract output features from feasible road images. The output features are input into different branches of the feature pyramid transfer module for processing and transfer at different levels.

[0162] The cross-modal fusion module is used to fuse the outputs from the processing layer and the feature pyramid transfer module to obtain a coarse prediction region map and a coarse prediction boundary map of the road image.

[0163] The boundary refinement perception module is used to input the output of the cross-modal fusion module and the output of the feature pyramid transfer module as input features to the boundary refinement perception module to establish a road image segmentation model.

[0164] The prediction module is used to sample the coarse prediction region map and coarse prediction boundary map, the first region map and the first boundary map, the second region map and the second boundary map to the original image size, construct a loss function to calculate the error respectively, use the obtained error to update the road image segmentation model in reverse, input the feasible road image into the updated model, and obtain and output the corresponding road image segmentation prediction map.

[0165] This embodiment also provides a computing device suitable for boundary-aware multimodal feasible road segmentation, including:

[0166] The system includes a memory and a processor. The memory stores computer-executable instructions, and the processor executes these instructions to implement the boundary-aware multimodal feasible road segmentation method proposed in the above embodiments.

[0167] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the boundary-aware multimodal feasible road segmentation method proposed in the above embodiments.

[0168] The storage medium proposed in this embodiment and the method for implementing boundary-aware multimodal feasible road segmentation proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0169] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0170] Example 2

[0171] Referring to Table 1, an embodiment of the present invention provides a multimodal feasible road segmentation method based on boundary awareness. To verify its beneficial effects, a comparison of several schemes is provided.

[0172] To verify the effectiveness of this invention, we used F β ,S α E ε The MAE and BDE indices are calculated based on the accuracy of feasible road image segmentation methods obtained through boundary perception.

[0173] ①F β It is a comprehensive measurement indicator, defined as the weighted harmonic mean of precision and recall, expressed as,

[0174]

[0175] β is usually taken as 0.3;

[0176] ②S α Widely used to measure the structural similarity between an original image and a test image, it is defined as a weighted sum of object-oriented structural similarity and region-oriented structural similarity, expressed as:

[0177] S α =α*S o +(1-α)*S r

[0178] ③E ε It is a binary foreground map measurement method that uses enhanced alignment to improve the accuracy and robustness of the evaluation, and can be represented as follows:

[0179]

[0180] ④ The MAE index is calculated as the average of the absolute errors between the predicted and actual values ​​in the forecast chart, expressed as:

[0181]

[0182] ⑤ BDE is used to measure the boundary accuracy of segmentation results, and can be expressed as,

[0183]

[0184] Table 1 lists the comparative data of image segmentation methods proposed in recent years. Upward-pointing arrows indicate that a higher performance index is better, such as F... β S α E ε The downward arrow indicates that the smaller the performance indicator, the better, such as MAE and BDE.

[0185] Table 1 Comparison and verification results

[0186]

[0187] The data in Table 1 shows that the network model of this invention exhibits the best performance in various tests, which further confirms that our technology has a stronger ability to capture multi-scale targets when dealing with complex street scenes. By utilizing the boundary refinement perception module, our scheme can optimize the region map and boundary map layer by layer, effectively reducing the inaccuracy of image boundaries caused by differences in scale and viewpoint. This optimization significantly improves the boundary accuracy of image segmentation and reduces the problem of incorrect segmentation of regions by the model.

[0188] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A boundary-aware multimodal feasible road segmentation method, characterized in that, include: Output features are extracted from feasible road images. These output features are then input into different branches of the feature pyramid transfer module for processing and transfer at different levels, including: Remove the last fully connected layer in the Swin-B_224 backbone network and convert the RGB image... The output features of each layer are obtained through the Swin-B_22 backbone network. , , , , Remove the last fully connected layer in the ResNet-18 backbone network and convert the RGB image... The output features of each layer are obtained through the ResNet-18 backbone network. , , , , The output of the fourth layer of the ResNet-18 backbone network Third layer output Second layer output First layer output The inputs are respectively fed into the first, second, third, and fourth branches of the feature pyramid transfer module. Linear transformations and attention calculations are applied through the multi-head self-attention module, followed by concatenation, convolution, compression, and recalibration processes to obtain the output of the first branch. Second branch output Third branch output Fourth branch output ; By fusing the outputs from the processing layer and the feature pyramid transfer module through a cross-modal fusion module, a coarse predicted region map and a coarse predicted boundary map of the road image are obtained, including: The output of the Swin-B_224 backbone network and Input to the first preprocessing layer, and The output is obtained by concatenating convolutions, batch normalization, and ReLU activation. , Output the second layer of the same-sized Swin-B_224 backbone network. With the output of the fourth layer of the ResNet-18 backbone network Input to the second preprocessing layer, the With the The output is obtained by concatenating convolutions, batch normalization, and ReLU activation. , Output and output Input to the third preprocessing layer, the With the The output is obtained by concatenating convolutions, batch normalization, and ReLU activation. , Output the second branch of the feature pyramid transfer module. and the first layer output of the Swin-B_224 backbone network Input cross-modal fusion module, the and stated After each element is convolved, the corresponding elements are summed, and then the result is fed into a hierarchical split convolutional layer to obtain the output. , The and stated Input region prediction branch concatenation convolution obtains a coarse prediction region map , will the Input boundary prediction branch convolution obtains a coarse prediction boundary map ; The outputs of the cross-modal fusion module and the feature pyramid transfer module are used as input features to the boundary refinement perception module to establish a road image segmentation model; The images obtained from the cross-modal fusion module and the boundary refinement perception module are sampled to the original image size, loss functions are constructed to calculate the errors respectively, and the obtained errors are used to update the road image segmentation model in reverse. Input feasible road images into the updated model, obtain and output the corresponding road image segmentation prediction map.

2. The boundary-aware multimodal feasible road segmentation method as described in claim 1, characterized in that, The outputs of the cross-modal fusion module and the feature pyramid transfer module are used as input features to the boundary refinement perception module to establish a road image segmentation model, including: Roughly predicted area map Roughly predicted boundary map and the third branch output of the feature pyramid transfer module The first module of the input boundary refinement perception is used to obtain the first region map and the first boundary map. The process expression is: in, This means upsampling the first input to the same size as the second input. This indicates that the data will be convolved. This means concatenating the two inputs along the channel dimension and then performing a convolution. This indicates that the input will undergo reverse attention processing. For a rough prediction of the region map, This is the output of the third branch of the feature pyramid transfer module. for and The values ​​are obtained through sampling, convolution, anti-attention operations, and concatenation of convolution processes. for and By concatenating the values ​​from the convolution process for and By concatenating the values ​​from the convolution process for and splicing convolution values This is the first region map. This is the first boundary diagram; The first region map of the first module is refined to perceive the boundary. First boundary diagram and the fourth branch output of the feature pyramid transfer module The second module of the input boundary refinement perception is used to obtain the second region map and the second boundary map. The process expression is: in, This means upsampling the first input to the same size as the second input. This indicates that the data will be convolved. This means concatenating the two inputs along the channel dimension and then performing a convolution. This indicates that the input will undergo reverse attention processing. This is the first region map. This is the output of the fourth branch of the feature pyramid transfer module. for and The values ​​are obtained through sampling, convolution, anti-attention operations, and concatenation of the convolution process. for and By concatenating the values ​​from the convolution process for and By concatenating the values ​​from the convolution process for and splicing convolution values This is the second region map. This is the second boundary diagram.

3. The boundary-aware multimodal feasible road segmentation method as described in claim 2, characterized in that, The images obtained from the cross-modal fusion module and the boundary refinement perception module are sampled to the original image size, and loss functions are constructed to calculate the errors respectively. The obtained errors are then used to back-update the road image segmentation model, including: The expression for sampling to the original image size is: in, Indicates the first i The predicted region map after upsampling of the layer. Indicates the first i The layer is an upsampled prediction boundary map. CF represents the prediction result from the cross-modal fusion module, BA1 represents the prediction result from boundary refinement sensing module one, and BA2 represents the prediction result from boundary refinement sensing module two. It is an RGB image; The loss function expression is: in, Represents the binary cross-entropy function. Represents a cross-joint function. The error between the predicted region map and the ground truth map, and the error between the predicted boundary map and the boundary map of the ground truth map; The errors of the coarse predicted region map and coarse predicted boundary map, the first region map and the first boundary map, the second region map and the second boundary map obtained by the loss function and the actual labeled map are weighted and summed, and the resulting errors are used to update the road image segmentation model in reverse.

4. The boundary-aware multimodal feasible road segmentation method as described in any one of claims 1-3, characterized in that, The feasible road images are input into the updated model, and the corresponding road image segmentation prediction maps are obtained and output, including: Rough prediction area map, first area map and second area map through After layer activation, the output obtains the feasible road image segmentation prediction map.

5. A system for boundary-aware multimodal feasible road segmentation, characterized in that, The method of claim 1 is used, comprising: The feature extraction module is used to extract output features from feasible road images. The output features are input into different branches of the feature pyramid transfer module for processing and transfer at different levels. The cross-modal fusion module is used to fuse the outputs from the processing layer and the feature pyramid transfer module to obtain a coarse prediction region map and a coarse prediction boundary map of the road image. The boundary refinement perception module is used to input the output of the cross-modal fusion module and the output of the feature pyramid transfer module as input features to the boundary refinement perception module to establish a road image segmentation model. The prediction module is used to sample the coarse prediction region map and coarse prediction boundary map, the first region map and the first boundary map, the second region map and the second boundary map to the original image size, construct a loss function to calculate the error respectively, use the obtained error to update the road image segmentation model in reverse, input the feasible road image into the updated model, and obtain and output the corresponding road image segmentation prediction map.

6. An electronic device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the boundary-aware multimodal feasible road segmentation method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the boundary-aware multimodal feasible road segmentation method according to any one of claims 1 to 4.

Images