Road scene segmentation method, device, electronic device and storage medium

Abstract

The present application provides a road scene segmentation method, device, electronic device and storage medium, including: obtaining a road scene image to be segmented; inputting the road scene image to be segmented into a feature extraction layer in a pre-optimized road scene cutting model, so that the feature extraction layer performs convolution calculation based on the size and feature content of the road scene image to be segmented, and determines a feature map of the road scene to be segmented; wherein the road scene cutting model is obtained by replacing the standard convolution in the initial neural network model with a decoupled dynamic convolution filter; inputting the feature map of the road scene into a category annotation layer in the road scene cutting model, so that the category annotation layer performs image category annotation on the feature map of the road scene to be segmented, and outputs a segmentation result map of the road scene image; achieving adaptability to the feature content of the road scene image, and improving the accuracy of segmenting the road scene image.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a road scene segmentation method, device, electronic device and storage medium. Background Art

[0002] Road scene segmentation is an efficient and accurate segmentation technology. Due to the complexity of road scenes, different objects and different weather have a great impact on the understanding and perception of road scenes. In these scene understandings, target detection works better for targets with fixed shapes, and can frame the target with a box. However, for some objects without shapes, such as the sky and the road, target detection is ineffective. Road scene segmentation can identify multiple objects and correctly and timely obtain the semantic information of the road scene, which provides a guarantee for the accuracy of subsequent steps and is an important part of autonomous driving technology.

[0003] At present, due to the complexity of road scenes, which include urban streets, highways, and rural roads, we also need to consider the impact of different weather conditions, such as rain, snow, sunny days, or foggy weather. There are many objects on the road, which also has a great impact on the understanding of road scenes. Due to these reasons, the segmentation effect of road scene semantic segmentation is greatly challenged. Therefore, how to quickly and accurately segment road scenes has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a road scene segmentation method, device, electronic device and storage medium, which can adaptively perform convolution calculations according to the size and feature content of the road scene image to be segmented by utilizing the feature extraction layer in the road scene cutting model, and determine different feature maps corresponding to different road scene images to be segmented, thereby achieving adaptability to the feature content of the road scene image and improving the accuracy of road scene image segmentation.

[0005] The present application provides a road scene segmentation method, which includes:

[0006] Acquire a road scene image to be segmented;

[0007] Inputting the road scene image to be segmented into a feature extraction layer in a pre-optimized road scene segmentation model, so that the feature extraction layer performs convolution calculation based on the size and feature content of the road scene image to be segmented, and determines a feature map of the road scene to be segmented; wherein the road scene segmentation model is obtained by replacing the standard convolution in the initial neural network model with a decoupled dynamic convolution filter;

[0008] The feature map of the road scene is input into the category annotation layer in the road scene cutting model, so that the category annotation layer performs image category annotation on the feature map of the road scene to be segmented, and outputs a segmentation result map of the road scene image; wherein different colors in the segmentation result map of the road scene image represent different image categories.

[0009] In a possible implementation, the step of inputting the road scene image to be segmented into a feature extraction layer in a pre-optimized road scene segmentation model so that the feature extraction layer performs convolution calculation based on the size and feature content of the road scene image to be segmented to determine a feature map of the road scene to be segmented includes:

[0010] Inputting the road scene image to be segmented into a first low-resolution sublayer in the feature extraction layer, and performing convolution calculation on the road scene image to be segmented by a decoupled dynamic convolution filter in the first low-resolution sublayer to obtain a first feature map;

[0011] The first feature map is input into the next resolution sub-layer for feature extraction, and after the feature extraction of multiple resolution sub-layers in the road scene segmentation model is completed, the obtained target feature map is subjected to feature fusion to generate a feature map of the road scene to be segmented.

[0012] In a possible implementation, the road scene image to be segmented is input into the first low-resolution sub-layer in the feature extraction layer, and the decoupled dynamic convolution filter in the first low-resolution sub-layer performs convolution calculation on the road scene image to be segmented to obtain a first feature map, including:

[0013] The decoupled dynamic convolution filter calculates feature weights based on the feature content of the road scene image to be segmented to determine target filtering parameters;

[0014] The road scene image to be segmented is subjected to convolution processing based on the target filtering parameters to determine a first feature map.

[0015] In a possible implementation, the decoupled dynamic convolution filter is composed of a spatial filter and a channel filter, and the decoupled dynamic convolution filter performs feature weight calculation based on the feature content of the road scene image to be segmented to determine the target filtering parameters, including:

[0016] Determine a spatial filter weight based on the spatial filter on the spatial branch, and determine a channel filter weight based on the channel filter on the channel branch;

[0017] Based on the spatial filter weight and the channel filter weight, a target filter parameter is determined.

[0018] In a possible implementation, the first feature map is input into the next resolution sub-layer for feature extraction until multiple resolution sub-layers in the road scene segmentation model complete feature extraction, and then the obtained target feature map is subjected to feature fusion to generate a feature map of the road scene to be segmented, including:

[0019] Inputting the first feature map into a second low-resolution sub-layer for feature extraction to obtain a second feature map, and inputting the second feature map into a third low-resolution sub-layer for feature extraction to obtain a first low-resolution feature map;

[0020] Inputting the second feature map into the first high-resolution sub-layer for feature extraction to obtain a first high-resolution feature map, performing feature fusion on the first high-resolution feature map and the first low-resolution feature map and inputting them into the fourth low-resolution sub-layer for feature extraction to obtain a second low-resolution feature map;

[0021] Performing feature fusion on the first high-resolution feature map and the first low-resolution feature map and inputting them into a second high-resolution sublayer for feature extraction to obtain a second high-resolution feature map;

[0022] The second high-resolution feature map and the second low-resolution feature map are feature fused and input into the next high-resolution sub-layer and the next low-resolution sub-layer for feature extraction. After multiple high-resolution sub-layers and low-resolution sub-layers have completed feature extraction, the obtained target feature map is feature fused to generate a feature map of the road scene to be segmented.

[0023] The embodiment of the present application further provides a road scene segmentation device, the road scene segmentation device comprising:

[0024] An acquisition module, used for acquiring a road scene image to be segmented;

[0025] A determination module, used for inputting the road scene image to be segmented into a feature extraction layer in a pre-optimized road scene segmentation model, so that the feature extraction layer performs convolution calculation based on the size and feature content of the road scene image to be segmented, and determines a feature map of the road scene to be segmented; wherein the road scene segmentation model is obtained by replacing the standard convolution in the initial neural network model with a decoupled dynamic convolution filter;

[0026] An image annotation module is used to input the feature map of the road scene into the category annotation layer in the road scene segmentation model, so that the category annotation layer performs image category annotation on the feature map of the road scene to be segmented, and outputs a segmentation result map of the road scene image; wherein different colors in the segmentation result map of the road scene image represent different image categories.

[0027] In a possible implementation manner, when the determination module is used to input the road scene image to be segmented into a feature extraction layer in a pre-optimized road scene cutting model so that the feature extraction layer performs convolution calculation based on the size and feature content of the road scene image to be segmented to determine a feature map of the road scene to be segmented, the determination module is specifically used to:

[0028] Inputting the road scene image to be segmented into a first low-resolution sublayer in the feature extraction layer, and performing convolution calculation on the road scene image to be segmented by a decoupled dynamic convolution filter in the first low-resolution sublayer to obtain a first feature map;

[0029] The first feature map is input into the next resolution sub-layer for feature extraction, and after the feature extraction of multiple resolution sub-layers in the road scene segmentation model is completed, the obtained target feature map is subjected to feature fusion to generate a feature map of the road scene to be segmented.

[0030] In a possible implementation manner, when the determination module is used to input the road scene image to be segmented into the first low-resolution sub-layer in the feature extraction layer, and the decoupled dynamic convolution filter in the first low-resolution sub-layer performs convolution calculation on the road scene image to be segmented to obtain a first feature map, the determination module is specifically used to:

[0031] The decoupled dynamic convolution filter calculates feature weights based on the feature content of the road scene image to be segmented to determine target filtering parameters;

[0032] The road scene image to be segmented is subjected to convolution processing based on the target filtering parameters to determine a first feature map.

[0033] An embodiment of the present application also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the road scene segmentation method as described above are performed.

[0034] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the road scene segmentation method as described above are executed.

[0035] The embodiments of the present application provide a road scene segmentation method, device, electronic device and storage medium, which obtain a road scene image to be segmented; input the road scene image to be segmented into a feature extraction layer in a pre-optimized road scene cutting model, so that the feature extraction layer performs convolution calculation based on the size and feature content of the road scene image to be segmented, and determines a feature map of the road scene to be segmented; wherein the road scene cutting model is obtained by replacing the standard convolution in the initial neural network model with a decoupled dynamic convolution filter; input the feature map of the road scene into a category annotation layer in the road scene cutting model, so that the category annotation layer performs image category annotation on the feature map of the road scene to be segmented, and outputs a segmentation result map of the road scene image; wherein different colors in the segmentation result map of the road scene image represent different image categories.

[0036] In this way, by utilizing the feature extraction layer in the road scene cutting model, convolution calculations can be adaptively performed according to the size and feature content of the road scene image to be segmented, and different feature maps corresponding to different road scene images to be segmented can be determined, thereby achieving adaptability to the feature content of the road scene image and improving the accuracy of road scene image segmentation.

[0037] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flowchart of a road scene segmentation method provided in an embodiment of the present application;

[0040] Figure 2 A detailed flow chart of a road scene segmentation method provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of road scene segmentation provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of the structure of a road scene segmentation device provided in an embodiment of the present application;

[0043] Figure 5A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art, under the guidance of the content of the present application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0045] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0046] In order to enable those skilled in the art to use the contents of this application, the following implementation is given in combination with the specific application scenario of "segmenting road scenes". For those skilled in the art, the general principles defined here can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application.

[0047] The following methods, devices, electronic devices or computer-readable storage media of the embodiments of the present application can be applied to any scenario where road scenes need to be segmented. The embodiments of the present application are not limited to specific application scenarios. Any solution using a road scene segmentation method, device, electronic device and storage medium provided by the embodiments of the present application is within the protection scope of the present application.

[0048] Research has found that at this stage, due to the complexity of road scenes, road scenes include urban streets, highways, and rural roads, etc., in addition to the impact of different weather conditions, such as rain, snow, sunny days, or foggy weather, there are many objects on the road, which also has a great impact on the understanding of road scenes. Due to these reasons, the segmentation effect of road scene semantic segmentation is greatly challenged. Therefore, how to quickly and accurately segment road scenes has become an urgent problem to be solved.

[0049] Model design based on lightweight road scene segmentation network is a popular research method. Many researchers want to use as little computation and parameters as possible while obtaining higher model accuracy. In order to reduce the amount of computation generated by semantic segmentation, there are generally two ways: reducing the image size and reducing the model complexity. Reducing the image size can most directly reduce the amount of computation, but the image will lose a lot of details, thus affecting the accuracy. Reducing the complexity of the model will weaken the feature extraction ability of the model, thus affecting the segmentation accuracy. Therefore, how to apply lightweight models in road scene segmentation tasks while taking into account both real-time performance and accuracy is quite challenging.

[0050] First, the application scenarios to which the present application is applicable are introduced. The present application can be applied in the field of data processing technology.

[0051] See also Figure 1 , Figure 1 Flow chart of a road scene segmentation method provided in an embodiment of the present application. Figure 1 As shown, the road scene segmentation method provided in the embodiment of the present application includes:

[0052] S101: Acquire a road scene image to be segmented.

[0053] In this step, a road scene image to be segmented is obtained. Here, the road scene image can be used in autonomous driving. The road scene image can protect other feature contents such as trees, pedestrians, sky, road conditions, weather conditions, etc.

[0054] Here, the road scene image of the vehicle during road driving can be obtained through a camera device, or in the field of unmanned driving, the road scene image of the driving process can be obtained using point cloud data. This part does not limit the method of obtaining the road scene image.

[0055] S102: Inputting the road scene image to be segmented into a feature extraction layer in a pre-optimized road scene cutting model, so that the feature extraction layer performs convolution calculation based on the size and feature content of the road scene image to be segmented, and determines a feature map of the road scene to be segmented; wherein the road scene cutting model is obtained by replacing the standard convolution in the initial neural network model with a decoupled dynamic convolution filter.

[0056] In this step, the road scene image to be segmented is input into the feature extraction layer of the pre-trained road scene cutting model, and the decoupled dynamic convolution filter in the feature extraction layer performs convolution calculation using the size and feature content of the road scene image to be segmented, and determines the target filter parameters obtained by the convolution calculation based on the size and feature content of the road scene image to be segmented. The target filter parameters are used to extract the image features in the road scene image to be segmented, and the feature map of the road scene to be segmented is obtained.

[0057] Here, different road scene images are input into the feature extraction layer, and the decoupled dynamic convolution filter in the feature extraction layer will calculate different target filter parameters according to the size and feature content of different road scene images, so that the decoupled dynamic convolution filter can be adaptively adjusted according to different road scene images, thereby avoiding the problem of inaccurate feature extraction in the process of extracting image features in road scene images using a unique target filter parameter. Here, the initial neural network model is trained using multiple sample images obtained; the standard convolution in the initial neural network model is replaced with a decoupled dynamic convolution filter, and an optimized road scene cutting model is obtained based on the replaced decoupled dynamic convolution filter. Among them, the standard convolution in the initial neural network model is 3*3Conv.

[0058] Here, the initial neural network model is the DDRNet network model, which starts with a trunk and then splits into two parallel deep branches with different resolutions. One deep branch generates a relatively high-resolution feature map, and the other extracts rich contextual information through multiple downsampling operations. In order to achieve efficient information fusion, multiple bilateral connections are bridged between the two branches.

[0059] Here, the size of the road scene image to be segmented is the pixel size of the road scene, such as 130*300, etc., and the feature content of the road scene image to be segmented is the feature information of the road scene, such as trees, pedestrians, etc. in the road scene.

[0060] For further information, see Figure 2 , Figure 2 A detailed flow chart of a road scene segmentation method provided in an embodiment of the present application. Figure 2 As shown, the road scene image to be segmented is input into a feature extraction layer in a pre-optimized road scene cutting model, so that the feature extraction layer performs convolution calculation based on the size and feature content of the road scene image to be segmented, and determines a feature map of the road scene to be segmented, including:

[0061] S201: Inputting the road scene image to be segmented into the first low-resolution sub-layer in the feature extraction layer, and the decoupled dynamic convolution filter in the first low-resolution sub-layer performs convolution calculation on the road scene image to be segmented to obtain a first feature map.

[0062] In this step, the road scene image to be segmented is input into the first low-resolution sublayer in the feature extraction layer in the road scene cutting model, and the decoupled dynamic convolution filter in the first low-resolution sublayer implements convolution calculation on the road scene image to be segmented. The convolution calculation result is used to extract features of the road scene image to be segmented, and the first feature map is determined.

[0063] Here, the feature extraction layer includes multiple low-resolution sub-layers and multiple high-resolution sub-layers, and each low-resolution sub-layer and high-resolution sub-layer includes a decoupled dynamic convolution filter.

[0064] Here, the decoupled dynamic convolution filter decouples the deep dynamic filter into spatial and channel dynamic filters, making the segmentation network learning process content adaptive, improving the feature expression ability, and thus improving the network's segmentation accuracy for road scenes.

[0065] Here, the first feature map is obtained by performing convolution calculation on the feature content of the road scene image to be segmented by the decoupled dynamic convolution filter in the first low-resolution sublayer of the feature extraction layer in the road scene cutting model, determining the target filtering parameters, and subsequently extracting the feature content of the road scene image to be segmented according to the target filtering parameters.

[0066] Furthermore, the road scene image to be segmented is input into the first low-resolution sub-layer in the feature extraction layer, and the decoupled dynamic convolution filter in the first low-resolution sub-layer performs convolution calculation on the road scene image to be segmented to obtain a first feature map, including:

[0067] A: The decoupled dynamic convolution filter calculates feature weights based on the feature content of the road scene image to be segmented to determine target filtering parameters.

[0068] Here, the decoupled dynamic convolution filter calculates feature weights according to the feature content of the road scene image to be segmented, and can quickly and accurately determine the target filtering parameters corresponding to different road scene images to be segmented according to different road scene images to be segmented, thereby realizing that the decoupled dynamic filter will make adjustments to different images (such as sidewalks, pedestrians, and traffic lights), and perform convolution processing with more suitable target filtering parameters, avoiding the problem of using the same standard convolution to process all road scene images to be segmented during the road scene segmentation process, resulting in the final road scene segmentation map being unclear.

[0069] Here, the decoupled dynamic convolution filter is composed of a spatial filter and a channel filter. In the decoupled dynamic convolution filter, a single convolution kernel is not used on each layer, but multiple parallel convolution kernels are dynamically aggregated according to attention. Attention dynamically adjusts the weight of each convolution kernel according to the input, thereby generating an adaptive decoupled dynamic convolution filter. Since attention is a function of the input, the decoupled dynamic convolution is no longer a linear function. Superimposing convolution kernels in a nonlinear way through attention has stronger representation capabilities. The decoupled dynamic convolution filter introduces spatial and channel filter branches. The channel branch has a very low computational complexity and consists of GAP+FC+ReLU+FC. Thanks to the small kernel size, the spatial branch greatly reduces the number of network parameters. Therefore, the decoupled dynamic convolution filter significantly improves the network expression capability while reducing the computational cost.

[0070] Furthermore, the decoupled dynamic convolution filter performs feature weight calculation based on the feature content of the road scene image to be segmented to determine the target filtering parameters, including:

[0071] a: A spatial filter weight is determined based on a spatial filter on a spatial branch, and a channel filter weight is determined based on a channel filter on a channel branch.

[0072] The spatial filter weights are determined by using the spatial filters on the spatial branches of the decoupled dynamic convolution filter, and the channel filter weights are determined by using the channel filters on the channel branches of the decoupled dynamic convolution filter.

[0073] Among them, the decoupled dynamic convolution filter can be expressed by the following formula:

[0074]

[0075] Among them, F (r,i) ∈R represents the current position, D SP is the spatial filter, is the spatial dynamic filter, D ch [P i -P j] is a channel filter, and F(r,j)∈R represents the input feature value at the jth pixel and the rth channel.

[0076] b: Determine target filtering parameters based on the spatial filtering weight and the channel filtering weight.

[0077] The target filter parameters are determined by fusing the spatial filter weights and the channel filter weights.

[0078] Among them, the decoupled dynamic convolution filter in the first low-resolution sub-layer generates a channel filter weight based on the input road scene image to be segmented, the channel branch generates a channel filter weight, the spatial branch generates a spatial filter weight, the two weights are fused to generate a filter, and then the generated filter is used to filter the input road scene image to be segmented.

[0079] B: Performing convolution processing on the road scene image to be segmented based on the target filtering parameters to determine a first feature map.

[0080] The target filter parameters are used to perform convolution processing and feature extraction on the road scene image to be segmented to obtain a first feature map.

[0081] The target filter parameters are determined according to the size and feature content of the input road scene image to be segmented, and the target filter parameters are not unique.

[0082] S202: Input the first feature map to the next resolution sub-layer for feature extraction, until the multiple resolution sub-layers in the road scene segmentation model complete feature extraction, and then perform feature fusion on the obtained target feature map to generate a feature map of the road scene to be segmented.

[0083] In this step, the first feature map obtained is input into the next resolution sub-layer for feature extraction. After feature extraction is completed for each resolution sub-layer in the road scene segmentation model, the obtained target feature map is subjected to feature fusion to generate a feature map of the road scene to be segmented.

[0084] Here, the feature extraction layer includes multiple resolution sub-layers, and each resolution sub-layer includes a decoupled dynamic convolution filter. In this way, each decoupled dynamic convolution filter in each resolution sub-layer can perform convolution calculations based on the input feature map, thereby improving the accuracy of extracting feature content in the feature map.

[0085] Furthermore, the first feature map is input into the next resolution sub-layer for feature extraction until multiple resolution sub-layers in the road scene segmentation model complete feature extraction, and then the obtained target feature map is subjected to feature fusion to generate a feature map of the road scene to be segmented, including:

[0086] (1): The first feature map is input into the second low-resolution sub-layer for feature extraction to obtain a second feature map, and the second feature map is input into the third low-resolution sub-layer for feature extraction to obtain a first low-resolution feature map.

[0087] Here, the first feature map determined by the first low-resolution sub-layer is input into the second low-resolution sub-layer, and the decoupled dynamic convolution filter in the second low-resolution sub-layer calculates the first feature map to determine the target filtering parameters, and uses the target filtering parameters to extract the color features, texture features, shape features, spatial relationship features and other features of the first feature map to determine the second feature map, and then the second feature map is input into the third low-resolution sub-layer, and the decoupled dynamic convolution filter in the third low-resolution sub-layer calculates the second feature map to determine the target filtering parameters, and uses the target filtering parameters to extract the color features, texture features, shape features, spatial relationship features and other features of the second feature map to determine the first low-resolution feature map.

[0088] Among them, the first low-resolution feature map is determined in the first low-resolution sub-layer.

[0089] (2) Inputting the second feature map into the first high-resolution sub-layer for feature extraction to obtain a first high-resolution feature map, performing feature fusion on the first high-resolution feature map and the first low-resolution feature map, and inputting them into the fourth low-resolution sub-layer for feature extraction to obtain a second low-resolution feature map.

[0090] Here, the second feature map determined by the second low-resolution sub-layer is input into the first high-resolution sub-layer, and the decoupled dynamic convolution filter in the first high-resolution sub-layer calculates the second feature map to determine the target filter parameters, and uses the target filter parameters to extract the color features, texture features, shape features, spatial relationship features and other features of the second feature map to determine the first high-resolution feature map. The first high-resolution feature map and the first low-resolution feature map are feature fused and input into the fourth low-resolution sub-layer, and the decoupled dynamic convolution filter in the fourth low-resolution sub-layer processes the feature map obtained by feature fusion of the first high-resolution feature map and the first low-resolution feature map to obtain the second low-resolution feature map.

[0091] (3) Fusing the first high-resolution feature map and the first low-resolution feature map and inputting the resulting features into a second high-resolution sublayer for feature extraction to obtain a second high-resolution feature map.

[0092] Here, the first high-resolution feature map and the first low-resolution feature map are feature fused and input into the decoupled dynamic convolution filter in the second high-resolution sub-layer, and the feature map obtained by feature fusion of the first high-resolution feature map and the first low-resolution feature map is processed to obtain a second high-resolution feature map.

[0093] (4) Fusing the second high-resolution feature map and the second low-resolution feature map and inputting them into the next high-resolution sub-layer and the next low-resolution sub-layer for feature extraction. After feature extraction is completed for multiple high-resolution sub-layers and low-resolution sub-layers, the obtained target feature map is subjected to feature fusion to generate a feature map of the road scene to be segmented.

[0094] Here, the second high-resolution feature map and the second low-resolution feature map are feature fused and simultaneously input into the next high-resolution sub-layer and the next low-resolution sub-layer for feature extraction of color features, texture features, shape features, spatial relationship features and other features. After multiple high-resolution sub-layers and low-resolution sub-layers have completed feature extraction, the obtained target feature map is feature fused to generate a feature map of the road scene to be segmented, thereby realizing that the road scene image to be segmented passes through the decoupled dynamic convolution filter in each resolution sub-layer. The decoupled dynamic convolution filter can dynamically adjust the weight of each convolution kernel of each decoupled dynamic convolution filter according to the input image, thereby realizing adaptive convolution processing of the input feature map.

[0095] Among them, during the upsampling and downsampling process of the road scene image to be segmented through multiple layers of low-resolution sub-layers and multiple layers of high-resolution sub-layers, each decoupled dynamic convolution filter in each resolution sub-layer will adaptively perform convolution calculation according to the input image to obtain the target filter parameters, and use the target filter parameters to extract the color features, texture features, etc. of the input graphics, so as to achieve the gradual clarity of the feature map output by each resolution sub-layer, until the last resolution sub-layer processes the feature map and outputs the feature map of the road scene to be segmented.

[0096] For further information, see Figure 3 , Figure 3 Schematic diagram of road scene segmentation provided in the embodiment of the present application. Figure 3As shown in the figure, the scene image to be segmented is input into the first low-resolution sub-layer in the feature extraction layer. The decoupled dynamic convolution filter in the first low-resolution sub-layer performs convolution calculation on the road scene image to be segmented to obtain the first feature map. The same goes for the following steps, so that the adaptive features are recombined to make the information of feature fusion increase from less to more. The new feature map formed by each layer will be used as the input of the next resolution sub-layer. Figure 3 As shown in the figure, the black solid lines represent up-sampling and down-sampling operations, and the black dotted lines represent bilateral fusion operations. For high-to-low fusion, before point-by-point summation, the high-resolution feature map is down-sampled through a 3×3 convolution sequence with a step size of 2. For low-resolution to high-resolution feature maps, 1×1 convolution is first used to compress the low-resolution feature map, and then bilinear interpolation is used to update the sample. The resolution sublayer dynamically adjusts the weight of each convolution kernel according to the input image, thereby generating an adaptive decoupled dynamic convolution, so that the road scene segmentation model can adaptively obtain feature information, while reducing the amount of network parameters, while improving the accuracy of the road scene segmentation model.

[0097] S103: Input the feature map of the road scene to the category annotation layer in the road scene segmentation model, so that the category annotation layer performs image category annotation on the feature map of the road scene to be segmented, and outputs a segmentation result map of the road scene image; wherein different colors in the segmentation result map of the road scene image represent different image categories.

[0098] In this step, the feature map of the road scene image to be segmented is input into the category annotation layer in the road scene cutting model, and the feature map of the road scene image to be segmented is annotated with image categories in the category annotation layer, so that different categories can be determined according to different colors in the segmentation result map of the road scene image.

[0099] Here, the blue color in the segmentation result image of the road scene image can represent the sky category, the white color can represent the road category, the red color can represent the pedestrian category, and the green color can represent the tree category. This part does not limit the category represented by each color.

[0100] In a specific embodiment, a road scene image to be segmented is obtained, and the road scene image to be segmented is input into a feature extraction layer in a road scene cutting model, wherein the feature extraction layer includes multiple resolution sub-layers, and each resolution sub-layer includes a decoupled dynamic convolution filter, and each decoupled dynamic convolution filter can perform convolution calculation on the size and feature content of the road scene image to be segmented, and determine a feature map of the road scene to be segmented, thereby realizing that the decoupled dynamic convolution filter can dynamically adjust the target filter parameters according to the input image information, thereby improving the accuracy in the feature extraction process of the road scene image to be segmented. Then, the feature map of the road scene to be segmented is input into a category annotation layer in the road scene cutting model, and the category annotation layer performs color category annotation according to the feature map of the road scene to be segmented, so as to obtain a segmentation result map of the road scene image.

[0101] An embodiment of the present application provides a road scene segmentation method, which includes acquiring a road scene image to be segmented; inputting the road scene image to be segmented into a feature extraction layer in a pre-optimized road scene cutting model, so that the feature extraction layer performs convolution calculation based on the size and feature content of the road scene image to be segmented, and determines a feature map of the road scene to be segmented; wherein the road scene cutting model is obtained by replacing the standard convolution in the initial neural network model with a decoupled dynamic convolution filter; inputting the feature map of the road scene into a category annotation layer in the road scene cutting model, so that the category annotation layer performs image category annotation on the feature map of the road scene to be segmented, and outputs a segmentation result map of the road scene image; wherein different colors in the segmentation result map of the road scene image represent different image categories.

[0102] In this way, by utilizing the feature extraction layer in the road scene cutting model, convolution calculations can be adaptively performed according to the size and feature content of the road scene image to be segmented, and different feature maps corresponding to different road scene images to be segmented can be determined, thereby achieving adaptability to the feature content of the road scene image and improving the accuracy of road scene image segmentation.

[0103] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of a road scene segmentation device provided in an embodiment of the present application. Figure 4 As shown, the road scene segmentation device 400 includes:

[0104] An acquisition module 410 is used to acquire a road scene image to be segmented;

[0105] The determination module 420 is used to input the road scene image to be segmented into a feature extraction layer in a pre-optimized road scene segmentation model, so that the feature extraction layer performs convolution calculation based on the size and feature content of the road scene image to be segmented, and determines a feature map of the road scene to be segmented; wherein the road scene segmentation model is obtained by replacing the standard convolution in the initial neural network model with a decoupled dynamic convolution filter;

[0106] The image annotation module 430 is used to input the feature map of the road scene into the category annotation layer in the road scene segmentation model, so that the category annotation layer performs image category annotation on the feature map of the road scene to be segmented, and outputs a segmentation result map of the road scene image; wherein different colors in the segmentation result map of the road scene image represent different image categories.

[0107] Further, when the determination module 420 is used to input the road scene image to be segmented into the feature extraction layer in the pre-optimized road scene cutting model so that the feature extraction layer performs convolution calculation based on the size and feature content of the road scene image to be segmented to determine the feature map of the road scene to be segmented, the determination module 420 is specifically used to:

[0108] Inputting the road scene image to be segmented into a first low-resolution sublayer in the feature extraction layer, and performing convolution calculation on the road scene image to be segmented by a decoupled dynamic convolution filter in the first low-resolution sublayer to obtain a first feature map;

[0109] The first feature map is input into the next resolution sub-layer for feature extraction, and after the feature extraction of multiple resolution sub-layers in the road scene segmentation model is completed, the obtained target feature map is subjected to feature fusion to generate a feature map of the road scene to be segmented.

[0110] Further, when the determination module 420 is used to input the road scene image to be segmented into the first low-resolution sub-layer in the feature extraction layer, and the decoupled dynamic convolution filter in the first low-resolution sub-layer performs convolution calculation on the road scene image to be segmented to obtain a first feature map, the determination module 420 is specifically used to:

[0111] The decoupled dynamic convolution filter calculates feature weights based on the feature content of the road scene image to be segmented to determine target filtering parameters;

[0112] The road scene image to be segmented is subjected to convolution processing based on the target filtering parameters to determine a first feature map.

[0113] Further, when the decoupled dynamic convolution filter is composed of a spatial filter and a channel filter, and the decoupled dynamic convolution filter performs feature weight calculation based on the feature content of the road scene image to be segmented and determines the target filtering parameters, the determination module 420 is specifically used to:

[0114] Determine a spatial filter weight based on the spatial filter on the spatial branch, and determine a channel filter weight based on the channel filter on the channel branch;

[0115] Based on the spatial filter weight and the channel filter weight, a target filter parameter is determined.

[0116] Further, when the determination module 420 is used to input the first feature map into the next resolution sub-layer for feature extraction, until the multiple resolution sub-layers in the road scene segmentation model complete feature extraction, the obtained target feature map is subjected to feature fusion to generate a feature map of the road scene to be segmented, the determination module 420 is specifically used to:

[0117] Inputting the first feature map into a second low-resolution sub-layer for feature extraction to obtain a second feature map, and inputting the second feature map into a third low-resolution sub-layer for feature extraction to obtain a first low-resolution feature map;

[0118] Inputting the second feature map into the first high-resolution sub-layer for feature extraction to obtain a first high-resolution feature map, performing feature fusion on the first high-resolution feature map and the first low-resolution feature map and inputting them into the fourth low-resolution sub-layer for feature extraction to obtain a second low-resolution feature map;

[0119] Performing feature fusion on the first high-resolution feature map and the first low-resolution feature map and inputting them into a second high-resolution sublayer for feature extraction to obtain a second high-resolution feature map;

[0120] The second high-resolution feature map and the second low-resolution feature map are feature fused and input into the next high-resolution sub-layer and the next low-resolution sub-layer for feature extraction. After multiple high-resolution sub-layers and low-resolution sub-layers have completed feature extraction, the obtained target feature map is feature fused to generate a feature map of the road scene to be segmented.

[0121] An embodiment of the present application provides a road scene segmentation device, which includes: an acquisition module, used to acquire a road scene image to be segmented; a determination module, used to input the road scene image to be segmented into a feature extraction layer in a pre-optimized road scene cutting model, so that the feature extraction layer performs convolution calculation based on the size and feature content of the road scene image to be segmented, and determines a feature map of the road scene to be segmented; wherein the road scene cutting model is obtained by replacing the standard convolution in the initial neural network model with a decoupled dynamic convolution filter; an image annotation module, used to input the feature map of the road scene into a category annotation layer in the road scene cutting model, so that the category annotation layer performs image category annotation on the feature map of the road scene to be segmented, and outputs a segmentation result map of the road scene image; wherein different colors in the segmentation result map of the road scene image represent different image categories.

[0122] In this way, by utilizing the feature extraction layer in the road scene cutting model, convolution calculations can be adaptively performed according to the size and feature content of the road scene image to be segmented, and different feature maps corresponding to different road scene images to be segmented can be determined, thereby achieving adaptability to the feature content of the road scene image and improving the accuracy of road scene image segmentation.

[0123] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown in , the electronic device 500 includes a processor 510 , a memory 520 and a bus 530 .

[0124] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 communicates with the memory 520 via the bus 530. When the machine-readable instructions are executed by the processor 510, the above-mentioned Figure 1 as well as Figure 2 The specific implementation of the steps of the road scene segmentation method in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0125] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 as well as Figure 2 The specific implementation of the steps of the road scene segmentation method in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0126] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0127] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0130] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0131] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A road scene segmentation method, It is characterized in that The road scene segmentation method comprises: Acquire a road scene image to be segmented; Inputting the road scene image to be segmented into a feature extraction layer in a pre-optimized road scene segmentation model, so that the feature extraction layer performs convolution calculation based on the size and feature content of the road scene image to be segmented, and determines a feature map of the road scene to be segmented; wherein the road scene segmentation model is obtained by replacing the standard convolution in the initial neural network model with a decoupled dynamic convolution filter; Inputting the feature map of the road scene into the category annotation layer in the road scene segmentation model, so that the category annotation layer performs image category annotation on the feature map of the road scene to be segmented, and outputs a segmentation result map of the road scene image; wherein different colors in the segmentation result map of the road scene image represent different image categories; The step of inputting the road scene image to be segmented into a feature extraction layer in a pre-optimized road scene cutting model, so that the feature extraction layer performs convolution calculation based on the size and feature content of the road scene image to be segmented, and determines a feature map of the road scene to be segmented, comprises: Inputting the road scene image to be segmented into a first low-resolution sublayer in the feature extraction layer, and performing convolution calculation on the road scene image to be segmented by a decoupled dynamic convolution filter in the first low-resolution sublayer to obtain a first feature map; Inputting the first feature map into the next resolution sub-layer for feature extraction, until multiple resolution sub-layers in the road scene segmentation model complete feature extraction, performing feature fusion on the obtained target feature map to generate a feature map of the road scene to be segmented; The step of inputting the road scene image to be segmented into the first low-resolution sub-layer in the feature extraction layer, and performing convolution calculation on the road scene image to be segmented by the decoupled dynamic convolution filter in the first low-resolution sub-layer to obtain a first feature map includes: The decoupled dynamic convolution filter calculates feature weights based on the feature content of the road scene image to be segmented to determine target filtering parameters; The road scene image to be segmented is subjected to convolution processing based on the target filtering parameters to determine a first feature map.

2. The road scene segmentation method according to claim 1, It is characterized in that in, The decoupled dynamic convolution filter is composed of a spatial filter and a channel filter. The decoupled dynamic convolution filter performs feature weight calculation based on the feature content of the road scene image to be segmented to determine the target filtering parameters, including: Determine a spatial filter weight based on the spatial filter on the spatial branch, and determine a channel filter weight based on the channel filter on the channel branch; Based on the spatial filter weight and the channel filter weight, a target filter parameter is determined.

3. The road scene segmentation method according to claim 1, It is characterized in that The step of inputting the first feature map to the next resolution sub-layer for feature extraction until multiple resolution sub-layers in the road scene segmentation model complete feature extraction, and then performing feature fusion on the obtained target feature map to generate a feature map of the road scene to be segmented, includes: Inputting the first feature map into a second low-resolution sub-layer for feature extraction to obtain a second feature map, and inputting the second feature map into a third low-resolution sub-layer for feature extraction to obtain a first low-resolution feature map; Inputting the second feature map into the first high-resolution sub-layer for feature extraction to obtain a first high-resolution feature map, performing feature fusion on the first high-resolution feature map and the first low-resolution feature map and inputting them into the fourth low-resolution sub-layer for feature extraction to obtain a second low-resolution feature map; Performing feature fusion on the first high-resolution feature map and the first low-resolution feature map and inputting them into a second high-resolution sublayer for feature extraction to obtain a second high-resolution feature map; The second high-resolution feature map and the second low-resolution feature map are feature fused and input into the next high-resolution sub-layer and the next low-resolution sub-layer for feature extraction. After multiple high-resolution sub-layers and low-resolution sub-layers have completed feature extraction, the obtained target feature map is feature fused to generate a feature map of the road scene to be segmented.

4. A road scene segmentation device, It is characterized in that The road scene segmentation device comprises: An acquisition module, used for acquiring a road scene image to be segmented; A determination module, used for inputting the road scene image to be segmented into a feature extraction layer in a pre-optimized road scene segmentation model, so that the feature extraction layer performs convolution calculation based on the size and feature content of the road scene image to be segmented, and determines a feature map of the road scene to be segmented; wherein the road scene segmentation model is obtained by replacing the standard convolution in the initial neural network model with a decoupled dynamic convolution filter; An image annotation module is used to input the feature map of the road scene into the category annotation layer in the road scene segmentation model, so that the category annotation layer performs image category annotation on the feature map of the road scene to be segmented, and outputs a segmentation result map of the road scene image; wherein different colors in the segmentation result map of the road scene image represent different image categories; When the determining module is used to input the road scene image to be segmented into the feature extraction layer in the pre-optimized road scene cutting model so that the feature extraction layer performs convolution calculation based on the size and feature content of the road scene image to be segmented to determine the feature map of the road scene to be segmented, the determining module is specifically used to: Inputting the road scene image to be segmented into a first low-resolution sublayer in the feature extraction layer, and performing convolution calculation on the road scene image to be segmented by a decoupled dynamic convolution filter in the first low-resolution sublayer to obtain a first feature map; Inputting the first feature map into the next resolution sub-layer for feature extraction, until multiple resolution sub-layers in the road scene segmentation model complete feature extraction, performing feature fusion on the obtained target feature map to generate a feature map of the road scene to be segmented; When the determination module is used to input the road scene image to be segmented into the first low-resolution sub-layer in the feature extraction layer, and the decoupled dynamic convolution filter in the first low-resolution sub-layer performs convolution calculation on the road scene image to be segmented to obtain a first feature map, the determination module is specifically used to: The decoupled dynamic convolution filter calculates feature weights based on the feature content of the road scene image to be segmented to determine target filtering parameters; The road scene image to be segmented is subjected to convolution processing based on the target filtering parameters to determine a first feature map.

5. An electronic device, It is characterized in that include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the road scene segmentation method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the road scene segmentation method according to any one of claims 1 to 3 are executed.

Images