Semantic segmentation method and device of image, equipment and storage medium

Abstract

The invention discloses an image semantic segmentation method and device, equipment and a storage medium. The method comprises the following steps: firstly, inputting a preprocessed streetscape image data set, and processing through a backbone network to obtain a first feature image set and an attention feature image set; then, the attention feature image set is sent into a DCP module for feature fusion and up-sampling operation to generate a second feature image set; then, performing feature fusion on the first feature image set and the second feature image set to form a third feature image set; and finally, the third feature image set is transmitted into an image classification network for image semantic segmentation, and an accurate semantic segmentation result is generated. According to the method, the attention mechanism and the DCP module are combined, the recognition capability of feature importance differences between different channels and different spatial positions in the image is improved, the precision and robustness of semantic segmentation are improved, the method is suitable for a complex and changeable streetscape environment, and the semantic segmentation efficiency is improved. And more accurate basic data support is provided for environmental perception in the automatic driving technology.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, apparatus, device and storage medium for semantic segmentation of an image. Background Art

[0002] The bank's mobile financial service vehicle is an innovative model in the field of financial services. It is equipped with a variety of financial service terminals such as self-service smart counters and mobile marketing tablets, which can provide convenient mobile financial services, solve the problem of insufficient bank branch coverage in some areas at a low cost, and significantly improve the inclusiveness and convenience of financial services. However, traditional mobile financial service vehicles face problems such as driver shortage, high operating costs, and limited service time, which restrict their further development and promotion.

[0003] Bank mobile financial service vehicles mainly serve communities, towns, factory areas, campuses and other areas. The traffic environment in these scenarios is relatively simple, and it is feasible to realize autonomous driving in such areas in terms of national policies and technical applications.

[0004] Environmental perception is the first step in autonomous driving. Its task is to identify and detect key information such as traffic targets, obstacles, and drivable areas in the vehicle's driving environment. The accuracy, robustness, and real-time performance of environmental perception directly affect the results of subsequent decisions and control modules of the autonomous driving system. Therefore, high-precision environmental perception is the basis for the implementation of autonomous driving applications. Semantic segmentation refers to the pixel-level classification of various traffic targets and environments (such as vehicles, pedestrians, roads, drivable areas, etc.), which is one of the important tasks of environmental perception.

[0005] Semantic segmentation methods are mainly divided into traditional non-deep learning methods and deep learning methods. Traditional methods usually use low-dimensional manual features such as image edges and textures for segmentation, but the representation ability of low-dimensional features is insufficient, making it difficult to achieve high-precision semantic segmentation. In addition, traditional methods rely on the selection of manual features, are easily affected by subjective factors, have low robustness, and are difficult to maintain good performance in various scenarios. In contrast, semantic segmentation methods based on deep learning have the advantages of automatic feature extraction, strong robustness, and high accuracy. They extract multi-scale features of images through deep neural networks, achieve end-to-end pixel-level classification of images, and significantly improve the accuracy and efficiency of semantic segmentation. They are currently the mainstream semantic segmentation methods. However, existing semantic segmentation methods based on deep learning usually obtain image feature representations from the perspective of global features, ignoring the differences in feature importance between different channels and between different spatial positions. Summary of the invention

[0006] Based on the above problems, the present application provides a semantic segmentation method, apparatus, device and storage medium for an image, which can enhance the ability to recognize differences in feature importance between different channels and between different spatial positions in an image, thereby improving the accuracy and robustness of semantic segmentation.

[0007] The embodiments of the present application disclose the following technical solutions:

[0008] A method for semantic segmentation of an image, the method comprising:

[0009] The preprocessed street view image dataset is input into the backbone network for processing to obtain the first feature atlas and the attention feature atlas;

[0010] Input the attention feature atlas into the dilated convolutional pyramid module (DCP) module for feature fusion and upsampling to obtain a second feature atlas;

[0011] Performing feature fusion on the first feature atlas and the second feature atlas to obtain a third feature atlas;

[0012] The third feature atlas is input into an image classification network to perform image semantic segmentation to obtain a semantic segmentation result.

[0013] In a possible implementation, the step of inputting the preprocessed street view image dataset into the backbone network for processing to obtain a first feature atlas and an attention feature atlas includes:

[0014] Inputting the preprocessed street view image data set into the first layer and the second layer structure of the backbone network to perform step-by-step feature extraction to obtain the first feature atlas;

[0015] Inputting the first feature atlas into the third, fourth and fifth layer structures of the backbone network to perform step-by-step feature extraction to obtain a fourth feature atlas;

[0016] Adding the first feature atlas and the fourth feature atlas to obtain a preliminary attention feature atlas;

[0017] The preliminary attention feature atlas is input into the sixth layer of the backbone network for processing to obtain the attention feature atlas.

[0018] In one possible implementation,

[0019] The first layer of the backbone network includes a 7×7 convolution layer, a batch normalization layer BN layer, a rectified linear unit Relu activation function layer and a maximum pooling layer in sequence;

[0020] The structure of the second layer of the backbone network includes a residual layer consisting of three bottleneck residual blocks;

[0021] The structure of the third layer of the backbone network includes a 1×1 convolution layer, a BN layer and a Relu activation function layer in sequence;

[0022] The structure of the fourth layer of the backbone network includes a 3×3 convolution layer, a BN layer and a Relu activation function layer in sequence;

[0023] The structure of the fifth layer of the backbone network includes a 1×1 convolutional layer, a BN layer and a channel-space dual attention module CSAM in sequence; the CSAM includes a cascaded channel attention module CAM and a spatial attention module SAM;

[0024] The structure of the sixth layer of the backbone network includes a Relu activation function layer.

[0025] In a possible implementation, the structure of the image classification network includes, in sequence: a 3×3 convolution layer, a BN layer, a Relu activation function, and a 1×1 convolution layer.

[0026] In a possible implementation, the method further includes:

[0027] The street view image dataset is normalized using the mean value and the standard deviation to obtain the preprocessed street view image dataset.

[0028] A semantic segmentation device for an image, the device comprising:

[0029] A first input unit, used to input the preprocessed street view image dataset into the backbone network for processing to obtain a first feature atlas and an attention feature atlas;

[0030] A second input unit is used to input the attention feature atlas into the DCP module for feature fusion and upsampling to obtain a second feature atlas;

[0031] A feature fusion unit, used for fusing the first feature atlas and the second feature atlas to obtain a third feature atlas;

[0032] The third input unit is used to input the third feature atlas into the image classification network to perform image semantic segmentation to obtain a semantic segmentation result.

[0033] In a possible implementation manner, the first input unit specifically includes:

[0034] A fourth input unit, used for inputting the preprocessed street view image data set into the first layer and the second layer structure of the backbone network to perform step-by-step feature extraction to obtain the first feature atlas;

[0035] A fifth input unit, used for inputting the first feature atlas into the third, fourth and fifth layer structures of the backbone network to perform step-by-step feature extraction to obtain a fourth feature atlas;

[0036] A feature atlas adding unit, configured to add the first feature atlas and the fourth feature atlas to obtain a preliminary attention feature atlas;

[0037] The sixth input unit is used to input the preliminary attention feature atlas into the sixth layer of the backbone network for processing to obtain the attention feature atlas.

[0038] In one possible implementation,

[0039] The first layer of the backbone network includes a 7×7 convolutional layer, a BN layer, a Relu activation function layer, and a maximum pooling layer in sequence;

[0040] The structure of the second layer of the backbone network includes a residual layer consisting of three bottleneck residual blocks;

[0041] The structure of the third layer of the backbone network includes a 1×1 convolution layer, a BN layer and a Relu activation function layer in sequence;

[0042] The structure of the fourth layer of the backbone network includes a 3×3 convolution layer, a BN layer and a Relu activation function layer in sequence;

[0043] The structure of the fifth layer of the backbone network includes a 1×1 convolutional layer, a BN layer, and a channel-space dual attention module CSAM in sequence;

[0044] The structure of the sixth layer of the backbone network includes a Relu activation function layer; the CSAM includes a cascaded channel attention module CAM and a spatial attention module SAM.

[0045] A semantic segmentation device for an image comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the semantic segmentation method for the image as described above is implemented.

[0046] A computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the above-mentioned method for semantic segmentation of an image.

[0047] Compared with the prior art, this application has the following beneficial effects:

[0048] The present application provides a method, apparatus, device and storage medium for semantic segmentation of an image. Specifically, when executing the semantic segmentation method of an image provided by an embodiment of the present application, the preprocessed street view image data set is first input into the backbone network for preliminary processing to generate a first feature atlas and an attention feature atlas. Next, the attention feature atlas is sent to the dilated convolution pyramid (DCP) module for multi-scale feature fusion and upsampling operations to generate a second feature atlas. Then, the first feature atlas is fused with the second feature atlas in the spatial and channel dimensions to form a comprehensive third feature atlas. Finally, the third feature atlas is input into the image classification network for the final image semantic segmentation processing, and a semantic segmentation result with precise annotations is output. The multi-scale information fusion mechanism introduced in the present application can better capture the feature relationship at different spatial positions. Through the DCP module and the attention mechanism, the features at different scales are effectively integrated, which helps to fully perceive details and global information, thereby enhancing the recognition of differences in feature importance. At the same time, the upsampling process of the DCP module can help integrate features at different scales, effectively understand the feature differences between different spatial positions, and enhance the detection and description capabilities of subtle structures and important objects. In addition, in the feature fusion process of the first and second feature atlases, the feature information from different channels and spatial positions is fully considered, which is conducive to the model's more comprehensive understanding of the differences and associations between various parts of the image and improves the ability to identify differences in feature importance. In addition, the result of feature fusion is input into the classification network as the third feature atlas for semantic segmentation, which further integrates and optimizes various feature representations and improves the accuracy and robustness of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 A method flow chart of a method for semantic segmentation of an image provided in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of the structure of a backbone network provided in an embodiment of the present application;

[0052] Figure 3 A method flow chart of a method for acquiring a feature atlas provided in an embodiment of the present application;

[0053] Figure 4A schematic diagram of the structure of an image classification network provided in an embodiment of the present application;

[0054] Figure 5 A schematic diagram of the structure of a semantic segmentation device for an image provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] To facilitate understanding of the technical solutions provided by the embodiments of the present application, the background technology involved in the embodiments of the present application will be described below.

[0056] The bank's mobile financial service vehicle model is an innovation in the field of financial services. By providing terminal devices such as self-service smart counters and mobile marketing tablets, it realizes convenient mobile financial services, solves the problem of insufficient regional bank branches, and enhances the convenience of financial services.

[0057] Bank mobile financial service vehicles mainly serve communities, towns and campuses with simple traffic environments, and it is feasible to realize autonomous driving in these areas. Environmental perception is the basis of autonomous driving. Accurate environmental perception is crucial for subsequent decision-making and control, and semantic segmentation is one of the key tasks.

[0058] Semantic segmentation methods are divided into two categories: traditional and deep learning. Traditional methods use low-dimensional manual features for segmentation, but their representation capabilities are insufficient and their robustness is low. In contrast, deep learning-based methods have advantages such as automatic feature extraction and high accuracy. However, existing deep learning methods usually fail to effectively identify the differences in features of different channels and spatial locations, which affects the accuracy of semantic segmentation.

[0059] In order to solve this problem, an embodiment of the present application provides a method, apparatus, device and storage medium for semantic segmentation of an image. First, a preprocessed street view image data set is input into the backbone network to obtain a first feature atlas and an attention feature atlas. Then, the attention feature atlas is sent to the hole convolution pyramid module DCP, and after feature fusion and upsampling, a second feature atlas is generated. Subsequently, the first feature atlas and the second feature atlas are feature fused to form a third feature atlas. Finally, the third feature atlas is input into the image classification network for image semantic segmentation, and an accurate semantic segmentation result is finally obtained. The present application effectively improves the ability to recognize the differences in feature importance between different channels and spatial positions in the image through the design of multi-stage feature extraction, fusion and attention mechanisms, thereby improving the accuracy and robustness of semantic segmentation.

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0061] See also Figure 1 , which is a flow chart of a method for semantic segmentation of an image provided by an embodiment of the present application, such as Figure 1 As shown, the semantic segmentation method of the image may include steps S101-S104:

[0062] S101: Input the preprocessed street view image dataset into the backbone network for processing to obtain a first feature atlas and an attention feature atlas.

[0063] The preprocessed street view image dataset is input into the backbone network for processing to obtain the first feature atlas and the attention feature atlas, which means that a series of preprocessing operations are first performed on the street view image data to ensure that these images have a uniform format and a suitable scale. Then, these preprocessed image data are input into the backbone network for further processing.

[0064] In a possible implementation, the method further includes:

[0065] The street view image dataset is normalized using the mean value and the standard deviation to obtain the preprocessed street view image dataset.

[0066] Normalizing the Street View image dataset usually involves using the mean and standard deviation to adjust the range and distribution of the data to make it more suitable for training neural network models. In this process, the mean and standard deviation of all pixel values ​​in the Street View image dataset are first calculated. Then, each pixel value is converted to a standardized value relative to the entire dataset by subtracting the mean and dividing by the standard deviation.

[0067] This normalization process helps eliminate the scale differences between different images, making the data distribution more stable and close to zero mean and unit variance. This can help avoid problems such as gradient disappearance or explosion. Finally, a preprocessed street view image dataset is obtained, in which the pixel values ​​are adjusted to a certain standard range.

[0068] In one possible implementation, the backbone network includes a six-layer structure. Figure 2 , Figure 2 A schematic diagram of the structure of a backbone network provided in an embodiment of the present application is shown in FIG. Figure 2InPut in is the input of the first layer of the backbone network.

[0069] like Figure 2 As shown in the figure, the first layer of the backbone network includes a 7×7 convolutional layer (Convolutional Layer, Conv), a batch normalization (BatchNormalization, BN) layer, a rectified linear unit (Rectified LinearUnit, Relu) activation function layer and a maximum pooling layer (MaxPooling Layer, MaxPool) in sequence;

[0070] The structure of the second layer of the backbone network includes a residual layer consisting of three bottleneck residual blocks (Bottleneck Residual Block, Bottleneck);

[0071] The structure of the third layer of the backbone network includes a 1×1 convolutional layer, a BN layer, and a Relu activation function layer in sequence;

[0072] The structure of the fourth layer of the backbone network consists of a 3×3 convolutional layer, a BN layer, and a Relu activation function layer in sequence;

[0073] The structure of the fifth layer of the backbone network includes a 1×1 convolutional layer, a BN layer, and a channel-spatial attention module (CSAM) in sequence; the CSAM includes a cascaded channel attention module (CAM) and a spatial attention module (SAM);

[0074] The structure of the sixth layer of the backbone network includes a Relu activation function layer. The sixth layer of the backbone network is connected to the fifth layer through an adder, and the adder is also connected to the input port of the backbone network.

[0075] See also Figure 3 , Figure 3 A method flow chart of a method for obtaining a feature atlas provided in an embodiment of the present application. Accordingly, S101 inputs the preprocessed street view image data set into the backbone network for processing to obtain a first feature atlas and an attention feature atlas, which can be specifically implemented through steps A1-A4:

[0076] A1: Input the preprocessed street view image dataset into the first and second layer structures of the backbone network to perform step-by-step feature extraction to obtain the first feature atlas.

[0077] The preprocessed street view image dataset is input into the first and second layers of the backbone network for step-by-step feature extraction to obtain the first feature atlas, which refers to the process of extracting key features from the original image through a series of hierarchical operations in the deep learning model. The first layer of the backbone network includes a 7×7 convolution layer, a BN layer, a Relu activation function layer, and a maximum pooling layer. The combination of these layers is used to perform preliminary feature extraction and downsampling operations on the image data, thereby capturing the basic features in the image and reducing the dimension of the data.

[0078] Subsequently, the input data is processed by the first layer and passed to the second layer of the backbone network, which consists of a residual layer consisting of three bottleneck residual blocks. The design of the residual block enables the network to better learn the residual information, that is, the difference between the original input and the expected output, thereby helping to deepen the learning depth of the network and improve the effect of feature expression. By stacking multiple residual blocks, the network can more effectively learn complex feature representations, further improving the representation ability and generalization performance of the model. Overall, inputting the street view image dataset into these layer structures realizes step-by-step feature extraction and conversion, which helps to provide more representative and abstract feature representations for subsequent tasks and improves the model's understanding and processing capabilities of street view image data.

[0079] Specifically, the convolution layer is used to identify patterns and features in the image, the BN layer helps to accelerate convergence and stabilize the training process. The Relu activation function layer introduces nonlinearity to improve the network's expressiveness, while the maximum pooling layer is used to downsample and reduce the data dimension. Subsequently, the image data processed by the first layer will be further passed to the second layer structure of the backbone network, which contains multiple bottleneck residual blocks (Bottleneck Residual Block, Bottleneck) for deeper feature extraction and representation learning. The entire process aims to effectively extract key features from the original image data, provide rich information for subsequent image processing tasks, and gradually build a more abstract and meaningful feature representation through the combination and interaction between different layers, laying the foundation for the model's learning and inference on street view image data.

[0080] A2: Input the first feature atlas into the third, fourth and fifth layer structures of the backbone network to perform step-by-step feature extraction to obtain a fourth feature atlas.

[0081] The first feature atlas is input into the third, fourth and fifth layers of the backbone network for step-by-step feature extraction to obtain the fourth feature atlas. This process describes how to further extract and transform features through operations at different levels in a deep learning model to obtain a richer and more abstract representation. The third layer of the backbone network includes a 1×1 convolution layer, a BN layer and a Relu activation function layer; the fourth layer consists of a 3×3 convolution layer, a BN layer and a Relu activation function layer; the fifth layer includes a 1×1 convolution layer, a BN layer and a CSAM module, where the CSAM module consists of a cascaded CAM and a spatial attention module SAM.

[0082] Specifically, after the first feature atlas is processed by the third layer, the feature information is further refined and processed through the application of 1×1 convolution operation, batch normalization and Relu activation function; then in the fourth layer, the 3×3 convolution operation, BN layer and Relu activation function further enhance the richness and expressiveness of the features; finally, in the fifth layer, the use of 1×1 convolution operation, BN layer and CSAM module can introduce channel and spatial attention mechanisms to help the network better focus on important areas and features, and improve the sensitivity and responsiveness to input data. Through this step-by-step feature extraction method, the network generates the fourth feature atlas after multiple rounds of processing, which contains more abstract and meaningful feature representations, providing more accurate and effective input features for subsequent tasks, thereby improving the performance and generalization ability of the overall model. The whole process shows how to gradually refine and transform feature information through a deep network structure, so as to achieve a deeper understanding and processing of input data.

[0083] A3: Add the first feature atlas and the fourth feature atlas to obtain a preliminary attention feature atlas.

[0084] The first feature atlas and the fourth feature atlas are added together to obtain a preliminary attention feature atlas. This step describes how to combine feature information extracted at different levels in a deep learning model to enhance the model's understanding and representation of input data. When the first feature atlas and the fourth feature atlas are added together, the feature representations obtained at different processing stages are actually fused to obtain a more comprehensive and comprehensive feature representation.

[0085] Specifically, the first feature atlas represents the original feature representation after processing in the first few layers, which contains basic image information and preliminary extracted features; while the fourth feature atlas has undergone deeper processing and transformation and has a more advanced and abstract feature representation. By adding these two feature atlases, the network can take into account the information of both low-level and high-level features at the same time, thereby generating a richer and more comprehensive preliminary attention feature atlas. This fusion method helps to comprehensively utilize the feature information extracted at different levels and provide more representative and meaningful feature representations for subsequent tasks. Ultimately, the preliminary attention feature atlas provides the entire model with richer and more comprehensive feature information, laying a solid foundation for subsequent tasks such as image semantic segmentation. This process shows how feature fusion can be used to enhance the model's ability to represent input data.

[0086] A4: Input the preliminary attention feature atlas into the sixth layer of the backbone network for processing to obtain the attention feature atlas.

[0087] The preliminary attention feature atlas is input into the sixth layer of the backbone network for processing to obtain the attention feature atlas. This step reveals how the deep learning model continues to process and transform the fused features in the last layer to further improve the quality and richness of feature representation. The sixth layer of the backbone network only includes the Relu activation function layer, which introduces nonlinear transformations to help the network better learn the relationship between complex data distributions and features.

[0088] By feeding the preliminary attention feature atlas into the sixth layer, the role of the Relu activation function can bring further nonlinear transformations, thereby deepening the expressiveness of the features and enhancing the network's sensitivity to the data. This process helps to further optimize and adjust the feature representation, thereby better capturing the key information and patterns in the input data. The attention feature atlas obtained after processing the activation function layer can more accurately focus on the features that are critical to the task and improve the understanding and inference of the input data. Ultimately, the obtained attention feature atlas represents the result of the fully processed and optimized feature representation, providing more representative and effective features for subsequent processing.

[0089] S102: Input the attention feature atlas into the DCP module for feature fusion and upsampling to obtain a second feature atlas.

[0090] The attention feature atlas is input into the DCP module for feature fusion and upsampling to obtain the second feature atlas, which shows how the deep learning model can further process the features through a complex module structure to achieve higher-level feature fusion and image reconstruction. The DCP module is designed to integrate feature maps of different receptive field sizes and effectively combine various contextual information to enhance the network's understanding and representation of data.

[0091] When the attention feature atlas is input into the DCP module, the receptive field is first expanded using dilated convolution to capture information in a wider range. Then, feature maps of different scales are fused through a pyramid structure to improve the diversity and richness of features. At the same time, during the feature fusion process, the DCP module also uses upsampling technology to increase the resolution of the feature map, so that the network can better restore and reconstruct the details of the input data. This design that combines multiple technologies helps to express the abstract representation and functionality of the input data.

[0092] The second feature atlas obtained after processing by the DCP module obtains a more representative and advanced feature representation, which contains features of different scales, richer contextual information and higher resolution. These features are very important for identifying objects, scenes or performing other visual tasks because they are more capable of capturing the essential characteristics and key details of the data. Ultimately, the second feature atlas provides a more powerful and complex feature representation, providing more accurate and valuable input for the next task. This process shows how to optimize the feature extraction and image reconstruction process through a modular and hierarchical approach, thereby enhancing the power and robustness of deep learning models.

[0093] S103: Fusing the first feature atlas and the second feature atlas to obtain a third feature atlas.

[0094] The first feature atlas and the second feature atlas are fused to obtain the third feature atlas, which describes how the deep learning model combines feature information extracted at different levels to generate a more comprehensive, integrated and meaningful feature representation. In this process, feature fusion plays a key role, which helps to enhance the understanding and representation of input data.

[0095] When the first feature atlas and the second feature atlas are fused, it is actually an organic combination of two feature representations from different processing stages. By fusing the two together, it is possible to comprehensively utilize the feature information extracted at different levels, thereby generating a richer and more comprehensive feature representation. This fusion method helps to comprehensively consider the information of low-level and high-level features, and improve the abstract representation and understanding of the input data.

[0096] The third feature atlas obtained after feature fusion represents a new feature representation generated by integrating information from different stages and different feature levels. Such feature sets are usually more helpful in achieving more accurate and effective results in various tasks because they are more capable of capturing the multi-faceted features and key information of the data. The third feature atlas provides the model with more comprehensive and rich information and provides more effective input features for the next task.

[0097] S104: Input the third feature atlas into an image classification network to perform image semantic segmentation to obtain a semantic segmentation result.

[0098] The third feature atlas is input into the image classification network for image semantic segmentation, and the semantic segmentation result is obtained, which shows how to use the high-level features processed by feature fusion to perform complex image segmentation tasks. In this process, the structure of the image classification network includes 3×3 convolution layers, BN (Batch Normalization) layers, Relu activation functions, and 1×1 convolution layers in sequence. The design of these layers helps to deeply mine and transform features to achieve accurate semantic segmentation.

[0099] When the third feature atlas is input into the image classification network, the features are first further spatially transformed and extracted through a 3×3 convolutional layer to capture more complex and abstract information. Subsequently, the BN layer can normalize the feature response, which helps to solve the gradient vanishing or exploding problem of the feature atlas and improves the convergence speed and stability of the network. The Relu activation function introduces nonlinear transformations to help the network better learn the nonlinear features and patterns of the data. Finally, a 1×1 convolutional layer is used to compress features and adjust dimensions to generate the final semantic segmentation results.

[0100] After being processed by the image classification network, the semantic segmentation results obtained will classify and label the image at the pixel level according to the input third feature atlas, and classify different regions into various semantic categories. This process involves inferring and classifying each pixel in the image to identify the boundaries and shapes of different objects and regions in the image. In this way, the input image can be understood and analyzed more deeply and meticulously, thereby achieving a more detailed description and detection of the image content.

[0101] The entire process shows how to combine high-level features and a specially designed classification network architecture to achieve the semantic segmentation task of an image. This process fully utilizes the multi-level feature extraction and representation capabilities of the deep learning model, providing strong support for achieving accurate and effective image segmentation results.

[0102] In a possible implementation, the structure of the image classification network includes: a 3×3 convolution layer, a BN layer, a Relu activation function, and a 1×1 convolution layer. Figure 4 . Figure 4 A schematic diagram of the structure of an image classification network provided in an embodiment of the present application.

[0103] Based on the contents of S101-S104, it can be known that based on the preprocessing of the street view image data set, the first feature atlas and the attention feature atlas are extracted through the backbone network. Subsequently, the attention feature atlas is processed using the atrous convolutional pyramid module DCP module to achieve feature fusion and upsampling, and generate a second feature atlas. Furthermore, a feature fusion operation is performed on the basis of the first feature atlas and the second feature atlas to obtain a third feature atlas. Finally, the third feature atlas is input into the image classification network for image semantic segmentation to complete the pixel-level classification of the image content, thereby producing accurate semantic segmentation results. By combining the attention mechanism and the DCP module, the present application improves the ability to recognize differences in feature importance between different channels and between different spatial positions in the image, thereby improving the accuracy and robustness of semantic segmentation.

[0104] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a semantic segmentation device for an image provided in an embodiment of the present application. Figure 5 As shown, the semantic segmentation device of the image includes:

[0105] A first input unit 501 is used to input the preprocessed street view image dataset into the backbone network for processing to obtain a first feature atlas and an attention feature atlas;

[0106] A second input unit 502 is used to input the attention feature atlas into the DCP module for feature fusion and upsampling to obtain a second feature atlas;

[0107] A feature fusion unit 503 is used to fuse the first feature atlas and the second feature atlas to obtain a third feature atlas;

[0108] The third input unit 504 is used to input the third feature atlas into the image classification network to perform image semantic segmentation to obtain a semantic segmentation result.

[0109] In a possible implementation, the first input unit 501 specifically includes:

[0110] A fourth input unit, used for inputting the preprocessed street view image data set into the first layer and the second layer structure of the backbone network to perform step-by-step feature extraction to obtain the first feature atlas;

[0111] A fifth input unit, used for inputting the first feature atlas into the third, fourth and fifth layer structures of the backbone network to perform step-by-step feature extraction to obtain a fourth feature atlas;

[0112] A feature atlas adding unit, configured to add the first feature atlas and the fourth feature atlas to obtain a preliminary attention feature atlas;

[0113] The sixth input unit is used to input the preliminary attention feature atlas into the sixth layer of the backbone network for processing to obtain the attention feature atlas.

[0114] In one possible implementation,

[0115] The first layer of the backbone network includes a 7×7 convolutional layer, a BN layer, a Relu activation function layer, and a maximum pooling layer in sequence;

[0116] The structure of the second layer of the backbone network includes a residual layer consisting of three bottleneck residual blocks;

[0117] The structure of the third layer of the backbone network includes a 1×1 convolution layer, a BN layer and a Relu activation function layer in sequence;

[0118] The structure of the fourth layer of the backbone network includes a 3×3 convolution layer, a BN layer and a Relu activation function layer in sequence;

[0119] The structure of the fifth layer of the backbone network includes a 1×1 convolutional layer, a BN layer, and a channel-space dual attention module CSAM in sequence;

[0120] The structure of the sixth layer of the backbone network includes a Relu activation function layer; the CSAM includes a cascaded channel attention module CAM and a spatial attention module SAM.

[0121] In a possible implementation, the structure of the image classification network includes, in sequence: a 3×3 convolution layer, a BN layer, a Relu activation function, and a 1×1 convolution layer.

[0122] In a possible implementation manner, the device further includes:

[0123] The preprocessing unit is used to perform normalization processing on the street view image dataset using the average value and the standard deviation to obtain the preprocessed street view image dataset.

[0124] In addition, an embodiment of the present application also provides a semantic segmentation device for an image, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the semantic segmentation method for the image as described above is implemented.

[0125] In addition, an embodiment of the present application further provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the semantic segmentation method of the image as described above.

[0126] The embodiment of the present application provides a semantic segmentation device for an image, which first uses a first input unit 501 to input a preprocessed street view image data set into a backbone network for processing to obtain a first feature atlas and an attention feature atlas, and uses a second input unit 502 to input the attention feature atlas into a DCP module for feature fusion and upsampling to obtain a second feature atlas. The feature fusion unit 503 performs feature fusion on the first feature atlas and the second feature atlas to obtain a third feature atlas, so that the third input unit 504 inputs the third feature atlas into an image classification network for image semantic segmentation to obtain a semantic segmentation result. By combining the attention mechanism and the DCP module, the present application improves the ability to recognize differences in feature importance between different channels and between different spatial positions in an image, thereby improving the accuracy and robustness of semantic segmentation.

[0127] The above is a detailed introduction to the semantic segmentation method, device, equipment and storage medium of an image provided by the present application. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

[0128] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0129] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims

1. A semantic segmentation method for an image, characterized in that: The method comprises: The preprocessed street view image dataset is input into the backbone network for processing to obtain the first feature atlas and the attention feature atlas; Input the attention feature atlas into the dilated convolutional pyramid module (DCP) module for feature fusion and upsampling to obtain a second feature atlas; Performing feature fusion on the first feature atlas and the second feature atlas to obtain a third feature atlas; The third feature atlas is input into an image classification network to perform image semantic segmentation to obtain a semantic segmentation result.

2. The method according to claim 1, characterized in that: The method of inputting the preprocessed street view image dataset into the backbone network for processing to obtain a first feature atlas and an attention feature atlas includes: Inputting the preprocessed street view image data set into the first layer and the second layer structure of the backbone network to perform step-by-step feature extraction to obtain the first feature atlas; Inputting the first feature atlas into the third, fourth and fifth layer structures of the backbone network to perform step-by-step feature extraction to obtain a fourth feature atlas; Adding the first feature atlas and the fourth feature atlas to obtain a preliminary attention feature atlas; The preliminary attention feature atlas is input into the sixth layer of the backbone network for processing to obtain the attention feature atlas.

3. The method according to claim 2, characterized in that The first layer of the backbone network includes a 7×7 convolution layer, a batch normalization layer BN layer, a rectified linear unit Relu activation function layer and a maximum pooling layer in sequence; The structure of the second layer of the backbone network includes a residual layer consisting of three bottleneck residual blocks; The structure of the third layer of the backbone network includes a 1×1 convolution layer, a BN layer and a Relu activation function layer in sequence; The structure of the fourth layer of the backbone network includes a 3×3 convolution layer, a BN layer and a Relu activation function layer in sequence; The structure of the fifth layer of the backbone network includes a 1×1 convolutional layer, a BN layer, and a channel-space dual attention module CSAM in sequence; The CSAM includes a cascaded channel attention module CAM and a spatial attention module SAM; The structure of the sixth layer of the backbone network includes a Relu activation function layer.

4. The method according to claim 1, characterized in that: The structure of the image classification network includes: a 3×3 convolution layer, a BN layer, a Relu activation function and a 1×1 convolution layer in sequence.

5. The method according to claim 1, characterized in that The method further comprises: The street view image dataset is normalized using the mean value and the standard deviation to obtain the preprocessed street view image dataset.

6. A semantic segmentation device for an image, characterized in that: The device comprises: A first input unit, used to input the preprocessed street view image dataset into the backbone network for processing to obtain a first feature atlas and an attention feature atlas; A second input unit is used to input the attention feature atlas into the DCP module for feature fusion and upsampling to obtain a second feature atlas; A feature fusion unit, used for fusing the first feature atlas and the second feature atlas to obtain a third feature atlas; The third input unit is used to input the third feature atlas into the image classification network to perform image semantic segmentation to obtain a semantic segmentation result.

7. The device according to claim 6, characterized in that The first input unit specifically includes: A fourth input unit, used for inputting the preprocessed street view image data set into the first layer and the second layer structure of the backbone network to perform step-by-step feature extraction to obtain the first feature atlas; A fifth input unit, used for inputting the first feature atlas into the third, fourth and fifth layer structures of the backbone network to perform step-by-step feature extraction to obtain a fourth feature atlas; A feature atlas adding unit, configured to add the first feature atlas and the fourth feature atlas to obtain a preliminary attention feature atlas; The sixth input unit is used to input the preliminary attention feature atlas into the sixth layer of the backbone network for processing to obtain the attention feature atlas.

8. The device according to claim 7, characterized in that The first layer of the backbone network includes a 7×7 convolutional layer, a BN layer, a Relu activation function layer, and a maximum pooling layer in sequence; The structure of the second layer of the backbone network includes a residual layer consisting of three bottleneck residual blocks; The structure of the third layer of the backbone network includes a 1×1 convolution layer, a BN layer and a Relu activation function layer in sequence; The structure of the fourth layer of the backbone network includes a 3×3 convolution layer, a BN layer and a Relu activation function layer in sequence; The structure of the fifth layer of the backbone network includes a 1×1 convolutional layer, a BN layer, and a channel-space dual attention module CSAM in sequence; The structure of the sixth layer of the backbone network includes a Relu activation function layer; the CSAM includes a cascaded channel attention module CAM and a spatial attention module SAM.

9. A semantic segmentation device for an image, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image semantic segmentation method as described in any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method for semantic segmentation of an image as described in any one of claims 1 to 5.

Images