Unstructured road scene instance segmentation method and system based on improved SOLO

Abstract

The present invention relates to an unstructured road scene instance segmentation method and system based on improved SOLO. The method comprises: preprocessing an acquired image dataset and dividing the image dataset into a training set and a test set; inputting the training set into a pre-built improved SOLOV2 instance segmentation model and training the model to obtain a trained improved SOLOV2 instance segmentation model; wherein the improved SOLOV2 instance segmentation model adds an enhancement path to the neck network of the SOLOV2 instance segmentation model to reduce the loss in the feature transfer process, and adopts dual attention to guide feature selection to adaptively select important features; inputting the test set into the trained improved SOLOV2 instance segmentation model and outputting the instance segmentation result. The present invention can reduce feature loss and improve the segmentation capability of objects of different scales.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an unstructured road scene instance segmentation method and system based on improved SOLO. Background Art

[0002] With the development of artificial intelligence, intelligent driving technology has emerged. Rapid and accurate environmental perception of road scenes is the primary task of autonomous driving. Currently, road scenes for vehicles can be divided into two categories: structured roads with regular shapes, such as urban trunk roads and highways. Unstructured roads, such as rural streets and non-arterial urban roads, have blurred boundaries, irregular shapes, and even partial damage. On unstructured roads, the positions of vehicles and pedestrians are relatively random, and objects overlap and occlude, making segmentation difficult.

[0003] The method combining maximum entropy threshold segmentation and edge detection published in existing literature performs a preliminary segmentation of the road image based on an entropy threshold, then corrects it through edge detection. Finally, it scans the pixels in the central area of ​​the image to extract the largest connected domain as the segmented road result. However, the lightweight unstructured road segmentation method published in existing literature modifies the MobilenetV2 feature extraction network for pixel prediction and introduces channel attention and dilated convolution to optimize segmentation results. Using an encoder-decoder structure, spatial attention and the edge detection Laplace operator are introduced to extract multi-level information to achieve a more accurate segmentation model.

[0004] The above segmentation methods achieve good results in most scenarios, but most of them are semantic segmentation methods and cannot distinguish between object instances. Instance segmentation, a key research area in computer vision, not only detects the location of objects but also labels the edges of each instance, simultaneously completing object detection and semantic segmentation. Existing instance segmentation methods can be divided into one-stage and two-stage methods. Two-stage methods, such as Mask-RCNN, first determine the region of interest (ROI) using anchors, then perform regression and mask prediction on this region. While highly accurate, they are complex and slow, making them unsuitable for practical applications. HTC employs a cascaded architecture to refine the two tasks, achieving good results. However, this multi-stage design comes with a relatively high computational cost. Deep Snake first gives an initial contour and uses recurrent convolution to continuously learn and adjust it, gradually deforming it to the true contour of the object, thereby obtaining accurate object boundaries and completing instance segmentation. One-stage methods, such as YOLACT, divide instance segmentation into two parallel subtasks, accelerating network computation. SOLO, proposed by WANG et al., distinguishes different instances based on object location and size. It divides the image into an S×S grid, with each grid responsible for classifying and predicting a single instance. SOLOV2 builds on SOLO by introducing a dynamic mask head and Matrix NMS, reducing computational complexity and further improving segmentation speed. However, these existing semantic segmentation methods are unable to simultaneously capture both object outlines and instances. In unstructured road scenes, where multiple objects overlap and vary significantly in scale, they are prone to missed or incorrect detections, resulting in poor segmentation accuracy. Summary of the Invention

[0005] In view of the above problems, the purpose of the present invention is to provide an unstructured road scene instance segmentation method and system based on improved SOLO, which can reduce feature loss and improve the segmentation ability of objects of different scales.

[0006] To achieve the above-mentioned objectives, in the first aspect, the technical solution adopted by the present invention is: an unstructured road scene instance segmentation method based on improved SOLO, which comprises: preprocessing the acquired image data set, dividing the image data set into a training set and a test set; inputting the training set into a pre-built improved SOLOV2 instance segmentation model, training the model, and obtaining a trained improved SOLOV2 instance segmentation model; wherein, the improved SOLOV2 instance segmentation model adds an enhancement path to the neck network of the SOLOV2 instance segmentation model to reduce the loss in the feature transfer process, and adopts dual attention to guide feature selection to adaptively select important features; inputting the test set into the trained improved SOLOV2 instance segmentation model, and outputting the instance segmentation result.

[0007] Furthermore, the acquired image dataset is preprocessed, including:

[0008] The image dataset is an unstructured road scene driving dataset, and the image dataset is converted into COCO format;

[0009] The image dataset is subjected to motion blur, horizontal flip and highlight bar adjustment in sequence.

[0010] Furthermore, the improved SOLOV2 instance segmentation model is constructed, including:

[0011] Add an enhanced path to the neck network and build a PAFPN network structure to shorten the transmission path between low-level and high-level information;

[0012] A dual attention module is added between the output of ResNet and each layer of the top-down path of the PAFPN network. The dual attention module consists of two channel attention modules and one spatial attention module. The channel attention focuses on channel information to determine important features, and the spatial attention focuses on spatial information to determine the location of important information.

[0013] Furthermore, the PAFPN network structure is constructed, including: adding a bottom-up connection path on the right side of FPN, and connecting different layers through a building block.

[0014] Furthermore, different layers are connected through a building block, including:

[0015] Perform convolution on a certain layer of features with a stride of 2 and a convolution kernel size of 3×3 to adjust the size of the feature;

[0016] The adjusted features are fused with the features of the FPN horizontal output, and finally the features of the next layer are obtained by convolution processing with a step size of 1 and a convolution kernel size of 3×3.

[0017] Furthermore, the PAFPN network structure is constructed, which also includes:

[0018] For the feature branch, the output of PAFPN is fused and all layers except the bottom layer are upsampled to obtain features of 1 / 4 the size of the input image. The top layer uses the convolution CoordConv method to add two channels to record the input coordinates and perceive the spatial information of the features.

[0019] The 4-layer features are added together and then subjected to 1*1 convolution, group normalization, and ReLU activation as the input of the feature branch;

[0020] The kernel branch and category branch directly use the output of PAFPN.

[0021] Furthermore, the channel attention module includes:

[0022] Divide the input feature F into two branches;

[0023] The first branch performs global average pooling in each channel to aggregate all feature information, and the second branch performs maximum pooling in each channel to retain only the most effective features;

[0024] The output feature size of the two branches after different pooling is C×1×1, where C represents the number of channels;

[0025] The pooled features are sequentially subjected to 1x1 convolution channel dimensionality reduction, nonlinear ReLU function activation, and 1x1 convolution channel dimensionality increase to restore the dimension and obtain the average pooled branch output feature F. avgpool And the maximum pooling branch output feature F maxpool ;

[0026] For feature F avgpool and F maxpool After adding fusion, the sigmoid function is used for activation and fused with the feature F to obtain the final output F out .

[0027] Furthermore, the spatial attention module includes:

[0028] The number of channels of feature F is adjusted to 1, and the output size is 1×H×W. The output features of maximum pooling and average pooling are merged in the channel dimension to obtain 2×H×W features. The merged features are convolved with a 7×7 convolution kernel;

[0029] The sigmoid function is used to activate and output a single-channel feature vector with spatial information weight, which is fused with the feature F to achieve adaptive refinement and obtain the final output F. out .

[0030] In the second aspect, the technical solution adopted by the present invention is: an unstructured road scene instance segmentation system based on improved SOLO, which includes: a first processing module, which preprocesses the acquired image data set and divides the image data set into a training set and a test set; a second processing module, which inputs the training set into a pre-built improved SOLOV2 instance segmentation model, trains the model, and obtains a trained improved SOLOV2 instance segmentation model; wherein, the improved SOLOV2 instance segmentation model adds an enhancement path to the neck network of the SOLOV2 instance segmentation model to reduce the loss in the feature transfer process, and adopts dual attention to guide feature selection to adaptively select important features; an output module, which inputs the test set into the trained improved SOLOV2 instance segmentation model and outputs the instance segmentation result.

[0031] In a third aspect, the technical solution adopted by the present invention is: a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by a computing device, the computing device executes any one of the above methods.

[0032] The present invention has the following advantages due to the adoption of the above technical solution:

[0033] 1. This paper introduces the PAFPN structure based on SOLOV2, adds a new bottom-up path after FPN, reduces feature loss, enhances feature information fusion, and improves the segmentation ability of objects of different scales.

[0034] 2. The present invention adds a dual attention mechanism, focusing on channel and position features at the same time, filtering out irrelevant or unimportant information, improving the feature expression ability of the network, and better grasping detailed information.

[0035] 3. The present invention can ensure the instance segmentation speed while further improving the model accuracy, and obtain a high-level instance segmentation model suitable for unstructured road scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 4 is a flow chart of an unstructured road scene instance segmentation method based on improved SOLO in an embodiment of the present invention;

[0037] Figure 2 2 is a diagram of a SOLO instance segmentation network structure according to an embodiment of the present invention;

[0038] Figure 3 This is a diagram of a mask branch structure in an embodiment of the present invention;

[0039] Figure 4 This is a diagram of the PAFPN network structure in an embodiment of the present invention;

[0040] Figure 5 Schematic diagram of feature branch processing in an embodiment of the present invention;

[0041] Figure 6 2. It is a schematic diagram of adding a dual attention module in an embodiment of the present invention;

[0042] Figure 7 Schematic diagram of adding a channel attention module in an embodiment of the present invention;

[0043] Figure 8 Schematic diagram of adding a spatial attention module in an embodiment of the present invention;

[0044] Figure 9 This is a loss curve diagram of the training process in an embodiment of the present invention;

[0045] Figure 101 is a diagram showing the prediction effects of different instance segmentation networks in an embodiment of the present invention;

[0046] Figure 11a This is an example segmentation effect diagram under a motion blur environment according to an embodiment of the present invention;

[0047] Figure 11b This is an example segmentation effect diagram under a strong light environment in an embodiment of the present invention;

[0048] Figure 11c This is an example segmentation effect diagram in a night environment in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of the present invention.

[0050] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0051] Since most current vision-based environmental perception methods are semantic segmentation methods, they are unable to simultaneously obtain target outlines and instances. In unstructured road scenes, multiple targets overlap and have large scale differences, which makes it easy to miss or misdetect, resulting in poor segmentation accuracy. Therefore, the present invention introduces the Path Aggregation Feature Pyramid Network (PAFPN) and dual attention mechanism based on the instance segmentation network SOLOV2, making full use of features to improve fusion efficiency and thus improve instance segmentation accuracy. First, an enhanced path is added to the FPN (Feature Pyramid Network) structure to reduce the loss in the feature transfer process. Secondly, dual attention is used to guide feature selection, adaptively select important features, suppress redundant information, improve the ability to extract detailed features, enhance the feature representation of category branches and mask branches, and thus improve the accuracy of mask prediction. The present invention has more accurate control of instance boundaries, and compared with SOLOV2 and Mask R-CNN, the average accuracy is improved by 2.0% and 2.2% respectively, and the detection frame rate is increased to 6.1. At the same time, it has good segmentation performance in different environments.

[0052] In one embodiment of the present invention, a method for segmenting unstructured road scenes based on improved SOLO is provided. In this embodiment, Figure 1 As shown, the method includes the following steps:

[0053] 1) After preprocessing the acquired image dataset, the image dataset is divided into a training set and a test set;

[0054] 2) Inputting the training set into a pre-built improved SOLOV2 instance segmentation model and training the model to obtain a trained improved SOLOV2 instance segmentation model; wherein the improved SOLOV2 instance segmentation model adds an enhancement path to the neck network of the SOLOV2 instance segmentation model to reduce the loss in the feature transfer process, and adopts dual attention to guide feature selection to adaptively select important features;

[0055] Among them, important features include color, texture, shape, size, boundary and context information;

[0056] 3) Input the test set into the trained improved SOLOV2 instance segmentation model and output the instance segmentation result.

[0057] In step 1) above, the dataset cannot cover all scenarios because the vehicle's environment is ever-changing during autonomous driving. Using a dataset without data augmentation to train the model fails to learn sufficiently rich features and is not universally applicable to segmenting complex driving scenarios. Therefore, preprocessing the dataset eliminates underfitting and overfitting, which are common problems in model training. This allows for better model training and results in higher generalization capabilities.

[0058] Therefore, in this embodiment, the acquired image data set is preprocessed, including the following steps:

[0059] 1.1) The image dataset is an unstructured road scene driving dataset. The image dataset is converted into COCO format;

[0060] In this example, the open-source Indian Driving Dataset (IDD) for unstructured scenes is used. The IDD includes 10,004 images of rural and urban road scenes, most in 1080p resolution, with a few in 720p resolution. The images are annotated with instances from 13 categories. The dataset is processed for unstructured road driving scenes, retaining seven instance categories: person, rider, bicycle, car, truck, bus, and motorcycle. A total of 7,640 annotated images were selected for this study and converted to COCO format for training purposes.

[0061] 1.2) The image dataset is subjected to dynamic blur, horizontal flip and bright bar adjustment in sequence.

[0062] In this embodiment, dynamic blur is used to simulate the situation where the camera-captured image is blurred due to vehicle bumps; horizontal flipping is used to enhance the directional invariance of the network model and help the instance segmentation network learn instances at different locations; since the color and texture features of the camera-captured images vary under different weather conditions, the image is randomly adjusted in brightness through brightness adjustment to adapt to different ambient lighting conditions.

[0063] In the above step 2), SOLO is a single-stage real-time instance segmentation network based on mask prediction. The network structure mainly consists of four parts: ResNet backbone, FPN feature pyramid network, category branch, and mask branch. Figure 2 shown.

[0064] For an image with an input size of 1920x1080, the convolution layer with the same output feature size is called a Stage. ResNet101 has 5 Stages. The output of the first Stage, which occupies a large amount of memory, is discarded. The outputs of the other 4 Stages are relatively large compared to the original Stage. Figure 1 Four different feature sizes are used: / 4, 1 / 8, 1 / 16, and 1 / 32. The output of the last residual block from Stage 2 to Stage 5 is used as the input for FPN. The ResNet101 feature extraction network is shown in Table 1.

[0065] Table 1 ResNet101 feature extraction network

[0066]

[0067] In the table, in_size and out_size represent the sizes of input and output features respectively; W, H, and C represent the width, height, and number of channels of the features respectively.

[0068] SOLO divides the image into S×S grids. For instances in the image, each instance corresponds to a unique grid according to its position and size. The output of the semantic category branch is S×S×C, where C is the total number of categories. The mask branch generates an instance mask for each grid. This branch contains a total of S×S channels. The kth channel is used to predict the instance of the (i, j)th position grid of the semantic category branch, k=i×S+j, and a maximum of S×S instances can be predicted. Each instance is assigned to an independent grid as the center position category according to its own center coordinates, and then encoded to obtain S×S channels to generate an instance mask of the corresponding category. In the subsequent SOLOV2, a dynamic mask terminal was introduced, which decomposed the mask branch into two parallel branches. The kernel branch is used for convolution kernel learning, and the feature branch is used for feature learning. Finally, the outputs of the two branches are combined to obtain the final mask prediction. The mask branch structure is as follows Figure 3 shown.

[0069] The original SOLO model uses an FPN structure for prediction of features at different scales. After extracting features from the input image through ResNet, the required four-layer output features are obtained. These features are then fused via lateral connections through the top-down FPN structure. Because the number and size of feature channels vary across different layers, 1×1 convolution and interpolation are used to adjust the number of channels and feature size. After fusion, the features of each layer of the FPN are processed by the mask branch and the semantic branch to obtain the final prediction result. Features at different levels contain inconsistent information. Low-level features are larger in size and retain more spatial location information and details, while high-level features are smaller in size and contain richer semantic information and a larger receptive field.

[0070] ResNet101 is a bottom-up architecture consisting of 101 layers. Transmitting shallow information through such a large number of layers often results in significant loss. FPN, on the other hand, is a top-down architecture, which results in a longer path for information transmission from lower layers to higher layers, leading to significant loss of target location information. Furthermore, FPN is a unidirectional structure, only fusing adjacent feature layers. This means that features in one layer are only fused with those in the previous layer, making it difficult to fully utilize shallow features.

[0071] In order to improve the instance segmentation accuracy of unstructured road scenes, the improved SOLOV2 instance segmentation model constructed in the present invention includes the following steps:

[0072] 2.1) Add an enhanced path to the neck network and build a PAFPN network structure to shorten the transmission path between low-level and high-level information;

[0073] 2.2) A dual attention module (DSA) is added between the output of ResNet and each layer of the top-down path of the PAFPN network (i.e., between layers P2-P5 of the PAFPN network). The dual attention module consists of two channel attention modules and one spatial attention module. The channel attention focuses on channel information to determine important features, and the spatial attention focuses on spatial information to determine the location of important information.

[0074] In the above step 2.1), the PAFPN network structure is constructed, specifically: a bottom-up connection path is added to the right side of the FPN, and different layers are connected through a building block, such as Figure 4 shown.

[0075] The different layers are connected through a building block, which includes the following steps:

[0076] 2.1.1) Perform convolution on a certain layer of feature Mi with a stride of 2 and a convolution kernel size of 3×3 to adjust the feature size;

[0077] 2.1.2) The adjusted features are fused with the feature Pi+1 output by the FPN horizontal output, and finally a convolution process with a stride of 1 and a convolution kernel size of 3×3 is used to obtain the features of the next layer.

[0078] PAFPN utilizes two transmission methods: top-down and bottom-up. The top-down transmission method transmits high-level semantic information, while the bottom-up transmission method transmits positional information. This structure significantly shortens the transmission path between low-level and high-level information, avoiding the problem of target position information loss caused by the multi-layer convolution operations of low-level information in the ResNet structure. The bottom-up connection (enhancement) path accelerates the flow of features, fully utilizing the precise position information stored in low-level information, and improving model accuracy.

[0079] In the above step 2.1), the PAFPN network structure is constructed. For the feature branch, the following steps are also included:

[0080] 2.1.3) For the feature branch, the PAFPN output is fused and all but the bottom layer is upsampled to produce features 1 / 4 the size of the input image. The top layer uses the CoordConv convolution method to add two channels to record the input coordinates, perceive the spatial information of the features, and ensure the input of positional encoding;

[0081] 2.1.4) After the four layers of features are added together, they are processed by 1*1 convolution, GN (group norm) and ReLU activation as the input of the feature branch, as shown in the following example: Figure 5 As shown;

[0082] 2.1.5) The kernel branch and category branch directly use the output of PAFPN, such as Figure 5 shown.

[0083] In step 2.2) above, the attention mechanism in deep learning originates from the fact that human vision can quickly scan the entire image and selectively focus on important parts while ignoring irrelevant or unimportant parts. This mechanism can focus on the key areas of the task from the global image while suppressing irrelevant information. This avoids the interference of a large number of irrelevant features, allocates more resources to process important information, and thus improves the feature extraction ability of the neural network. In unstructured road driving scenarios, vehicles and pedestrians easily overlap, and the boundaries are blurred and complex, which often leads to segmentation errors and makes it difficult to distinguish instances.

[0084] Therefore, in this embodiment, Figure 6 As shown in Figure 1, a dual attention module is added between the output of ResNet and the output of PAFPN network. ResNet outputs C2, C3, C4, and C5 are used for pyramid structure fusion. All channels of these feature maps have the same weight, that is, they all receive the same attention. However, some channels often contain some useless features, so the channel attention module is introduced to enhance the features of important channels and suppress the features of unimportant channels. The channel attention module is as follows: Figure 7 shown.

[0085] Specifically, a dual attention module is added between the output of ResNet and the output of the PAFPN network. The addition of the channel attention module includes the following steps:

[0086] 2.2.1) Divide the input feature F into two branches;

[0087] 2.2.2) The first branch performs global average pooling within each channel to aggregate all feature information. The second branch performs maximum pooling within each channel to retain only the most significant features. In this embodiment, the most significant features related to the target category are considered the most significant features, such as the edge, texture, or shape of the target.

[0088] Among them, the first branch performs global average pooling in each channel to aggregate all feature information. For the global average pooling c of feature F, avgpool The expression is as follows:

[0089]

[0090] Where H and W represent the height and width of the feature map; F(i, j) represents the pixel at the (i, j)th position;

[0091] The second branch performs maximum pooling in each channel, retaining only the most effective features, and the maximum pooling c maxpool The expression is as follows:

[0092] C maxpool =MAX[F(i,j)]i∈[1,H], j∈[1,W] (2)

[0093] 2.2.3) The output feature size of the two branches after different pooling is C×1×1, where C represents the number of channels;

[0094] 2.2.4) The pooled features are sequentially subjected to 1x1 convolution channel dimensionality reduction to reduce the amount of computation, nonlinear ReLU function activation and 1x1 convolution channel dimensionality increase, and the dimension is restored to obtain the average pooling branch output feature F avgpool And the maximum pooling branch output feature F maxpool ;

[0095]

[0096]

[0097] 2.2.5) For feature F avgpool and F maxpool After adding fusion, the sigmoid function is used for activation and fused with the feature F to obtain the final output F out .

[0098] Among them, the final output F out for:

[0099]

[0100] Where, σ represents the sigmoid activation function; f conv2 Respectively represent dimensionality reduction convolution and dimensionality increase convolution; f ReLu Represents the ReLU activation function; Represents element-wise matrix dot product.

[0101] In the above step 2.2), due to the horizontal connection of the original FPN network from bottom to top and from top to bottom, there is a lack of spatial dependency between pixels. Using channel attention alone cannot solve the spatial correlation problem. Therefore, the spatial attention mechanism is introduced to obtain the important position areas in space. The spatial attention module is as follows: Figure 8 shown.

[0102] Adding a spatial attention module involves the following steps:

[0103] 2.2.2.1) Adjust the number of channels of feature F to 1, and the output size to 1 × H × W. Combine the output features of max pooling and average pooling in the channel dimension to obtain a 2 × H × W feature. Convolve the combined feature with a 7 × 7 convolution kernel to obtain a larger receptive field.

[0104] Among them, unlike the pooling operation of channel attention, the pooling operation of spatial attention is performed on the channel axis;

[0105] 2.2.2.2) The sigmoid function is used to activate and output a single-channel feature vector with spatial information weight, which is then fused with the feature F to achieve adaptive refinement and obtain the final output F. out .

[0106] Final output F out Public:

[0107]

[0108] Example: This example uses a Linux desktop computer running Ubuntu 18.04, an Intel i3-3220 CPU with 16GB of RAM, and a GeForce RTX 2080Ti GPU with 11GB of video memory. The instance segmentation network model was built using the mmdetection framework and programmed in Python. CUDA 10.0 was used for the Unified Compute Device Architecture.

[0109] 1) Instance segmentation model training parameter setting:

[0110] In this embodiment, the instance segmentation model accepts two images each time, that is, batch_size=2, and training one batch is one iteration. The entire data set is trained once as one epoch, and a total of 36 epochs are trained. The SGD optimizer is used to update the network weights, with a momentum of 0.9. A warm-up learning rate strategy is used to avoid model oscillation and failure to converge due to an excessively large initial learning rate. The learning rate for the first 1000 iterations is increased from 3x10 -4 Increases linearly to 3x10 -2 , and then the learning rate is reduced to 1 / 10 of the original in the 27th and 33rd epochs, so that the gradient gradually converges to the optimal range.

[0111] 2) Evaluation indicators:

[0112] In this embodiment, the average precision AP is used to evaluate the instance segmentation network model. The intersection-over-union ratio is introduced. For instance segmentation, IOU is the ratio of the intersection and union of the predicted mask and the true mask area. 50 and AP 75 represents the average precision when the threshold is set to 0.5 and 0.75, AP L 、AP M and AP S Represents the average accuracy of large, medium and small scale targets respectively

[19] These indicators can fully evaluate the model from different scales.

[0113] The IOU expression is as follows:

[0114]

[0115] Where T p Correctly predicted positive samples; F p 、F n are incorrectly predicted positive and negative samples, respectively.

[0116] 3) Loss function:

[0117] In this embodiment, the category classification loss is calculated using the focal loss function to alleviate sample imbalance, and the mask loss is calculated using the Dice loss function. The total loss function expression is as follows:

[0118] L=L cate +λL tns (9)

[0119] L cate represents the category branch loss, λ is a hyperparameter, and this paper sets λ=3, L ins represents the mask prediction loss, L cate The expression is as follows:

[0120] L cate = -α(1-p) γ log(p) (10)

[0121] Where α represents the category weight, γ represents the weight of the difficult-to-distinguish sample. In this paper, α=0.25 and γ=2 are set, and p represents the predicted probability of the category. ins The expression is as follows:

[0122]

[0123]

[0124] N pos Indicates the number of positive samples; m k The predicted value of the mask, is the true value of the mask; Indicates m k and Dice loss function; Indicates when =1 when p is equal to 1, otherwise equal to 0; x,y Represents the mask prediction value of the feature map (x, y) position; q x,y Represents the true value of the mask at the position of the feature map (x, y). The loss curve in the training process of this article is as follows Figure 9 shown.

[0125] 4) Improve model ablation experiment

[0126] In order to verify the effectiveness of the improved model of the present invention, an ablation experiment was designed to test the impact of each improved part on the model. This embodiment uses the PAFPN structure and introduces the dual attention module DSA to form an improved model based on SOLOV2, and uses AP, AP 50 、AP 75 、AP L 、AP M 、AP s The model performance is evaluated using seven indicators: , FPS, and FPS. The results on the test set are shown in Table 2:

[0127] Table 2 Comparison of ablation experiment results

[0128]

[0129] As shown in Table 2, after using the PAFPN module, AP, AP50, and AP75 increased by 1.2%, 1.4%, and 1.4%, respectively. For objects of different scales, the average precision of instance segmentation for large, medium, and small scales increased by 2.3%, 2.2%, and 0.5%, respectively. The FPS increased from 5.7 to 6.1, indicating that PAFPN's bottom-up enhancement path enables more effective feature fusion, thereby improving instance segmentation. The channel attention component of the dual attention module uses max pooling to extract local features and global pooling to capture the global context of features. After fusion, it guides the importance of channels. The features of the current layer and the previous layer are fused after determining which are important features through channel attention. Spatial attention is then used to guide the spatial location of important features, greatly improving the model's feature expression ability and further improving accuracy without significantly affecting speed. Compared to SOLOV2, the integration of the PAFPN and DSA modules improves AP, AP50, and AP75 by 2.0%, 2.4%, and 2.5%, respectively. The average segmentation accuracy of large, medium, and small instance sizes increases by 3.8%, 3.0%, and 0.6%, respectively, and the FPS increases by 0.4, demonstrating the effectiveness of this architecture.

[0130] 5) Performance comparison of different instance segmentation models:

[0131] We selected mainstream instance segmentation network models, including YOLACT, Mask-RCNN, SOLO, and SOLOV2, for performance comparison with the model in this embodiment (Ours). To ensure the effectiveness of the comparison, all of these models used ResNet101 as the backbone extraction network and were trained on the IDD dataset with instance annotations selected in this paper. Model performance was evaluated on the test set using six metrics: AP, AP50, AP75, APL, APM, and APS. Table 3 shows the performance comparison of different instance segmentation models.

[0132] Table 3 Comparative experimental results of different instance segmentation models

[0133]

[0134] As shown in Table 3, the segmentation accuracy of our method improves by 7.0%, 2.2%, 4.2%, and 2.0% compared to YOLACT, Mask-RCNN, YOLACT, SOLO, and SOLOV2, respectively. Segmentation accuracy at other scales also improves to varying degrees. A comprehensive analysis comparing the performance of our method with current mainstream instance segmentation models demonstrates that our method offers superior segmentation performance.

[0135] The method of the present invention and the instance segmentation results of Mask-rcnn, SOLO, and SOLOV2 are visualized. Figure 10 As shown in the figure, when the middle of the vehicle is completely obscured by a person, SOLO and MaskRCNN predict one vehicle as two instances, while the method of the present invention correctly predicts them. When the two riders overlap significantly, SOLO and MaskRCNN's mask predictions show some errors. Although SOLOV2 can correctly distinguish them, its edge segmentation is poor. The method of the present invention can correctly distinguish instances and more precisely process contours.

[0136] Under different environmental conditions, the segmentation effect of the method of the present invention is as follows: Figures 11a to 11c As shown in the figure, it can be seen that the method of the present invention can achieve relatively accurate segmentation in the case of dynamic blur, strong light, night, etc. Therefore, the model of the present invention has good generalization ability.

[0137] In summary, the method of the present invention introduces the PAFPN structure, adds an enhancement path, speeds up information flow, improves feature fusion efficiency, and improves the accuracy of the model. By adding a dual attention module and efficiently utilizing position and channel information, a better segmentation effect is achieved. The detection frame rate is increased from 5.7 to 6.1, and the average accuracy is increased by 7.0% and 2.2% compared with SOLO and SOLOV2 respectively. Compared with other mainstream networks, there are different degrees of improvement. After data set preprocessing to increase data diversity, the trained model can cope with different environmental conditions and has strong robustness. Experiments show that the improved model of the present invention achieves a balance between speed and accuracy and has good performance.

[0138] In one embodiment of the present invention, a system for segmenting unstructured road scenes based on improved SOLO is provided, which includes:

[0139] The first processing module preprocesses the acquired image data set and divides the image data set into a training set and a test set;

[0140] The second processing module inputs the training set into a pre-built improved SOLOV2 instance segmentation model and trains the model to obtain a trained improved SOLOV2 instance segmentation model. The improved SOLOV2 instance segmentation model adds an enhancement path to the neck network of the SOLOV2 instance segmentation model to reduce the loss in the feature transfer process, and adopts dual attention to guide feature selection to adaptively select important features.

[0141] The output module inputs the test set into the trained improved SOLOV2 instance segmentation model and outputs the instance segmentation results.

[0142] In the above embodiment, the acquired image data set is preprocessed, including:

[0143] The image dataset is an unstructured road scene driving dataset, and the image dataset is converted into COCO format;

[0144] The image dataset is subjected to motion blur, horizontal flip and highlight bar adjustment in sequence.

[0145] In the above embodiment, the improved SOLOV2 instance segmentation model constructed includes:

[0146] Add an enhanced path to the neck network and build a PAFPN network structure to shorten the transmission path between low-level and high-level information;

[0147] A dual attention module is added between the output of ResNet and the output of the PAFPN network. The dual attention module includes two channel attention modules and one spatial attention module. The channel attention focuses on the channel information to determine important features, and the spatial attention focuses on the spatial information to determine the location of important information.

[0148] In this embodiment, constructing the PAFPN network structure includes: adding a bottom-up connection path on the right side of the FPN, and connecting different layers through a building block.

[0149] Specifically, different layers are connected through a building block, including:

[0150] Perform convolution on a certain layer of features with a stride of 2 and a convolution kernel size of 3×3 to adjust the size of the feature;

[0151] The adjusted features are fused with the features of the FPN horizontal output, and finally the features of the next layer are obtained by convolution processing with a step size of 1 and a convolution kernel size of 3×3.

[0152] In this embodiment, building a PAFPN network structure further includes:

[0153] For the feature branch, the output of PAFPN is fused and all layers except the bottom layer are upsampled to obtain features of 1 / 4 the size of the input image. The top layer uses the CoordConv method to add two channels to record the input coordinates and perceive the spatial information of the features.

[0154] The 4-layer features are added together and then activated by 1*1 convolution, GN and ReLU as the input of the feature branch;

[0155] The kernel branch and category branch directly use the output of PAFPN.

[0156] In the above embodiment, a dual attention module is added between the output of ResNet and the output of PAFPN network, wherein the channel attention module includes:

[0157] Divide the input feature F into two branches;

[0158] The first branch performs global average pooling in each channel to aggregate all feature information, and the second branch performs maximum pooling in each channel to retain only the most effective features;

[0159] The output feature size of the two branches after different pooling is C×1×1, where C represents the number of channels;

[0160] The pooled features are sequentially subjected to 1x1 convolution channel dimensionality reduction, nonlinear ReLU function activation, and 1x1 convolution channel dimensionality increase to restore the dimension and obtain feature F. avgpooland F maxpool ;

[0161] For feature F avgpool and F maxpool After adding fusion, the sigmoid function is used for activation and fused with the feature F to obtain the final output F out .

[0162] The spatial attention module includes:

[0163] The number of channels of feature F is adjusted to 1, and the output size is 1×H×W. The output features of maximum pooling and average pooling are merged in the channel dimension to obtain 2×H×W features. The merged features are convolved with a 7×7 convolution kernel;

[0164] The sigmoid function is used to activate and output a single-channel feature vector with spatial information weight, which is fused with the feature F to achieve adaptive refinement and obtain the final output F. out .

[0165] The system provided in this embodiment is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for specific processes and detailed contents, which will not be repeated here.

[0166] A computing device provided in one embodiment of the present invention may be a terminal and may include: a processor, a communications interface, memory, a display, and an input device. The processor, communications interface, and memory communicate with each other via a communications bus. The processor is configured to provide computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. When executed by the processor, the computer program implements an unstructured road scene instance segmentation method based on an improved SOLO. The internal memory provides an environment for the operating system and computer program in the non-volatile storage medium to run. The communications interface is configured to communicate with an external terminal via wired or wireless communication, where wireless communication may be achieved via Wi-Fi, a network management provider, NFC (near field communication), or other technologies. The display may be a liquid crystal display or an electronic ink display. The input device may be a touchscreen covering the display, a keypad, a trackball, or a touchpad provided on the computing device housing, or an external keyboard, touchpad, or mouse. The processor may invoke logic instructions in the memory.

[0167] In addition, the logical instructions in the above-mentioned memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0168] In one embodiment of the present invention, a computer program product is provided, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments.

[0169] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores server instructions. The computer instructions enable a computer to execute the methods provided in the above embodiments.

[0170] The above embodiment provides a computer-readable storage medium, whose implementation principle and technical effects are similar to those of the above method embodiment, and will not be repeated here.

[0171] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0172] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0173] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An unstructured road scene instance segmentation method based on improved SOLO, characterized by: include: After preprocessing the acquired image dataset, the image dataset is divided into a training set and a test set; The training set is input into a pre-built improved SOLOV2 instance segmentation model, and the model is trained to obtain a trained improved SOLOV2 instance segmentation model. The improved SOLOV2 instance segmentation model adds an enhancement path to the neck network of the SOLOV2 instance segmentation model to reduce the loss in the feature transfer process, and adopts dual attention to guide feature selection to adaptively select important features. Input the test set into the trained improved SOLOV2 instance segmentation model and output the instance segmentation results; The improved SOLOV2 instance segmentation model constructed includes: Add an enhanced path to the neck network and build a PAFPN network structure to shorten the transmission path between low-level and high-level information; A dual attention module is added between the output of ResNet and each layer of the top-down path of the PAFPN network. The dual attention module consists of two channel attention modules and one spatial attention module. The channel attention focuses on channel information to determine important features, and the spatial attention focuses on spatial information to determine the location of important information. Constructing the PAFPN network structure includes: adding a bottom-up connection path on the right side of the FPN, and connecting different layers through a building block; Channel attention module, including: Divide the input feature F into two branches; The first branch performs global average pooling in each channel to aggregate all feature information, and the second branch performs maximum pooling in each channel to retain only the most effective features; The output feature size of the two branches after different pooling is C×1×1, where C represents the number of channels; The pooled features are sequentially subjected to 1×1 convolution channel dimensionality reduction, nonlinear ReLU function activation, and 1×1 convolution channel dimensionality increase to restore the dimension and obtain the average pooling branch output feature F. avgpool And the maximum pooling branch output feature F maxpool ; For feature F avgpool and F maxpool After adding fusion, the sigmoid function is used for activation and fused with the feature F to obtain the final output F out ; Spatial attention module, including: The number of channels of feature F is adjusted to 1, and the output size is 1×H×W. The output features of maximum pooling and average pooling are merged in the channel dimension to obtain 2×H×W features. The merged features are convolved with a 7×7 convolution kernel; where H and W represent the height and width of the feature map; The sigmoid function is used to activate and output a single-channel feature vector with spatial information weight, which is fused with the feature F to achieve adaptive refinement and obtain the final output F. out .

2. The unstructured road scene instance segmentation method based on improved SOLO as claimed in claim 1, characterized in that: The acquired image dataset is preprocessed, including: The image dataset is an unstructured road scene driving dataset, and the image dataset is converted into COCO format; The image dataset is subjected to motion blur, horizontal flip and highlight bar adjustment in sequence.

3. The unstructured road scene instance segmentation method based on improved SOLO as claimed in claim 1, characterized in that: The different layers are connected by a building block, including: Perform convolution processing on each layer of features with a stride of 2 and a convolution kernel size of 3×3 to adjust the feature size; The adjusted features are fused with the features of the FPN horizontal output, and finally the features of the next layer are obtained by convolution processing with a step size of 1 and a convolution kernel size of 3×3.

4. The unstructured road scene instance segmentation method based on improved SOLO as claimed in claim 1, characterized in that: Building the PAFPN network structure also includes: For the feature branch, the output of PAFPN is fused and all layers except the bottom layer are upsampled to obtain features of 1 / 4 the size of the input image. The top layer uses the convolution CoordConv method to add two channels to record the input coordinates and perceive the spatial information of the features. The four layers of features are added together and then subjected to 1×1 convolution, group normalization, and ReLU activation as the input of the feature branch; The kernel branch and category branch directly use the output of PAFPN.

5. An unstructured road scene instance segmentation system based on improved SOLO, used to implement the unstructured road scene instance segmentation method based on improved SOLO according to any one of claims 1 to 4, characterized in that: include: The first processing module preprocesses the acquired image data set and divides the image data set into a training set and a test set; The second processing module inputs the training set into a pre-built improved SOLOV2 instance segmentation model and trains the model to obtain a trained improved SOLOV2 instance segmentation model. The improved SOLOV2 instance segmentation model adds an enhancement path to the neck network of the SOLOV2 instance segmentation model to reduce the loss in the feature transfer process, and adopts dual attention to guide feature selection to adaptively select important features. The output module inputs the test set into the trained improved SOLOV2 instance segmentation model and outputs the instance segmentation results.

6. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any one of the methods of claims 1 to 4 .

Images