A real-time instance segmentation method for urban street scenes based on lightweight attention

Abstract

The present invention discloses a real-time instance segmentation method for urban street scenes based on lightweight attention, comprising the following steps: Step 1: Reading an urban street scene dataset and performing preprocessing and data enhancement; Step 2: Using the image processed in Step 1 as input, extracting features through a backbone network and outputting a feature map; Step 3: Inputting the feature map into a focused linear attention module with a focusing function to further extract features so that they are focused on the target instance; Step 4: Fusing the feature map extracted by the focused attention module with feature maps of different sizes extracted by the backbone network; Step 5: Based on a decoupling head, obtaining a detection branch and a mask branch, inputting the fused features into the detection branch and the mask branch respectively to obtain a detection result and a segmentation mask, and finally synthesizing the detection result and the segmentation mask to obtain the instance segmentation result of the target. Compared with traditional pure convolutional neural networks and Transformer models, it effectively balances performance and speed.

Description

Technical Field

[0001] The present invention relates to the technical field of image instance segmentation, and in particular to a real-time instance segmentation method for urban street scenes based on lightweight attention. Background Art

[0002] Image segmentation technology is crucial to the development of autonomous driving and smart cities. By integrating advanced perception, decision-making, and execution technologies, autonomous driving systems enable vehicles to adapt to complex traffic environments in real time and accurately identify and separate various urban objects (such as vehicles and pedestrians). This technology not only enhances the environmental perception capabilities of autonomous vehicles but also plays a vital role in intelligent monitoring and traffic management in smart cities. By identifying objects such as pedestrians and vehicles, it improves monitoring accuracy and response speed, optimizes traffic flow management, and reduces congestion. However, existing urban street scene instance segmentation technologies face numerous challenges. For example, the complexity of urban environments and the computing performance of devices all place higher demands on the accuracy, efficiency, and robustness of algorithms.

[0003] Traditional instance segmentation typically uses a two-stage approach: first detecting the bounding box of an object, then performing pixel-wise segmentation within the bounding box. While models like Mask R-CNN achieve high accuracy, they are slow. To improve real-time performance, some work has reduced the number of network layers and parameters, but this can significantly reduce segmentation accuracy. In recent years, the success of the Transformer architecture in vision tasks has introduced attention mechanisms to instance segmentation. This is because the Transformer relies on a self-attention mechanism, which requires pairwise comparisons of all elements in the input sequence. In images, this means that each pixel must interact with all other pixels in the image, resulting in a significant increase in computational overhead. Some methods replace convolutional operations with attention mechanisms for instance segmentation, using a learnable query to directly predict the object category and mask in a single stage. For example, Mask2Former relies on complex decoder layers and a pixel decoder, and its mask attention can limit the receptive field of the query, affecting efficiency. While Mask2Former demonstrates promising performance, it requires significant computational resources for instance segmentation and is not yet suitable for real-time instance segmentation applications in urban street scenes.

[0004] Therefore, in urban street scene scenarios such as autonomous driving systems and smart cities, the computing resources of most devices are limited, and more efficient model design is needed to ensure real-time instance segmentation of images in embedded or resource-constrained environments. Summary of the Invention

[0005] This paper addresses the challenges of traditional convolutional neural networks and Transformer architecture models in real-time instance segmentation in urban street scenes, which face low accuracy and inability to meet real-time requirements. This paper proposes a lightweight attention-based real-time instance segmentation method for urban street scenes. Compared with traditional pure convolutional neural networks and pure Transformer models, it effectively balances performance and speed. This method integrates a focused linear attention module into the traditional convolutional neural network. This module includes linear attention, which improves computational efficiency while maintaining the core functionality of the attention mechanism. This module adjusts the feature directions of the query and key through a mapping function, making the attention weight more pronounced, thereby enhancing the model's ability to focus on the instance segmentation target.

[0006] In order to solve the above technical problems, the technical solution of the present invention is:

[0007] A real-time instance segmentation method for urban street scenes based on lightweight attention includes the following steps:

[0008] Step 1: Read the urban street view dataset and perform preprocessing and data enhancement;

[0009] Step 2: Take the image processed in step 1 as input, perform feature extraction through the backbone network and output a feature map. The backbone network includes a 3×3 convolution kernel and a C2f module.

[0010] Step 3: The feature map is input into a focused linear attention module with a focusing function to further extract features and focus them on the target instance;

[0011] Step 4: Fusing the feature map extracted by the focus attention module with the feature maps of different sizes extracted by the backbone network;

[0012] Step 5: Based on the decoupling head, the detection branch and the mask branch are obtained. The fused features are input into the detection branch and the mask branch respectively to obtain the detection result and the segmentation mask. Finally, the detection result and the segmentation mask are synthesized to obtain the instance segmentation result of the target.

[0013] Preferably, the preprocessing method includes performing a letterbox scaling operation on the urban street view dataset and unifying the image size to 640×640.

[0014] In the above technical solution, the long side of the image is first scaled to 640, and then the short side is scaled proportionally. If it cannot be scaled, the excess part is rounded up to fill it. This operation maintains the aspect ratio of the original image and ensures that the image is not distorted.

[0015] Preferably, the data enhancement method includes HSV enhancement, MixUp enhancement, and flipping. In the above technical solution, HSV enhancement is used to enhance the visual features of the image, and MixUp enhancement and flipping are used to enhance the diversity of the data and improve the generalization ability of the model.

[0016] Preferably, the backbone network mainly uses convolutional layers and C2f modules as basic building blocks.

[0017] The C2f module makes the network more lightweight while maintaining the concept of Cross-Stage Partial networks. This module divides the feature extraction process into two stages: compression and expansion. In the compression stage, convolution is used to reduce the dimensionality of the input feature map to reduce computational complexity. In the expansion stage, convolution and upsampling are used to restore the feature map to its original size and perform feature fusion. This design ensures the effectiveness of feature extraction, reduces computational complexity, and obtains richer gradient flow information.

[0018] As an advantage, the focusing function is to introduce a mapping function f for focusing capability. p , the expression is as follows:

[0019] φ p (x) = f p (ReLU(x))

[0020]

[0021] Among them, x **p Represents x element-wise power p and uses the ReLU function to ensure the non-negativity of the input and the validity of the denominator.

[0022] In the above technical solution, the mapping function simulates the original Softmax attention to focus on information-rich areas: by adjusting the direction of each query and key feature, similar query-key pairs are brought closer and dissimilar query-key pairs are pulled apart.

[0023] The Focused Linear Attention Module approach first incorporates a novel mapping function to mimic the original Softmax attention's focus on information-rich regions. Furthermore, it addresses the low-rank dilemma of conventional linear attention modules and employs deep convolution to restore feature diversity. In this way, the Focused Linear Attention Module achieves both linear complexity and high feature representation capabilities.

[0024] Preferably, the attention matrix of the focused linear attention module is integrated with a deep convolution module, and the final focused linear attention module can be expressed as:

[0025]

[0026] Where Sim(Q,K) is the similarity function, φ p is the focusing function, DWC is the depth convolution module, Q, K, Refers to query, key, and value respectively.

[0027] In the above technical solution, the DWC module can be regarded as a kind of attention, where each query only focuses on a few adjacent features in the space instead of all features. This locality ensures that even if the linear attention values ​​corresponding to two queries are the same, different outputs can still be obtained from different local features, thus maintaining feature diversity. By adding the DWC module, the rank of the attention map in the linear attention can be restored to full rank, maintaining the feature diversity of the original Softmax attention.

[0028] Preferably, in step 4, feature fusion is achieved through a feature pyramid network and a path aggregation network.

[0029] In the aforementioned technical solution, FPN passes strong semantic features from the top (small size, high number of convolutions, rich semantic information) to the bottom (large size, low number of convolutions), enhancing the entire pyramid. PAN adds a bottom-up pyramid after FPN. By introducing path aggregation, it aggregates shallow feature maps (low resolution but weak semantic information) and deep feature maps (high resolution but rich semantic information), and transmits feature information along a specific path, passing strong positioning features from the lower layers upward. This can further enhance the expressive power of multi-scale features.

[0030] Preferably, the detection result: takes the fused features corresponding to the target instance as input, and obtains the category, border information and k mask coefficients through the detection branch.

[0031] Preferably, the segmentation mask is: taking the fused features corresponding to the target instance as input, and generating k mask prototype images through mask branches.

[0032] Preferably, in step 5, the method for synthesizing the instance segmentation result of the target is: multiplying k mask coefficients with k mask prototype images, adding the multiplication result to the category and border information, and then connecting to the Sigmoid nonlinear function to generate the final mask, that is, the instance segmentation result of the target.

[0033] The present invention has the following characteristics and beneficial effects:

[0034] Using the aforementioned technical solution, this method introduces a focused linear attention mechanism for instance segmentation, reducing the computational complexity of the attention mechanism while maintaining representational capabilities. Tests on the Cityscapes dataset demonstrate that this method effectively balances performance and speed compared to pure convolutional and pure attention approaches. This facilitates deployment in real-world applications on edge devices, such as autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 This is a flowchart of the real-time instance segmentation method of urban street scenes based on lightweight attention in the present invention.

[0037] Figure 2 This is a specific network framework diagram of the real-time instance segmentation method of the present invention.

[0038] Figure 3 This is a data label distribution diagram of the data set used in the method of the present invention.

[0039] Figure 4 This is an example segmentation effect diagram of the method of the present invention in a complex road traffic scene. DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0041] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings.

[0042] On the contrary, the present invention covers any alternatives, modifications, equivalents, and solutions that fall within the spirit and scope of the present invention as defined by the claims. Furthermore, to facilitate a better understanding of the present invention, certain specific details are described in detail below in the detailed description of the present invention. Those skilled in the art will be able to fully understand the present invention without these details.

[0043] This embodiment provides a real-time instance segmentation method for urban street scenes based on lightweight attention. Figure 1 As shown, the following steps are included:

[0044] Step S1: Obtain a city street view image dataset and preprocess and enhance the images in the dataset.

[0045] Specifically, the step S1 includes steps S1-1 to S1-4:

[0046] S1-1. Load the corresponding annotations in the image dataset, including category labels and instance labels.

[0047] S1-2. Perform an adaptive scaling operation on the image loaded in the image dataset. First, determine that the target size of the image processing is 640×640, and calculate the scaling ratio based on the original image and the target size, and take the minimum value of the aspect ratio for scaling. During the calculation process, a scaling limit is considered to prevent the image from being enlarged. Subsequently, the amount of padding required to reach the target size is calculated, and padding is performed around the image to avoid loss of image content. The padding operation uses the (114,114,114) color value to ensure the consistency of the filled area. The segmentation mask label will then be updated based on the adjusted image to align it with the scaled and padded image, thereby ensuring the accuracy of subsequent processing steps.

[0048] S1-3, performing MixUp image enhancement processing on the image loaded in the image dataset. The specific method is as follows:

[0049] First, a mixing coefficient λ between 0 and 1 is generated. Then, the pixel values ​​and labels of the two images are linearly combined according to this coefficient. Specifically, it creates a new image whose pixel values ​​are the weighted average of the pixel values ​​of the original image and the randomly selected image, with the weights determined by λ. At the same time, it also merges the label information of the two images. It can be expressed as:

[0050] img mix =img1*λ+img2*(1-λ) (1)

[0051] Where img1 and img2 are samples to be mixed, and the mixing coefficient λ is sampled from the Beta distribution and is obtained based on the following formula:

[0052] λ=Beta(α,β) (2)

[0053] The value of α,β is limited to maintain the expectation Always approximately 0.5.

[0054] S1-4. Perform HSV (hue, saturation, brightness value) enhancement and adjustment on the image loaded into the image dataset to enhance the visual features of the image.

[0055] It should be noted that HSV enhancement enhances the visual characteristics of an image by adjusting its hue, saturation, and brightness. The HSV color space is enhanced by randomly generating three multiplication factors and then multiplying them by the Hue, Saturation, and Value values. The HSV color space is constructed from the perspective of human perception and, compared to the RGB color space, is closer to how humans perceive color, thus enriching the characteristics of urban street scene images and enhancing the robustness of the model.

[0056] S1-5. Randomly flip the images of the urban street scene dataset horizontally and adjust the corresponding labels.

[0057] In step S2, the preprocessed image is fed into the backbone network for feature extraction. The backbone network primarily consists of 3×3 convolutional kernels and C2f modules. The convolutional modules and residual blocks are stacked serially multiple times, doubling the number of channels after each pooling operation. The C2f module makes the network more lightweight while maintaining the concept of cross-stage partial networks.

[0058] The C2f module divides the feature extraction process into two stages: compression and expansion. In the compression stage, convolution is used to reduce the dimensionality of the input feature map to reduce computational complexity. In the expansion stage, convolution and upsampling are used to restore the feature map to its original size and perform feature fusion. This design ensures effective feature extraction, reduces computational complexity, and obtains richer gradient flow information.

[0059] It's important to note that the residual block draws on the principles of the Residual Network (ResNet), which makes extensive use of residual connections. This allows for a very deep network structure. Traditional deep neural networks attempt to learn the target function H(x), but in a residual network, each layer actually learns a residual function F(x) = H(x) - x. This residual is then added to the input x to form H(x) = F(x) + x. This mechanism makes it easier for the network to learn the identity mapping, thereby alleviating the vanishing gradient problem.

[0060] Step S3: Input the feature map output by the backbone network into the focused linear attention module to further extract features and focus on the target instance.

[0061] Specifically, step S3 includes steps S3-1 to S3-8.

[0062] S3-1. Downsample the high-level feature map extracted by the backbone network and input it into the focused linear attention module.

[0063] S3-2. Process the input feature map. The original input feature map is divided into blocks of the same window size. In this method, the window is set to 20×20 and position encoding is added.

[0064] S3-3. Map the split blocks into vector representations: For each resulting block, a linear transformation is performed to map it into a one-dimensional feature vector. The feature map dimensions are flattened into a sequence, and then a transposition operation is performed to move the sequence dimension to the front. A linear layer is then used to map it into three parts: query, key, and value.

[0065] S3-4. Scale and normalize the query and key. Use the Softplus function to calculate a scaling factor. The Softplus function is a smooth nonlinear function that maps the input to a value greater than 0, similar in form to ln(1+e x ).

[0066] It's important to note that the scaling factor is a learnable parameter whose value is constantly adjusted during training to stabilize the self-attention computation and avoid vanishing or exploding gradients. Each element of the query and key is then divided by the scaling factor. This is intended to adjust the query and key values ​​to a suitable range so that subsequent dot product operations are not affected by numerical issues.

[0067] S3-5. Use a mapping function f p By adjusting the feature direction of each query and key, similar query-key pairs are brought closer together and the distance between dissimilar query-key pairs is increased. It can be expressed as:

[0068]

[0069] where x **p Represents the power p of x element-wise. The ReLU function is used to ensure the non-negativity of the input and the validity of the denominator.

[0070] S3-6. Calculate the self-attention weight. The Einstein summation convention (einsum) is used to perform matrix operations, calculate the dot product of the query and the key, and then apply the Softmax function to obtain the attention weight.

[0071] It should be noted that Einstein summation is a symbolic system for expressing vector and tensor operations. In the string expression of einsum, each letter represents a dimension, and the operator defines the relationship between these dimensions.

[0072] S3-7, Feature Map and Residual Connection: The values ​​are rearranged into feature maps, which are then processed through a deep convolutional (DWC) module to increase the representational power of the model and extract richer features, thereby addressing the limitations of linear attention. For each input channel, the output feature map O(c) of the DWC module is calculated as follows:

[0073]

[0074] Where I[c] is the c-th channel of the input feature map, O(c) is the c-th channel of the output feature map, and K(c) is the convolution kernel corresponding to channel c.

[0075] Furthermore, the attention matrix of the focused linear attention module is integrated with a deep convolution module. Based on the above focused linear attention module method, it can be expressed as:

[0076]

[0077] S3-8. Residual Connection and Output Processing. The processed feature map is residually connected to the original query to avoid vanishing or exploding gradients in deep networks. A projection layer then transforms the dimensions of the output features to match the input requirements of subsequent layers. Dropout is also applied to reduce overfitting.

[0078] Step S4: The features extracted by the focused linear attention module are fused with the features extracted by the C2f module in the backbone network through upsampling, and multi-level upsampling and downsampling are performed. This is achieved through the Feature Pyramid Network (FPN) and the Path Aggregation Network (PAN). FPN is top-down, passing down the strong semantic features of the high-level layers to enhance the entire pyramid. PAN is to add a bottom-up pyramid after FPN. By introducing the path aggregation method, the shallow feature map and the deep feature map are aggregated, and the feature information is passed along a specific path, and the strong positioning features of the low layer are passed up, which can further enhance the expression ability of multi-scale features.

[0079] Specifically, step S4 includes steps S4-1 to S4-8:

[0080] S4-1, upsample the feature map extracted by the focus attention module by a factor of two, and increase the resolution of the feature map by a factor of two;

[0081] S4-2, connect the feature map obtained in step S4-1 with the third-level feature map of the backbone network in step S2 to obtain a new feature map;

[0082] S4-3, the feature map obtained in S4-2 is processed by the C2f module and upsampled twice to double the resolution of the feature map;

[0083] S4-4, connecting the feature map obtained in step S4-3 with the second-level feature map of the backbone network in step S2 to obtain a new feature map;

[0084] S4-5. Process the feature map obtained in step S4-4 using the C2f module and perform two-fold downsampling through a 3×3 convolutional layer to reduce the resolution of the feature map by half.

[0085] S4-6, connecting the feature map obtained in step S4-5 with the feature map obtained in step S4-2 to obtain a new feature map;

[0086] S4-7, process the feature map obtained in S4-6 using the C2f module and perform two-fold downsampling through a 3×3 convolutional layer to reduce the resolution of the feature map by half;

[0087] S4-8. Connect the feature map extracted by the focus attention module with the feature map obtained in S4-7 to obtain a new feature map.

[0088] S5. Based on the decoupling head, the detection branch and the mask branch are obtained, and then the detection result and the segmentation mask are synthesized to obtain the instance segmentation result of the target. Specifically, for the detection branch, for each target object, k mask coefficients (Mask Coefficients) of category and bounding box information are output; for the mask branch, k mask prototypes (Mask Prototype) are output for the current input image. For each target object, the k mask coefficients are multiplied by the k mask prototypes, and then all the results are combined to obtain the instance segmentation result of the target object. It should be noted that k here is set to 32.

[0089] Specifically, such as Figure 2 As shown, step S5 includes steps S5-1 to S5-2:

[0090] Step S5-1: The feature map obtained by S4-4 is input to the segmentation head, 32 mask prototype maps are obtained through the mask branch, and 32 mask coefficients, the object detection box of the instance and the instance category index are obtained through the detection branch;

[0091] Step S5-2: The feature map obtained by S4-6 is input to the segmentation head, and 32 mask coefficients, the target detection box of the instance and the instance category index are obtained through the detection branch;

[0092] Step S5-3: The feature map obtained by S4-8 is processed by the C2f module and input to the segmentation head, and 32 mask coefficients, the target detection box of the instance and the instance category index are obtained through the detection branch;

[0093] Step S5-4: Combine the 32 mask coefficients obtained by the detection head in S5-3 with the 32 mask prototype images obtained by the segmentation head to obtain a mask for a specific instance. Specifically, the 32 mask coefficients and the 32 mask prototype images are linearly combined and multiplied to obtain the optimal mask associated with the detection box;

[0094] Step S5-5: Post-process the obtained mask by first performing confidence testing and non-maximum suppression to determine the valid target, and then perform mask processing.

[0095] Specifically, the system first checks whether the set confidence and IoU thresholds are between 0 and 1. If not, an error is reported indicating that the thresholds do not meet the requirements. Results that meet the confidence score are then selected. The results are sorted by score, and the highest-scoring portion is captured based on the maximum number of non-maximum suppressions set. The original image shape and dimensions are then obtained, and the resulting category index is mapped to the text category. The box, category, and mask are then mapped back to the original image scale. The final instance segmentation mask label or visualization image is saved as needed.

[0096] In order to verify the effect of the present invention, the following experiments were performed:

[0097] This paper uses the Cityscape dataset, which contains high-quality images from city streets, covering scenes of different weather, time of day and seasons. The dataset consists of 5000 images designated for training purposes and another 500 images assigned to the validation phase. The size of the image is 1024×2048 pixels. Each image is provided with dense pixel-level annotations covering 19 semantic classes such as roads, buildings, vehicles, etc. 8 of these classes have instance-level segmentation. Figure 3 The distribution of instance labels is shown. This shows that this dataset is suitable for evaluating image instance segmentation performance in autonomous driving and smart city traffic environments.

[0098] Experimental environment and parameters

[0099] The experimental configuration of the present invention is as follows: the operating system is Ubuntu 20.04.6, the processor is Intel Core TM An i9-13900K CPU with a main frequency of 3.0GHz and an NVIDIA RTX A6000 graphics card with 48GB of video memory was used. The deep learning framework Pytorch version 2.0.0 and Python version 3.9 were used to train and verify the invented method.

[0100] In this embodiment, the performance of the present invention is evaluated on the validation set of the Ciyscapes dataset. The experiment uses mAP, AP50, Para(M), FPS, and Runtime(ms) to evaluate the accuracy and performance of the method of the present invention.

[0101] It should be noted that Intersection over Union (IoU) is a metric that measures the degree of overlap between the model's predicted value and the true value. For object detection tasks, IoU is used to evaluate the overlap between the detection box and the true box; for image segmentation tasks, IoU is used to calculate the overlap between the predicted mask and the true mask. Its calculation formula is the ratio of the intersection area to the union area, as shown below:

[0102]

[0103] Among them, TP (True Positive) represents the number of correctly predicted positive samples, FP (False Positive) represents the number of incorrectly predicted positive samples, and FN (False Negative) represents the number of positive samples that cannot be predicted.

[0104] mAP is the mean average precision, and the calculation formula is as follows:

[0105]

[0106] Among them, mAP stands for average precision, which is the average AP value of all categories in a multi-category task. Here, C represents the number of categories. AP50 refers to the average precision when the IoU threshold is 0.5. The calculation method is similar to mAP, but it is only calculated when IoU = 0.5. It is defined as follows:

[0107] AP50=AP IoU＝0.5 (8)

[0108] Para(M) refers to the number of parameters contained in the model, which is an important indicator affecting the model's memory requirements and computational load. The expression for the parameter quantity is:

[0109]

[0110] Among them, L is the number of layers of the model, P i is the number of parameters in layer i. A model with fewer parameters is generally more efficient because it requires less memory and can be deployed on resource-constrained devices such as edge devices and mobile platforms. However, it is important to strike a balance between the number of parameters and model accuracy to ensure that performance is not compromised.

[0111] FPS refers to frames per second, an important indicator for measuring the real-time performance of model image inference. FPS is defined as follows:

[0112]

[0113] Where N is the number of frames processed and T is the total time required to process these frames. A higher FPS indicates that the model is able to process more frames in real time.

[0114] Runtime (ms) refers to the inference speed, which measures the time required for the model to process a single image or frame. It is measured in milliseconds (ms) per frame and is defined as follows:

[0115]

[0116] Where T is the total time required to process N frames. Fast inference speed is crucial for real-time applications because it ensures the model can respond promptly. This metric is affected by the model architecture and the hardware on which it is deployed. Optimizing inference speed often involves a trade-off between model complexity and computational efficiency.

[0117] The comparison of the verification results of the method of the present invention and the existing technology on the Cityscapes dataset is shown in Table 1.

[0118] Table 1. Verification results of different methods on the Cityscapes dataset

[0119]

[0120] The comparison of the verification results of the proposed method and the existing technology on the Cityscapes dataset is shown in Table 1. The proposed method has only 3.46M model parameters, while the mAP and AP50 reach 18.7 and 37.8 respectively, the FPS reaches 357, and the inference speed reaches 2.8ms. Figure 4 As shown in Figure 2, the invention surpasses the performance of most advanced real-time instance segmentation methods and achieves a good balance between accuracy and speed with a small number of parameters.

[0121] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A real-time instance segmentation method for urban street scenes based on lightweight attention, characterized by: The steps include: Step 1: Read the urban street view dataset and perform preprocessing and data enhancement; Step 2: Take the image processed in step 1 as input, perform feature extraction through the backbone network and output a feature map. The backbone network includes a 3×3 convolution kernel and a C2f module. Step 3: The feature map is input into a focused linear attention module with a focusing function to further extract features and focus them on the target instance; The focusing function is to introduce a mapping function f for focusing capability. p , the expression is as follows: Among them, x **p Represents the x element power p and uses the ReLU function to ensure the non-negativity of the input and the validity of the denominator; Step 4: Fusing the feature maps extracted by the focused linear attention module with the feature maps of different sizes extracted by the backbone network; The attention matrix of the focused linear attention module is integrated with the deep convolution module. The final focused linear attention module can be expressed as: Where Sim(Q, K) is the similarity function, φ p is the focusing function, DWC is the depth convolution module, Q, K, Refers to query, key and value respectively; Step 5: Based on the decoupling head, the detection branch and the mask branch are obtained. The fused features are input into the detection branch and the mask branch respectively to obtain the detection result and the segmentation mask. Finally, the detection result and the segmentation mask are combined to obtain the instance segmentation result of the target. The method to synthesize the instance segmentation result of the target is: multiply k mask coefficients with k mask prototype images, add the multiplication result with the category and border information, and then connect it to the Sigmoid nonlinear function to generate the final mask, which is the instance segmentation result of the target.

2. The method for real-time instance segmentation of urban street scenes based on lightweight attention according to claim 1, characterized in that: The preprocessing method includes performing a letterbox scaling operation on the urban street view dataset and unifying the image size to 640×640.

3. The method for real-time instance segmentation of urban street scenes based on lightweight attention according to claim 1, characterized in that: The data enhancement methods include HSV enhancement, MixUp enhancement, and flipping.

4. The method for real-time instance segmentation of urban street scenes based on lightweight attention according to claim 1, characterized in that: In step 3, the feature map input to the focused linear attention module needs to be downsampled first.

5. The method for real-time instance segmentation of urban street scenes based on lightweight attention according to claim 1, characterized in that: In step 4, feature fusion is achieved through a feature pyramid network and a path aggregation network.

6. The method for real-time instance segmentation of urban street scenes based on lightweight attention according to claim 1, characterized in that: The detection result: takes the fused features corresponding to the target instance as input, and obtains the category, border information and k mask coefficients through the detection branch.

7. The method for real-time instance segmentation of urban street scenes based on lightweight attention according to claim 6, characterized in that: The segmentation mask is: taking the fused features corresponding to the target instance as input, and generating k mask prototype images through the mask branch.

Images