Dense scene understanding method and system based on lightweight multi-task attention network

Abstract

The invention discloses a dense scene understanding method and system based on a lightweight multi-task attention network, and relates to the field of computer vision. The method comprises the following steps: constructing a lightweight multi-task attention network LW-MLTANet model; establishing a scene image data set for intensive scene understanding; the scene image data set understood by the dense scene comprises a plurality of images with image semantic segmentation annotation information, depth information and instance segmentation annotation information; the scene image data set is utilized to train the LW-MLTANet model, and an optimal dense scene understanding LW-MLTANet model is obtained; and inputting an image to be tested into the optimal LW-MLTANet model to obtain three task prediction results of semantic segmentation, depth estimation and instance segmentation. According to the method, specific task features are learned in global features, overuse of a memory is avoided, calculation loads are reduced, and meanwhile the model size and calculation cost are reduced.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method and system for understanding dense scenes based on a lightweight multi-task attention network. Background Technology

[0002] Deep neural networks have achieved remarkable results in many intensive prediction tasks, i.e., tasks that produce pixel-level predictions, such as semantic segmentation, instance segmentation, and depth estimation. Traditionally, these tasks were handled separately, i.e., a separate neural network was trained for each task.

[0003] Dense scene understanding algorithms must simultaneously understand the geometry and semantics of a scene, which can be achieved by using scene depth estimation and semantic segmentation tasks, which are closely related. This connection allows researchers to construct multi-task networks to perform both tasks concurrently, thus forming an interesting multi-task learning problem, as dense scene understanding involves the joint learning of various regression and classification tasks at different units and scales.

[0004] Multi-task learning aims to improve learning efficiency and prediction accuracy by learning multiple objectives from shared representations. Compared to single-task learning, multi-task learning reduces the time cost of the training phase while improving generalization performance and data efficiency. Therefore, multi-task learning for visual scene understanding is crucial in systems with excessively long computation times.

[0005] In the field of multi-task intensive prediction, most work is based on convolutional neural networks. Convolutional models capture spatial and context-dependent tasks within a relatively constrained perceptual field. In recent years, attention mechanisms have been widely applied in computer vision. They can help networks focus on key regions by mimicking human attention mechanisms. However, these works mainly focus on fusing different forms of information to improve task prediction accuracy without considering inference speed, limiting the application of models in real-time scenarios. Meanwhile, due to computational resource constraints, the application of multi-task learning models in real-world scenarios remains a challenge. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for understanding dense scenes based on a lightweight multi-task attention network. The model parameters are lightweight, inference can be performed in real time, and the prediction accuracy of each task is improved.

[0007] To achieve the above objectives, the present invention provides the following solution: A method for understanding dense scenes based on a lightweight multi-task attention network, the method comprising: A lightweight multi-task attention network model, LW-MLTANet, is constructed, comprising an input layer, an encoder, an efficient Transformer, a decoder, and a task-specific output layer. The encoder uses a pre-trained MobileNet-v2 network for image feature extraction. The decoder is a lightweight attention network based on the RefineNet architecture, LWA-RefineNet, which includes six lightweight attention network modules (LW-Attention), three lightweight chained residual pooling attention modules (LW-CRPA), and one lightweight chained residual pooling module (LW-CRP1). The LW-Attention module enhances the network's feature representation capabilities while reducing the number of channels. A scene image dataset for dense scene understanding is established, comprising multiple images used for semantic segmentation, depth estimation, and instance segmentation, along with their corresponding labels; the scene image dataset is then divided into a training set and a test set. The LW-MLTANet model is trained using the scene image dataset for dense scene understanding to obtain the optimal LW-MLTANet model for dense scene understanding. The images from the test set are input into the optimal dense scene understanding LW-MLTANet model to obtain prediction results for three tasks: semantic segmentation, depth estimation, and instance segmentation.

[0008] Furthermore, the specific task output layer includes three tasks: semantic segmentation, depth estimation, and instance segmentation. Each task passes the decoder output through a depth-wise pixel attention layer, a ReLU6 layer, and a 3×3 Conv2d layer, ultimately outputting the prediction results of the three tasks: semantic segmentation, depth estimation, and instance segmentation.

[0009] Furthermore, the efficient Transformer is used to connect the encoder and decoder, including: two LayerNorms, an efficient multi-head attention EMHA, and a multilayer perceptron MLP containing two fully connected layers; The efficient multi-head attention EMHA builds a reduction layer after Layer Norm to halve the number of channels. Then, the linear layer projects the feature map onto three matrices: Q, K, and V, and then uses feature segmentation to divide Q, K, and V into several equal segments. Each triple of these segments is then subjected to a scaled dot product attention operation.

[0010] Furthermore, training the LW-MLTANet model using the scene image dataset for dense scene understanding specifically includes: The scene image dataset is divided into a training set and a test set according to a preset ratio; The LW-MLTANet model is trained using the training set to obtain the optimal LW-MLTANet model for dense scene understanding.

[0011] Furthermore, the LW-MLTANet model is trained using a multi-task loss function, including: Multi-task loss function for: = The semantic segmentation loss function is the standard softmax cross-entropy loss: in, , , is a scale factor and It is a one-hot vector of the real label. It is the first The actual output value of the class node.

[0012] On the other hand, this invention protects a dense scene understanding system based on a lightweight multi-task attention network, which executes the aforementioned dense scene understanding method based on a lightweight multi-task attention network, including: The LW-MLTANet module is used to build a lightweight multi-task attention network LW-MLTANet model, which includes an input layer, encoder, efficient Transformer module, decoder, and task-specific output layer. The scene image dataset module is used to establish a scene image dataset, which includes multiple images used for semantic segmentation, depth estimation, and instance segmentation, along with their corresponding labels; the scene image dataset is divided into a training set and a test set. The model training module is used to train the LW-MLTANet model using the training set to obtain the optimal dense scene understanding LW-MLTANet model. The output module is used to input the segmented test set images into the optimal dense scene understanding LW-MLTANet model and output the results of the dense scene understanding task, which includes three tasks: semantic segmentation, depth estimation, and instance segmentation.

[0013] Furthermore, the efficient Transformer module is used to connect the encoder and decoder, and includes: two LayerNorms, an efficient multi-head attention EMHA module, and a multilayer perceptron (MLP) containing two fully connected layers; Two Layer Norm structures are used in Transformer to prevent gradient vanishing or gradient exploding problems; The EMHA module includes: a Reduction layer that halves the number of channels; a linear layer that projects the feature map onto three matrices: Q, K, and V; and a feature segmentation module that segments the Q, K, and V into several equal segments, performs a scaled dot product attention operation on the triples of the segments, and then uses a Concat layer and an Expansion layer to restore the number of channels.

[0014] Furthermore, the decoder is a lightweight attention network LWA-RefineNet, which includes a lightweight attention network module LW-Attention, a lightweight chained residual pooling attention module LW-CRPA, and a lightweight chained residual pooling module LW-CRP1.

[0015] Furthermore, the lightweight attention network module LW-Attention is used to enhance the feature representation capability of the network and reduce the dimensionality of the number of channels, including: a Conv2d layer, a BatchNorm2d layer, a ReLU, a Conv2d layer, a BatchNorm2d layer and a Sigmoid() activation layer connected in sequence. The lightweight chained residual pooling attention module LW-CRPA includes: an LW-CRP1 module, a pixel attention PA module, and an Upsample operation connected in sequence. The lightweight chained residual pooling module LW-CRP1 is used to capture background context information of large image regions by combining chained pooling layers and convolutional residuals, while reducing the number of channels.

[0016] Furthermore, the PA module includes: a 1×1 Conv2d layer, a BatchNorm2d layer, and a Sigmoid() activation layer connected in sequence; The specific task output layer includes three headers: semantic segmentation, depth estimation, and instance segmentation. Each task passes the decoder output through a depth-wise pixel attention layer, a ReLU6 layer, and a 3×3 Conv2d layer, ultimately outputting the prediction results for the three tasks: semantic segmentation, depth estimation, and instance segmentation.

[0017] Optionally, during the training of the LW-MLTANet model, the training set is augmented with data; the data augmentation includes random mirroring, image resizing, and image normalization.

[0018] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: This invention discloses a dense scene understanding method based on a lightweight multi-task attention network (LW-MLTANet), comprising an input layer, an encoder, an efficient Transformer, a decoder, and a task-specific output layer. Specifically, the encoder uses a pre-trained MobileNet-v2 network, which offers fast inference speed and powerful feature extraction capabilities. The efficient Transformer consists of an efficient multi-head attention network (EMHA) and an MLP, learning task-specific features from global features while avoiding excessive memory usage and computational load. The decoder is a lightweight attention network (LWA-RefineNet) based on the RefineNet architecture to reduce the number of parameters and floating-point operations. The encoder, efficient Transformer, and decoder form a shared network to receive input data and learn task-shared features. The shared network branches to the heads of the three tasks, forming the task-specific output module. The task-specific output module includes three tasks: semantic segmentation, depth estimation, and instance segmentation. Each task passes the decoder output through a depth-wise pixel attention layer, a ReLU6 layer, and a 3×3 Conv2d layer, ultimately outputting the prediction results for semantic segmentation, depth estimation, and instance segmentation. After training the LW-MLTANet model, the optimal dense scene understanding model can perform real-time predictions for three tasks: semantic segmentation, depth estimation, and instance segmentation, thereby improving the model's accuracy. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of a dense scene understanding method based on a lightweight multi-task attention network provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the LW-MLTANet model structure provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the high-efficiency Transformer module provided in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the high-efficiency multi-head attention EMHA module provided in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram of the network structure of the LW-Attention module provided in Embodiment 1 of the present invention; Figure 6This is a diagram showing the results of understanding dense scenes as provided in Embodiment 1 of the present invention; Figure 7 This is a schematic diagram of a dense scene understanding system based on a lightweight multi-task attention network provided in Embodiment 2 of the present invention; Figure 8 This is a schematic diagram of the structure of the LW-MLTANet module provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Example 1 like Figure 1 As shown in this embodiment, a dense scene understanding method based on a lightweight multi-task attention network includes the following steps: Step 1: Construct a lightweight multi-task attention network, LW-MLTANet, comprising an input layer, an encoder, an efficient Transformer, a decoder, and a task-specific output layer. Specifically, the encoder uses a pre-trained MobileNet-v2 network, the efficient Transformer consists of an efficient multi-head attention network (EMHA) and an MLP, and the decoder is a lightweight attention network, LWA-RefineNet, based on the RefineNet architecture. The encoder, efficient Transformer, and decoder form a shared network to receive input data and learn task-shared features. The shared network branches to the heads of the three tasks, forming the task-specific output module. The task-specific output module includes three tasks: semantic segmentation, depth estimation, and instance segmentation. For each task, the decoder output is passed through a depth-wise pixel attention layer, a ReLU6 layer, and a 3×3 Conv2d layer, ultimately outputting the prediction results for semantic segmentation, depth estimation, and instance segmentation. like Figure 2 As shown, the LW-MLTANet model specifically includes: The input image is a 224×224×3 image used for dense scene understanding. Feature extraction is performed using a MobileNet-v2 network pre-trained with an encoder. The feature map output from MobileNet-v2 is then fed into an efficient Transformer for further feature extraction.

[0024] like Figure 3 As shown, the high-efficiency Transformer module is used to connect the encoder and decoder, and includes: two LayerNorms, a high-efficiency multi-head attention EMHA, and an MLP; Two Layer Norm structures are used in Transformer to prevent gradient vanishing or gradient exploding problems and improve model stability.

[0025] Highly efficient multi-head attention EMHA follows Layer Norm, such as Figure 4 As shown, a reduction layer is established to halve the number of channels, thereby reducing some computation. Then, the linear layer projects the feature map onto three matrices: Q (Query) K (key) and V (Value), then use the Feature Split module to... Q , K and V The segment is divided into s equal segments. Each triple of these segments is then subjected to Scaled Dot-Product Attention (SDPA), as shown in the formula: Where d is the embedding dimension, It is the first A query matrix, It is the first Transpose of the key matrix It is the first A matrix of values; all outputs of SDPA ( …, The entire output feature is generated by concatenating these features through a Concat layer. These operations decompose a large matrix into smaller matrices, then compute them, and finally merge them, thereby reducing the computational load. Finally, an Expansion layer is used to restore the number of channels.

[0026] The decoder is a lightweight attention network LWA-RefineNet, which includes 6 lightweight attention network modules LW-Attention, 3 lightweight chained residual pooling attention modules LW-CRPA, and 1 lightweight chained residual pooling module LW-CRP1. The LW-Attention module is used to enhance the feature representation ability of the network and reduce the dimensionality of the number of channels, thereby reducing the amount of computation and the number of parameters and improving computational efficiency. like Figure 5 As shown, the LW-Attention module includes: sequentially connected Conv2d layer, BatchNorm2d layer, ReLU, Conv2d layer, BatchNorm2d layer and Sigmoid() activation layer; The LW-CRPA module includes: an LW-CRP1 module, a pixel attention PA module, and an Upsample operation connected in sequence; The LW-CRP1 module captures background context information of large image regions by combining chained pooling layers and convolutional residuals, while reducing the number of channels to reduce computational cost and parameter count, thereby improving computational efficiency. The LW-CRP1 module includes: a ReLU6 and an LW-CRP module connected in sequence; The LW-CRP module includes a chain of multiple pooling blocks, each pooling block consisting of a... MaxPool2d and a maximum pooling layer It consists of Conv2d layers. Each pooling block takes the output of the previous pooling block as input. Then, through the summation of residual connections, the output feature maps of all pooling blocks are fused together with the input feature maps; The PA module can facilitate the recovery of edge detail information, thereby improving segmentation performance; The PA module includes: sequentially connected The Conv2d layer, BatchNorm2d layer, and Sigmoid() activation layer.

[0027] The specific task output module includes three headers: semantic segmentation, depth estimation, and instance segmentation. For each task, the decoder output is passed through a depth-wise pixel attention layer, a ReLU6 layer, and a 3×3 Conv2d layer, and finally outputs the prediction results of the three tasks: semantic segmentation, depth estimation, and instance segmentation.

[0028] Step 2: Create the Cityscapes scene image dataset, which includes: Figure 6 The dense scene understanding result image provided by the present invention; establish a scene image dataset; the scene image dataset includes multiple images used for semantic segmentation, depth estimation and instance segmentation and their corresponding labels.

[0029] The Cityscapes dataset was selected, specifically including the following three datasets: gtFine_trainvaltest, leftImg8bit_trainvaltest, and disparity_trainvaltest.

[0030] The Cityscapes dataset, primarily promoted by Mercedes-Benz, provides an image segmentation dataset for autonomous driving environments. It is used to evaluate the performance of visual algorithms in semantic understanding of urban scenes. Cityscapes contains street scenes from 50 cities with different scenes, backgrounds, and seasons, mainly captured under good weather conditions. It provides 5000 finely annotated images, 20000 coarsely annotated images, and 39 categories of annotated objects.

[0031] The `leftImg8bit_trainvaltest` and `gtFine_trainvaltest` datasets are important components of the Cityscapes dataset, primarily used for image segmentation and instance segmentation tasks. `leftImg8bit_trainvaltest` contains the original images, and `gtFine_trainvaltest` contains the labeled files, each containing three folders: `train`, `val`, and `test`. The training set (`train`) contains 2975 images, the validation set (`val`) contains 500 images, and for the test set (`test`), only the original images are provided without detailed labels. In the detailed labeled dataset folder (`gtFine`), each image corresponds to four label files: a color-coded image for visualization (`_color.png`), an instance I-image for instance segmentation (`_instanceIds.png`), a label image for semantic segmentation (`_labelsIds.png`), and a JSON file containing the original human-annotated information (`_polygons.json`). The validation set is used to test the model in experiments.

[0032] Step 3: Train the LW-MLTANet model using the Cityscapes dataset to obtain the optimal LW-MLTANet model for dense scene understanding.

[0033] Split the dataset: Use the train set as the training set and the Val set as the test set.

[0034] The LW-MLTANet model is trained using the training set. The optimal LW-MLTANet model for dense scene understanding is obtained through training.

[0035] Multi-task loss function for training LW-MLTANet model for: = in, , , is a scale factor and In the experiment, the following settings were implemented. .

[0036] The semantic segmentation loss function is the standard softmax cross-entropy loss: in, It is a one-hot vector of the real label. It is the first The actual output value of the class node.

[0037] Depth estimation uses inverse Huber loss: in, It is a pixel h The difference between the predicted label and the true label at the given location. t =0.2max( The threshold is the loss function that can provide a more noticeable gradient at locations with smaller depth differences, thus helping to train the network better.

[0038] Instance segmentation uses a discriminative loss function: Variance term: Distance term: Regularization term: in, It is the number of clusters in the actual labels, i.e., the number of instances. It is a cluster c The number of pixels in It is the first p The embedding vector generated by each pixel It is a cluster C The embedded mean vector corresponding to all pixels in the array. and It is the mean vector of different instances. and To represent different instances, Indicates hinge. and These represent the margins for variance and distance loss, respectively. This will make To the mean vector Closer, and This will cause the cluster centers to move further apart. In the experiment, we set... ; =0.5, =1.5, and using the 2-norm.

[0039] During training, data augmentation was applied to the training set using methods including random mirroring, image scaling (Resize = (224, 224)), and image normalization (Normalize) with parameters set as follows: scale = 1 / 255, depth_scale = 250, mean = (0.485, 0.456, 0.406), and standard deviation (std = (0.229, 0.224, 0.225)). The training batch size was 4, using the Adam optimizer with a learning rate (lr = 0.0001), weight decay (weight_decay = 0.0001), and 600 training epochs. The optimal dense scene understanding LW-MLTANet model was obtained and saved through training.

[0040] Step 4: The segmented test set images are resized (Resize=224, 224), normalized (with parameters set as: scale=1 / 255, depth_scale=250, mean=(0.485, 0.456, 0.406), and standard deviation=(0.229, 0.224, 0.225). This data is then input into the optimal dense scene understanding LW-MLTANet model, outputting the results of the dense scene understanding task, which includes the results of semantic segmentation, depth estimation, and instance segmentation. Figure 6 As shown.

[0041] This invention designs an efficient Transformer connecting the encoder and decoder, a lightweight attention network LWA-RefineNet, a task-specific output module, and a multi-task loss function for model training. This dense scene understanding method based on a lightweight multi-task attention network can simultaneously and accurately complete three tasks: semantic segmentation, depth estimation, and instance segmentation in real time, thus improving the accuracy of each task.

[0042] Example 2 Figure 7 This embodiment provides a dense scene understanding system based on a lightweight multi-task attention network, comprising: The LW-MLTANet module is used to build lightweight multi-task attention network LW-MLTANet models, including an input layer, encoder, efficient Transformer, decoder, and task-specific output layer. Figure 8 This is a schematic diagram of the specific structure of the LW-MLTANet module, including: input layer, MobileNet-v2 encoder, high-efficiency Transformer, LWA-RefineNet decoder, and multi-task output layer.

[0043] The scene image dataset module is used to create the Cityscapes scene image dataset, which includes multiple images used for semantic segmentation, depth estimation, and instance segmentation, along with their corresponding labels; the scene image dataset is divided into a training set and a test set. The model training module is used to train the LW-MLTANet model using the Cityscapes dataset to obtain the optimal LW-MLTANet model for dense scene understanding. The output module is used to input the segmented test set images into the optimal dense scene understanding LW-MLTANet model and output the results of the dense scene understanding task, which includes three tasks: semantic segmentation, depth estimation, and instance segmentation.

[0044] The specific functions implemented by each module are as described in the previous embodiment one.

[0045] In summary, this invention constructs a lightweight multi-task attention network, LW-MLTANet, comprising an input layer, an encoder, an efficient Transformer, a decoder, and task-specific output layers. Specifically, the encoder uses a pre-trained MobileNet-v2 network, which boasts fast inference speed and powerful feature extraction capabilities. The efficient Transformer, composed of efficient multi-head attention (EMHA) and an MLP, learns task-specific features from global features while avoiding excessive memory usage and computational load. The decoder is a lightweight attention network, LWA-RefineNet, based on the RefineNet architecture, to reduce the number of parameters and floating-point operations. The encoder, efficient Transformer, and decoder form a shared network to receive input data and learn task-shared features. The shared network branches to the heads of the three tasks, forming task-specific output modules. These modules perform three tasks: semantic segmentation, depth estimation, and instance segmentation. Each task passes the decoder's output through a depth-wise pixel attention layer, a ReLU6 layer, and a 3×3 Conv2d layer, ultimately outputting the prediction results for semantic segmentation, depth estimation, and instance segmentation. After training the LW-MLTANet model, the optimal dense scene understanding LW-MLTANet model can perform real-time predictions for three tasks: semantic segmentation, depth estimation, and instance segmentation, thereby improving the model's accuracy.

[0046] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0047] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for understanding dense scenes based on a lightweight multi-task attention network, characterized in that, include: A lightweight multi-task attention network model, LW-MLTANet, is constructed, comprising an input layer, an encoder, an efficient Transformer, a decoder, and a task-specific output layer. The encoder uses a pre-trained MobileNet-v2 network to extract features from the image. The decoder is a lightweight attention network based on the RefineNet architecture, LWA-RefineNet, which includes six lightweight attention network modules (LW-Attention), three lightweight chained residual pooling attention modules (LW-CRPA), and one lightweight chained residual pooling module (LW-CRP1). The LW-Attention module enhances the network's feature representation capabilities while reducing the number of channels. A scene image dataset for dense scene understanding is established, which includes multiple images for semantic segmentation, depth estimation, and instance segmentation, along with their corresponding labels. The LW-MLTANet model is trained using the scene image dataset for dense scene understanding to obtain the optimal LW-MLTANet model for dense scene understanding. The images from the test set are input into the optimal dense scene understanding LW-MLTANet model to obtain prediction results for three tasks: semantic segmentation, depth estimation, and instance segmentation.

2. The dense scene understanding method based on a lightweight multi-task attention network according to claim 1, characterized in that, The specific task output layer includes three tasks: semantic segmentation, depth estimation, and instance segmentation. Each task passes the decoder output through a depth-wise pixel attention layer, a ReLU6 layer, and a 3×3 Conv2d layer, ultimately outputting the prediction results of the three tasks: semantic segmentation, depth estimation, and instance segmentation.

3. The dense scene understanding method based on a lightweight multi-task attention network according to claim 1, characterized in that, The efficient Transformer is used to connect the encoder and decoder, and includes: two Layer Norms, an efficient multi-head attention EMHA, and a multilayer perceptron MLP containing two fully connected layers; The efficient multi-head attention EMHA builds a reduction layer after Layer Norm to halve the number of channels. Then, the linear layer projects the feature map onto three matrices: Q, K, and V, and then uses feature segmentation to divide Q, K, and V into several equal segments. Each triple of these segments is then subjected to a scaled dot product attention operation.

4. The dense scene understanding method based on a lightweight multi-task attention network according to claim 1, characterized in that, Training the LW-MLTANet model using the scene image dataset for dense scene understanding specifically includes: The scene image dataset is divided into a training set and a test set according to a preset ratio; The LW-MLTANet model is trained using the training set to obtain the optimal LW-MLTANet model for dense scene understanding.

5. The dense scene understanding method based on a lightweight multi-task attention network according to claim 1, characterized in that, The LW-MLTANet model is trained using a multi-task loss function, including: Multi-task loss function for: = ； The semantic segmentation loss function is the standard softmax cross-entropy loss: ； in, , , is a scale factor and It is a one-hot vector of the real label. It is the first The actual output value of the class node.

6. A dense scene understanding system based on a lightweight multi-task attention network, executing the dense scene understanding method based on a lightweight multi-task attention network as described in any one of claims 1-5, characterized in that, include: The LW-MLTANet module is used to build a lightweight multi-task attention network LW-MLTANet model, which includes an input layer, encoder, efficient Transformer module, decoder, and task-specific output layer. The scene image dataset module is used to establish a scene image dataset, which includes multiple images used for semantic segmentation, depth estimation, and instance segmentation, along with their corresponding labels. The scene image dataset is divided into a training set and a test set; The model training module is used to train the LW-MLTANet model using the training set to obtain the optimal dense scene understanding LW-MLTANet model. The output module is used to input the segmented test set images into the optimal dense scene understanding LW-MLTANet model and output the results of the dense scene understanding task, which includes three tasks: semantic segmentation, depth estimation, and instance segmentation.

7. The dense scene understanding system based on a lightweight multi-task attention network according to claim 6, characterized in that, The high-efficiency Transformer module is used to connect the encoder and decoder, and includes: two Layer Norms, a high-efficiency multi-head attention EMHA module, and a multilayer perceptron MLP containing two fully connected layers; Two Layer Norm structures are used in Transformer to prevent gradient vanishing or gradient exploding problems; The EMHA module includes: a Reduction layer that halves the number of channels; a linear layer that projects the feature map onto three matrices: Q, K, and V; and a feature segmentation module that segments the Q, K, and V into several equal segments, performs a scaled dot product attention operation on the triples of the segments, and then uses a Concat layer and an Expansion layer to restore the number of channels.

8. The dense scene understanding system based on a lightweight multi-task attention network according to claim 6, characterized in that, The decoder is a lightweight attention network LWA-RefineNet, which includes a lightweight attention network module LW-Attention, a lightweight chained residual pooling attention module LW-CRPA, and a lightweight chained residual pooling module LW-CRP1.

9. The dense scene understanding system based on a lightweight multi-task attention network according to claim 8, characterized in that, The lightweight attention network module LW-Attention is used to enhance the feature representation capability of the network and reduce the dimensionality of the number of channels. It includes: a Conv2d layer, a BatchNorm2d layer, a ReLU, a Conv2d layer, a BatchNorm2d layer and a Sigmoid() activation layer connected in sequence. The lightweight chained residual pooling attention module LW-CRPA includes: an LW-CRP1 module, a pixel attention PA module, and an Upsample operation connected in sequence. The lightweight chained residual pooling module LW-CRP1 is used to capture background context information of large image regions by combining chained pooling layers and convolutional residuals, while reducing the number of channels.

10. The dense scene understanding system based on a lightweight multi-task attention network according to claim 9, characterized in that, The PA module includes: a 1×1 Conv2d layer, a BatchNorm2d layer, and a Sigmoid() activation layer connected in sequence; The specific task output layer includes three headers: semantic segmentation, depth estimation, and instance segmentation. Each task passes the decoder output through a depth-wise pixel attention layer, a ReLU6 layer, and a 3×3 Conv2d layer, ultimately outputting the prediction results for the three tasks: semantic segmentation, depth estimation, and instance segmentation.

Images