A scene image semantic segmentation and surface normal estimation method

Abstract

The application discloses a scene image semantic segmentation and surface normal estimation method, and belongs to the technical field of image processing.The method comprises the following steps: image acquisition and pretreatment, building a multi-task learning model and training, and then predicting the to-be-tested image based on the trained model to obtain the scene image semantic segmentation and surface normal estimation output.The original PAD-Net model is improved in the application, and grouping learning is performed on the tasks in the initial multi-task learning prediction stage, then the intermediate prediction information is fused through a multi-modal distillation module, and finally the final prediction result is generated, so that the prediction accuracy of the existing multi-task learning method can be improved, and the original PAD-Net model is greatly improved; meanwhile, the relationship between the tasks is fully mined in the model training process, more abundant visual features are provided for the interaction and fusion of information, the stability of the original model can be improved, the training converges faster, the shock is reduced, and the final multi-task prediction result is obviously promoted.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and more specifically, relates to a method for semantic segmentation of scene images and surface normal estimation. Background Technology

[0002] In traditional deep learning, different visual tasks are solved individually by a single neural network; this is known as single-task learning. In contrast, multi-task learning utilizes a single deep neural network to handle multiple different tasks simultaneously. Solving multiple problems at once is common and more reasonable in real-world scenarios. For example, when the human eye perceives a scene, it simultaneously determines the object's category, size, and distance. This approach also has many real-world applications. For instance, in autonomous driving scenarios, to ensure safe vehicle operation, it's necessary to perceive the surrounding environment and simultaneously perform multiple visual tasks such as detecting lane lines, detecting pedestrians, and determining the distance and position of objects relative to the vehicle.

[0003] Multi-task learning methods have attracted extensive research and discussion in recent years, resulting in numerous achievements. However, how to share information among multiple tasks and how to reduce negative transfer between tasks remain unresolved challenges. Recently, a novel multi-task learning network model, PAD-Net, has been proposed. Unlike previous multi-task learning models, PAD-Net first generates intermediate prediction tasks, then uses a multimodal distillation module to extract information from these intermediate prediction tasks, and finally fuses the results to generate the final task prediction. Experimental results show that PAD-Net can significantly improve the prediction accuracy for multiple tasks. During the training phase, PAD-Net is used as the multi-task learning model, with a dataset containing images and multiple task labels as training samples input into the PAD-Net model, enabling it to extract corresponding visual features for each visual task. After feature extraction, initial multi-task learning predictions are generated, producing initial predictions for four different visual tasks. Subsequently, the generated initial task predictions are fused using a multimodal distillation module, enabling the sharing and interaction of different modal information from multiple visual tasks. The fused information is then input into the decoder prediction module to generate prediction results for each visual task. The difference between the prediction results and the true values ​​is calculated using a loss function, and the PAD-Net model is optimized using a backpropagation algorithm.

[0004] However, the relationship between tasks is the key to determining the learning effect of multi-task learning. The PAD-Net model uses the initial prediction to provide data of different modalities for the final task prediction, but it does not make full use of the relationship between tasks. When performing multi-task learning, different combinations of visual tasks affect the final task prediction results, which can easily lead to negative transfer and thus generate large prediction errors. Therefore, it is crucial to design a more reasonable and effective multi-task model. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a scene image semantic segmentation and surface normal estimation method. By improving the traditional PAD-Net model, it makes full use of the relationship between tasks to carry out multi-task learning, thereby outputting the prediction results of the influence of different visual task combinations.

[0006] The technical solution adopted in this invention is as follows:

[0007] A method for semantic segmentation and surface normal estimation of scene images, comprising the following steps:

[0008] Step 1, Image Acquisition and Preprocessing;

[0009] Several original images containing natural scenes were collected to obtain initial sample images, and sample augmentation processing was performed on the initial sample images.

[0010] After normalizing the size of the initial sample images and the expanded sample images, an ignored region is added to each image, and then each image is converted into a tensor type to obtain training samples.

[0011] Assign a corresponding task label to each training sample to form a training sample set;

[0012] Step 2: Build a multi-task learning model;

[0013] The multi-task learning model includes a front-end backbone encoder HRNet, an initial multi-task prediction module, a multimodal distillation module, and a decoder prediction module;

[0014] Among them, the output of the front-end backbone encoder HRNet is connected to the initial multi-task prediction module;

[0015] The initial multi-task prediction module includes two branches, upper and lower. The output of each branch is connected to the multimodal distillation module and the encoder prediction module in sequence. Each branch contains multiple parallel upsampling modules. Each upsampling module performs one task. The sampling result of each upsampling module corresponds to the initial prediction result of one task. The initial prediction results of all tasks in the upper and lower branches are input to the multimodal distillation module.

[0016] The multimodal distillation module includes a semantic segmentation multimodal distillation module and a surface normal multimodal distillation module, which are used to fuse the initial prediction results of multiple tasks in the upper and lower branches, respectively, and then input the fused results of the upper and lower branches into the decoder prediction module.

[0017] Step 3: Train the multi-task learning model;

[0018] Step 301: Set the number of training iterations (Epoch) and the batch size (batch_size) for each iteration;

[0019] Step 302: Randomly select batch_size training samples from the training sample set and input them into the front-end backbone encoder HRNet. HRNet extracts the image features of each training sample in parallel at different scales, and then performs feature fusion on the image features at different scales to obtain fused image features, and inputs the fused image features into the initial multi-task prediction module.

[0020] Step 303: In the initial multi-task prediction module, the upsampling modules of the upper and lower branches simultaneously upsample the fused image features input by HRNet to generate initial prediction results for multiple tasks, which are then input into the corresponding multimodal distillation module.

[0021] In the upper branch, the i-th training sample is passed through the upper branch. The initial prediction results output by each upsampling module are denoted as

[0022] In the next branch, the i-th training sample is passed through the next branch. The initial prediction results output by each upsampling module are denoted as

[0023] The training sample numbers for the current batch are i = 1, 2, ..., batch_size;

[0024] These represent the task numbers of the upper and lower branches, respectively, and the upsampling module numbers of the upper and lower branches;

[0025] Step 304: The multimodal distillation module fuses the initial prediction results of multiple tasks;

[0026] Step 304-1: In the previous branch, the semantic segmentation multimodal distillation module is used to process the initial prediction results of each task. Feature maps are generated after convolution.

[0027] Calculating feature maps using attention mechanisms attention map

[0028]

[0029] in, Indicates the first The attention weights for each task, σ(·) represents the normalized attention map. This represents the dot product operation;

[0030] Fusing the initial prediction results from multiple tasks:

[0031]

[0032] in, This represents the multi-task fusion result of the i-th training sample output from the upper branch, where k1 represents the non-task result among the task numbers in the upper branch. The task number is randomly assigned, T1 represents the number of tasks in the upper branch, and W... k1 This represents the attention weight for task k1;

[0033] Step 304-2: In the next branch, the initial prediction results for each task are processed using the surface normal multimodal distillation module. Feature maps generated by convolution operations

[0034] Calculating feature maps using attention mechanisms attention map

[0035]

[0036] in, Indicates the first The attention weights for each task, σ(·) represents the normalized attention map. This represents the dot product operation;

[0037] Fusing the initial prediction results from multiple tasks:

[0038]

[0039] in, This represents the multi-task fusion result of the i-th training sample output by the next branch, ⊙ represents the dot product operation, and k2 represents the non-task result among the task numbers in the next branch. The task number is randomly assigned, T2 represents the number of tasks in the next branch, and W... k2 This represents the attention weight for task k2;

[0040] Step 305: The decoder prediction module calculates the prediction results of semantic segmentation and surface normal estimation;

[0041] The decoder prediction module consists of two branches. In the upper branch, the decoder prediction module processes the multi-task fusion results. Upsampling is performed to restore the spatial resolution to that of the training samples, thus obtaining the semantic segmentation prediction results. In the next branch, the decoder prediction module processes the multi-task fusion results. Upsampling is performed to restore the spatial resolution to that of the training samples, thus obtaining the predicted surface normals.

[0042] Step 306: Calculate the loss function value after this iteration;

[0043] Calculate the loss function value after sampling by each upsampling module in the upper branch;

[0044] Calculate the loss function value after sampling for each upsampling module in the lower branch;

[0045] Calculate the upsampled loss function values ​​of the upper and lower branches of the decoder prediction module;

[0046] The obtained loss function values ​​are then weighted and fused to obtain the total loss function value after this iteration;

[0047] Step 307: Determine whether the total loss function value after the last two iterations has converged (i.e., whether the deviation between the two total loss function values ​​tends to 0). If yes, it means that the multi-task learning model has converged, training ends and step 4 is executed; otherwise, step 308 is executed.

[0048] Step 308: Determine if the current iteration number is less than the iteration number Epoch. If so, increment the iteration number by 1 and return to step 302; otherwise, end the training and execute step 4.

[0049] Step 4: After normalizing the size of the image to be tested, input it into the multi-task learning model trained in Step 3 to obtain the semantic segmentation result and surface normal estimation result of the image to be tested.

[0050] In the scene image semantic segmentation and surface normal estimation method of the present invention, the original images containing natural scenes are first collected and preprocessed to form a training set; then a multi-task learning model is built and trained. During the training process, the relationship between tasks is fully utilized to carry out multi-task learning, thereby outputting the prediction results under the influence of different visual task combinations until the model converges. Finally, the converged model is used to perform semantic segmentation and surface normal estimation on the test image.

[0051] Furthermore, in step 1, adding an ignored region specifically means: setting regions with an image normal of 0, no human body part annotations, or a depth of 0 as ignored regions, and setting the pixel value of pixels within the ignored region to 255.

[0052] Furthermore, in step 1, the sample augmentation process for the initial sample image specifically involves: first performing a horizontal flip with a probability of 0.5 on the image, and then performing a random rotation.

[0053] Furthermore, in step 306, when calculating the total loss function value, the weight of each loss function value is set to 1.

[0054] The technical solution provided by this invention brings at least the following beneficial effects:

[0055] (1) The present invention improves the original PAD-Net model by grouping tasks for learning in the initial multi-task learning and prediction stage, and then using a multimodal distillation module to fuse intermediate prediction information and generate the final prediction result. This can improve the prediction accuracy of existing multi-task learning methods and has a significant improvement over the original PAD-Net model.

[0056] (2) The relationship between tasks is the key to determining whether multi-task learning promotes or inhibits the learning effect. This invention fully explores the relationship between tasks during the model training process, providing richer visual features for information interaction and fusion, which can improve the stability of the original model, converge faster during training, reduce oscillations, and have a significant promoting effect on the final multi-task prediction results. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart of a scene image semantic segmentation and surface normal estimation method provided by an embodiment of the present invention;

[0059] Figure 2 This is a diagram of the structure of a multi-task learning model;

[0060] Figure 3 This is a model structure diagram of the front-end backbone encoder HRNet;

[0061] Figure 4 This is a schematic diagram of the integration of the multimodal distillation module. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0063] As one possible implementation, such as Figure 1 As shown, the specific implementation steps of a scene image semantic segmentation and surface normal estimation method provided in this embodiment of the invention include:

[0064] S1. Image acquisition and preprocessing;

[0065] S1.1. Obtain a large number of original images containing natural scenes for model training;

[0066] In this embodiment, 500 original images can be downloaded from both the PASCAL dataset and the NYUD-v2 dataset;

[0067] S1.2, Data preprocessing and augmentation;

[0068] Data preprocessing involves processing and augmenting data before training. Augmentation can expand the original dataset, increasing its richness. During training, it can enhance the generalization ability of the model and reduce overfitting.

[0069] In this embodiment, the original images from the downloaded NYUD-v2 and PASCAL datasets are first horizontally flipped with a probability of 0.5, and then rotated and scaled.

[0070] Then, the resolution of the image data is adjusted to a uniform size of m×n. In this embodiment, image data from the same dataset are adjusted to a uniform size. For example, the resolution of images downloaded from NYUD-v2 is adjusted to 640×480, and the resolution of images downloaded from the PASCAL dataset is adjusted to 512×512;

[0071] S1.3 Add Ignore Regions: In each image, set the regions with image normals of 0, no human body markings, or depths of 0 as ignore regions, and set the pixel value of the pixels in the ignore regions to 255.

[0072] Finally, the image data in the dataset is converted into tensors and normalized to (0, 1) using the formula: Image = Image / 255, where Image represents the image pixel value. A training sample is then obtained based on the converted image data, and a corresponding task label is assigned to each training sample to construct a training sample set (or training set for short).

[0073] S1.4 Construct a training set based on the image data processed in the above steps.

[0074] In this embodiment, two training sets are constructed: one consisting of images downloaded from the NYUD-v2 dataset and the other consisting of images downloaded from the PASCAL dataset. Both training sets can be used for the model training below.

[0075] S2. Build a multi-task learning model;

[0076] like Figure 2As shown, the multi-task learning model includes a front-end backbone encoder HRNet, an initial multi-task prediction module, a multimodal distillation module, and a decoder prediction module.

[0077] In this embodiment, the model structure of the front-end backbone encoder HRNet is as follows: Figure 3 As shown, there are four branches, each of which extracts feature features through several convolutional layers. They are named as the first to fourth branches from top to bottom. The feature map scales of the four branches decrease in that order. Multiple intermediate feature maps of the first branch are downsampled and then sent to the corresponding convolutional layer of the second branch. Multiple intermediate feature maps of the second branch are downsampled and then sent to the corresponding convolutional layer of the third branch. Multiple intermediate feature maps of the third branch are downsampled and then sent to the corresponding convolutional layer of the fourth branch.

[0078] The output feature maps of the penultimate convolutional layer of each branch are defined as feature maps f1″, f2″, f3″, and f4″, respectively, and the output feature maps of the penultimate convolutional layer of each branch are defined as feature maps f1′, f2′, f3′, and f4′, respectively. Feature map f1″ is downsampled and then fed into the last convolutional layer of the second, third, and fourth branches; feature map f2″ is downsampled and then fed into the last convolutional layer of the third and fourth branches; and feature map f3″ is downsampled and then fed into the last convolutional layer of the fourth branch. Simultaneously, feature map f4″ is upsampled and then fed into the last convolutional layer of the third, second, and first branches; feature map f3″ is upsampled and then fed into the last convolutional layer of the second and first branches; and feature map f2″ is upsampled and then fed into the last convolutional layer of the first branch. This achieves the fusion of the output feature maps of the penultimate convolutional layers of the four branches as the input to the last convolutional layer of each branch. Finally, f2′, f3′, and f4′ are upsampled respectively so that the spatial resolution of the upsampled feature map is the same as that of f1′. Then, the feature map f1′ and the upsampled feature maps f2′, f3′, and f4′ are concatenated according to the channel dimension to obtain the final fused image features. These fused image features will be input into the initial multi-task prediction module in the backend.

[0079] like Figure 3As shown, in this embodiment, the first branch sequentially includes four convolutional blocks and one convolutional layer; the second branch sequentially includes one convolutional layer, three convolutional blocks, and one convolutional layer; the third branch sequentially includes one convolutional layer, two convolutional blocks, and one convolutional layer; and the fourth branch sequentially includes one convolutional layer, one convolutional block, and one convolutional layer. Each convolutional block includes five stacked convolutional layers. The output feature map of the fourth convolutional layer of the first convolutional block of the first branch is downsampled and used as the input feature map of the first convolutional block of the second branch. The output feature map of the fourth convolutional layer of the second convolutional block of the first branch is downsampled and compared with the output feature map of the fourth convolutional layer of the second convolutional block of the second branch. Figure 1The input features are fed into the fifth convolutional layer of the second convolutional block of the second branch. The output feature maps of the fourth convolutional layer of the second convolutional block of the first branch and the fourth convolutional layer of the first convolutional block of the second branch are downsampled and then fused to form the input feature map of the first convolutional block of the third branch. Simultaneously, the output feature map of the fourth convolutional layer of the first convolutional block of the second branch is upsampled and fused with the output feature map of the fourth convolutional layer of the second convolutional block of the first branch before being input into the fifth convolutional layer of the second convolutional block of the first branch. That is, for the last convolutional layer of each convolutional block in each branch, its input includes the output feature map of the fourth convolutional layer of the current branch's convolutional block and the output feature maps of the fourth layers of the corresponding convolutional blocks of other branches (after upsampling or downsampling). The input feature map of the fourth branch is: the output feature map of the fourth convolutional layer of the third convolutional block of the first branch and the output feature map of the fourth convolutional layer of the second convolutional block of the second branch. The fused features of the output feature maps of the first convolutional block of the third branch and the output feature map of the fourth convolutional layer of the first convolutional block of the third branch; the input feature map of the fifth convolutional layer of the third convolutional block of the first branch includes: the output feature map of the fourth convolutional layer of this branch and the upsampled feature map of the output feature map of the fourth convolutional layer of the second convolutional block of the second branch, and the upsampled feature map of the output feature map of the fourth convolutional layer of the first convolutional block of the third branch; the output feature maps of non-this branch are upsampled to the same resolution as the output feature map of this branch, and then feature fusion is performed to obtain the input feature map of the fifth convolutional layer of this branch and the first convolutional block of this branch; and so on, the input feature map of the fifth convolutional layer of the second convolutional block of the second branch includes: the downsampled result of the output feature map of the fourth convolutional layer of the third convolutional block of the first branch, the output feature map of the fourth convolutional layer of this branch and the upsampled result of the output feature map of the fourth convolutional layer of the first convolutional block of the third branch. The input feature map of the fifth convolutional layer of the first convolutional block of the third branch includes: the downsampled result of the output feature map of the fourth convolutional layer of the third convolutional block of the first branch, the downsampled result of the output feature map of the fourth convolutional layer of the second convolutional block of the second branch, and the output feature map of the fourth convolutional layer of this branch and this convolutional block.

[0080] The HRNet in this embodiment extracts features in parallel at multiple scales of the input data and fuses feature information from different scales, thus having a stronger representational ability compared to ResNet.

[0081] In this embodiment, the initial multi-task prediction module mainly functions to upsample the feature map using multi-task learning, thereby generating an initial prediction result. Its structure includes two branches, upper and lower, with the output of each branch sequentially connected to the multimodal distillation module and the decoder prediction module. In this embodiment, as shown... Figure 2As shown, each branch contains two parallel upsampling modules, each performing one task. Based on the simultaneous performance of four visual tasks, the visual tasks are grouped and then multi-task learning is performed to fully explore the relationships between tasks. This can promote the improvement of accuracy in multi-task learning. The initial prediction results are still used as multimodal input data and input to the multimodal distillation module for sharing and fusion.

[0082] In this embodiment, the multimodal distillation module takes the initial multi-task prediction results output from the previous stage as input, enabling information from different tasks to interact during this stage, such as... Figure 4 As shown, this is an important module for feature sharing across multiple different tasks in the model.

[0083] In this embodiment, a total of 4 tasks are involved, Y in the figure. i 1 Y i 2 Y represents the initial prediction results of the two tasks in the upper branch of the multimodal distillation module. i 3 Y i 4 F represents the initial prediction results for the two tasks in the lower branch of the multimodal distillation module. i 1 F i 2 This indicates that the feature map F is generated by convolutional operations on the initial prediction results of each task using the semantic segmentation multimodal distillation module. i 3 F i 4 This represents the feature map generated by convolutional operations on the initial prediction results of each task using the surface normal multimodal distillation module. G represents the attention mechanism, and F represents the feature map generated by convolutional operations on the initial prediction results of each task. i o,1 F i o,4 F represents the multi-task fusion results of the upper and lower branches, respectively. i o,1 For semantic segmentation decoders, F i o,4 Used for surface normal estimation decoder.

[0084] In this embodiment, the decoder corresponds to the encoder. During the image encoding process, as the convolutional layers increase, the image size is gradually reduced. Therefore, the feature map size is increased in the decoder stage to restore the feature map to the original image size, thus generating the final prediction result of the model.

[0085] S3. Train a multi-task learning model;

[0086] S3.1 Set the number of training iterations Epoch=100, and the batch size is batch_size=16;

[0087] S3.2. Randomly select batch_size images from the training set and input them into the front-end backbone encoder HRNet. HRNet extracts the features of each image in parallel at different scales, and then fuses the feature information at different scales and inputs it into the initial multi-task prediction module.

[0088] S3.3 In the initial multi-task prediction module, the upsampling modules of the upper and lower branches simultaneously upsample the fused features of the HRNet input, generating initial prediction results for multiple tasks, which are then input into the corresponding multimodal distillation modules. In the upper branch, the initial prediction result of the i-th image after the t-th task is denoted as... In the next branch, the initial prediction result of the i-th image after the t-th task is denoted as... i = 1, 2, ..., batch_size, t = 1, 2, ..., T, where T represents the number of tasks; in this embodiment, T = 4, and the upper branch is used for the initial prediction of tasks 1 and 2, and the lower branch is used for the prediction of tasks 3 and 4.

[0089] S3.4 The multimodal distillation module fuses the initial prediction results of multiple tasks;

[0090] S3.4.1 In the previous branch, the semantic segmentation multimodal distillation module is used to process the initial prediction results of each task. Feature maps generated by convolution operations t = 1, 2;

[0091] Calculating feature maps using attention mechanisms attention map

[0092]

[0093] Among them, W t Let σ(·) represent the attention weight for the t-th task, and let σ(·) represent the normalized attention map. This represents the dot product operation;

[0094] Fusing the initial prediction results from multiple tasks:

[0095]

[0096] S3.4.2 In the next branch, the surface normal multimodal distillation module is used to process the initial prediction results of each task. Feature maps generated by convolution operations

[0097] Calculating feature maps using attention mechanisms attention map

[0098]

[0099] in, Indicates the first The attention weights for each task, σ(·) represents the normalized attention map. This represents the dot product operation;

[0100] Fusing the initial prediction results from multiple tasks:

[0101]

[0102] Where ⊙ represents the dot product operation, Indicates the first Feature maps for each task Indicates the first Attention weight for each task;

[0103] S3.5, The decoder prediction module calculates the prediction results of semantic segmentation and surface normal estimation;

[0104] S3.5.1 In the upper branch, the decoder prediction module upsamples the fused feature map to restore its resolution and obtain the semantic segmentation prediction result.

[0105] S3.5.2 In the next branch, the decoder prediction module upsamples the fused feature map to restore its resolution and obtain the predicted surface normal.

[0106] S3.6 Calculate the loss function value after this iteration;

[0107] S3.6.1 Calculate the loss function value after sampling by each upsampling module in the upper branch;

[0108] S3.6.2 Calculate the loss function value after sampling by each upsampling module in the lower branch;

[0109] S3.6.3 Calculate the upsampled loss function values ​​of the upper and lower branches of the decoder prediction module;

[0110] The total loss function value is obtained by adding the above four loss function values.

[0111] S3.7 Determine whether the difference between the total loss function values ​​after two adjacent iterations tends to 0. If so, it indicates that the multi-task learning model has converged, training ends, and proceed to step S4; otherwise, proceed to step S3.8.

[0112] S3.8 Determine if the current iteration number is less than the iteration number Epoch. If so, increment the current iteration number by 1 and return to step S3.2; otherwise, proceed to step S4.

[0113] S4. Perform image preprocessing on the image to be tested to match the input of the multi-task learning model. Then, input the preprocessed image to be tested into the trained multi-task learning model to output the semantic segmentation and surface normals of the image to be tested.

[0114] In this embodiment of the invention, when optimizing a multi-task learning model based on the backpropagation algorithm, after the model extracts image features to obtain prediction results for semantic segmentation and surface normal estimation tasks, the similarity between the prediction results and the true values ​​is calculated using a loss function, and the difference between the true values ​​and the predicted values ​​is used to optimize the model using the backpropagation algorithm.

[0115] In this embodiment of the invention, the cross-entropy loss function is used to calculate the loss function value after sampling by each upsampling module for the outputs of the two branches of the initial multi-task prediction module. For the two branches of the decoder prediction module, the upper branch (semantic segmentation prediction branch) uses a balanced cross-entropy loss function to calculate its upsampled loss function value, while the lower branch (surface normal prediction branch) uses L1 Loss as the loss function. The difference gradually decreases with the increase of training iterations, and the curve of the loss function plotted with the number of training iterations gradually converges, indicating that model optimization is complete.

[0116] After the multi-task learning model is trained, the validation set can be input into the model to obtain the prediction results of semantic segmentation and surface normal estimation. The prediction results can then be evaluated using model evaluation metrics. In this embodiment, the Mean Intersection over Union (mIoU) is used as the evaluation metric for the semantic segmentation task; the Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) are used as the evaluation metrics for the surface normal estimation task.

[0117] To demonstrate the accuracy of the predictions in this invention, this embodiment uses mIoU as the evaluation metric for semantic segmentation, where Intersection over Union (IoU) is the ratio of the intersection to the union of the predicted and ground truth values. RMSE is used as the evaluation metric for surface normal estimation, measuring the average of the squared distances between the predicted and actual values. This embodiment also evaluates the performance of the multi-task learning model against the performance of the single-task model for the corresponding visual task, employing Multi-task performance (the current mainstream multi-task learning evaluation method). This embodiment tested the model on the PASCAL and NYUD-v2 datasets, as shown in Tables 1 and 2. It can be seen that compared to the original PAD-Net model, the improved model of this invention shows a certain improvement in prediction accuracy for both semantic segmentation and surface normal estimation tasks, and the performance of the multi-task learning model is also significantly improved.

[0118] Table 1. Visual task prediction results on the PASCAL dataset under different experimental settings.

[0119]

[0120] Table 2 shows the prediction results of the visual task on the NYUD-v2 dataset under different experimental settings;

[0121]

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0123] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A method for semantic segmentation and surface normal estimation of scene images, characterized in that, Includes the following steps: Step 1, Image Acquisition and Preprocessing; Several original images containing natural scenes were collected to obtain initial sample images, and sample augmentation processing was performed on the initial sample images. After normalizing the size of the initial sample images and the expanded sample images, an ignored region is added to each image, and then each image is converted into a tensor type to obtain training samples. Specifically, the ignored region is set as an ignored region, where the image normal is 0, there is no human body part annotation, or the depth is 0. The pixel value of the pixels in the ignored region is 255. Assign a corresponding task label to each training sample to form a training sample set; Step 2: Build a multi-task learning model; The multi-task learning model includes a front-end backbone encoder HRNet, an initial multi-task prediction module, a multimodal distillation module, and a decoder prediction module; Among them, the output of the front-end backbone encoder HRNet is connected to the initial multi-task prediction module; The initial multi-task prediction module includes two branches, upper and lower. The output of each branch is connected to the multimodal distillation module and the encoder prediction module in sequence. Each branch contains multiple parallel upsampling modules. Each upsampling module performs one task. The sampling result of each upsampling module corresponds to the initial prediction result of one task. The initial prediction results of all tasks in the upper and lower branches are input to the multimodal distillation module. The multimodal distillation module includes a semantic segmentation multimodal distillation module and a surface normal multimodal distillation module, which are used to fuse the initial prediction results of multiple tasks in the upper and lower branches, respectively, and then input the fused results of the upper and lower branches into the decoder prediction module. Step 3: Train the multi-task learning model; Step 301: Set the number of training iterations (Epoch) and the batch size (batch_size) for each iteration; Step 302: Randomly select batch_size training samples from the training sample set and input them into the front-end backbone encoder HRNet. HRNet extracts the image features of each training sample in parallel at different scales, and then performs feature fusion on the image features at different scales to obtain fused image features, and inputs the fused image features into the initial multi-task prediction module. Step 303: In the initial multi-task prediction module, the upsampling modules of the upper and lower branches simultaneously upsample the fused image features input by HRNet to generate initial prediction results for multiple tasks, which are then input into the corresponding multimodal distillation module. In the upper branch, the first The training sample is processed by the first branch. The initial prediction results output by each upsampling module are denoted as ; In the next branch, the first The training sample is processed by the first branch. The initial prediction results output by each upsampling module are denoted as ; The number of training samples in the current batch ; , These represent the task numbers of the upper and lower branches, respectively, and the upsampling module numbers of the upper and lower branches; Step 304: The multimodal distillation module fuses the initial prediction results of multiple tasks; Step 304-1: In the previous branch, the semantic segmentation multimodal distillation module is used to process the initial prediction results of each task. Feature maps are generated after convolution. ; Calculating feature maps using attention mechanisms attention map : in, Indicates the first Attention weight for each task This represents a normalized attention map. This represents the dot product operation; Fusing the initial prediction results from multiple tasks: in, This represents the multi-task fusion result of the i-th training sample output from the upper branch. Indicates the non-task in the task number of the upper branch. The task number, This indicates the number of tasks in the upper branch. Indicates task Attention weight; Step 304-2: In the next branch, the initial prediction results for each task are processed using the surface normal multimodal distillation module. Feature maps generated by convolution operations ; Calculating feature maps using attention mechanisms attention map : in, Indicates the first Attention weight for each task This represents a normalized attention map. This represents the dot product operation; Fusing the initial prediction results from multiple tasks: in, This represents the multi-task fusion result of the i-th training sample output by the next branch. This represents the dot product operation. Indicates the non-task numbers in the next branch. The task number, Indicates the number of tasks in the next branch. Indicates task Attention weight; Step 305: The decoder prediction module calculates the prediction results of semantic segmentation and surface normal estimation; The decoder prediction module consists of two branches. In the upper branch, the decoder prediction module processes the multi-task fusion results. Upsampling is performed to restore the spatial resolution to that of the training samples, thus obtaining the semantic segmentation prediction results. In the next branch, the decoder prediction module processes the multi-task fusion results. Upsampling is performed to restore the spatial resolution to that of the training samples, thus obtaining the predicted surface normals. ; Step 306: Calculate the loss function value after this iteration; The cross-entropy loss function is used to calculate the loss function value of each upsampling module in the upper branch after sampling. The cross-entropy loss function is used to calculate the loss function value of each upsampling module in the lower branch after sampling. Calculate the upsampled loss function values ​​of the upper and lower branches of the decoder prediction module; wherein, the upper branch uses the balanced cross-entropy function to calculate the upsampled loss function value, and the lower branch uses L1 Loss as the loss function; The obtained loss function values ​​are then weighted and fused to obtain the total loss function value after this iteration; Step 307: Determine whether the total loss function value after the last two iterations has converged. If it has, it means that the multi-task learning model has converged, training ends and step 4 is executed; otherwise, step 308 is executed. Step 308: Determine if the current iteration number is less than the iteration number Epoch. If so, increment the iteration number by 1 and return to step 302; otherwise, end the training and execute step 4. Step 4: After normalizing the size of the image to be tested, input it into the multi-task learning model trained in Step 3 to obtain the semantic segmentation result and surface normal estimation result of the image to be tested.

2. The method as described in claim 1, characterized in that, In step 1, the sample augmentation process for the initial sample image is specifically performed by first performing a horizontal flip with a probability of 0.5 on the image, and then performing a random rotation.

3. The method as described in claim 1, characterized in that, In step 306, when calculating the total loss function value, the weight of each loss function value is set to 1.

Images