Training method of multimodal image segmentation model and multimodal image segmentation method

Abstract

The present application provides a training method for a multimodal image segmentation model and a multimodal image segmentation method, which relates to the technical field of image segmentation. The training method includes: collecting historical images and corresponding segmentation data, then dividing the historical images and their segmentation data into a training set and a test set according to a certain ratio, then using the training set to perform mask training on the first image prediction submodel, then inputting the test set into the trained first image prediction submodel to obtain predicted image data, then using the training set and the predicted image data to train the first image segmentation submodel, and completing the training of the image segmentation model. The present application further improves the safety and reliability of the autonomous driving system by improving the accuracy of predicting occluded areas and improving the accuracy of image segmentation.

Description

Technical Field

[0001] The present application relates to the technical field of image segmentation, and in particular to a training method of a multimodal image segmentation model and a multimodal image segmentation method. Background Art

[0002] With the rapid development of autonomous driving technology, image segmentation technology has become an indispensable part of autonomous driving systems. Image segmentation technology aims to divide an image into multiple regions with clear semantics, so as to accurately identify key elements such as roads, vehicles, and pedestrians. However, in autonomous driving scenarios, image segmentation technology faces many challenges, one of which is the identification and prediction of occluded areas.

[0003] During the autonomous driving process, sensors (especially cameras) may be blocked by various obstacles (such as other vehicles, pedestrians, trees, buildings, etc.), resulting in the inability to obtain some image information. This kind of occlusion is particularly common in complex and changing road environments, which poses a great challenge to image segmentation technology. If these blocked areas cannot be accurately predicted and accurate image segmentation cannot be achieved, the safety and reliability of the autonomous driving system will be seriously affected. Summary of the invention

[0004] In order to improve the accuracy of prediction of occluded areas and improve the accuracy of image segmentation, the present application provides a training method for a multimodal image segmentation model and a multimodal image segmentation method.

[0005] In a first aspect, the present application provides a method for training a multimodal image segmentation model, which adopts the following technical solution:

[0006] A training method for a multimodal image segmentation model, the image segmentation model comprising: a first image prediction sub-model for predicting an occluded area, and a first image segmentation sub-model for segmenting an image, the training method comprising:

[0007] First acquisition: collect historical images and corresponding segmentation data, and build training sets and test sets;

[0008] Model training: including first training, first prediction and second training;

[0009] First training: using the training set to perform mask training on the first image prediction sub-model to obtain the second image prediction sub-model;

[0010] First prediction: add mask blocks to the samples in the test set, input the test set with the mask blocks added into the second image prediction sub-model, and obtain predicted image data;

[0011] Second training: the first image segmentation sub-model is trained using the training set and the predicted image data to obtain a second image segmentation sub-model.

[0012] By adopting the above technical solution, the present application trains the first image prediction sub-model through masking, so that it can learn how to predict the information of the occluded area, which is suitable for processing occluded or missing image data. The masking training enhances the robustness of the first image prediction sub-model, so that it can better cope with complex scenes. The present application adds mask blocks to the test set samples and inputs them into the first image prediction sub-model (i.e., the second image prediction sub-model) that has been trained with masking, so as to evaluate the prediction ability of the second image prediction sub-model under occlusion. The predicted image data also provides additional data support for the subsequent training of the first image segmentation sub-model. The first image segmentation sub-model is trained in combination with the training set and the predicted image data to improve the segmentation performance of the first image segmentation sub-model under occlusion. In this way, the first image segmentation sub-model can learn how to perform effective and accurate image segmentation under occlusion.

[0013] Optionally, after performing the first acquisition step and before performing the model training step, the method further includes:

[0014] Multimodal data fusion: including second acquisition, first fusion, projection, first modeling and second fusion;

[0015] Second acquisition: acquiring radar point cloud data of the target object in the historical image, and acquiring LiDAR point cloud data of the target object based on the radar point cloud data;

[0016] First fusion: The depth information of radar point cloud data is fused with LiDAR point cloud data to obtain three-dimensional point cloud data containing depth information;

[0017] Projection: Projecting the three-dimensional point cloud data containing depth information onto the imaging plane of the camera to obtain RGBD image data;

[0018] First modeling: Based on the deep learning algorithm, a deep learning model is established, and the deep learning model is used to segment the RGBD image data to obtain the segmentation data of the RGBD image;

[0019] Second fusion: Fusing the segmentation data of the historical image with the segmentation data of the corresponding RGBD image to obtain fused segmentation data, and dividing the fused segmentation data into a new training set and a new test set according to a predetermined ratio.

[0020] By adopting the above technical solution, by acquiring the radar point cloud data and LiDAR point cloud data of the target object in the historical image, three-dimensional spatial information can be introduced, which is helpful to understand the shape, size and position of the object in the image. After that, the depth information of the radar point cloud data is fused with the LiDAR point cloud data, and the complementarity of the two data can be fully utilized to improve the accuracy and completeness of the three-dimensional point cloud data. The three-dimensional point cloud data containing the depth information is projected onto the imaging plane of the camera, and RGBD image data can be generated. This data contains both color information and depth information, which helps to improve the accuracy of image segmentation. The present application also establishes a deep learning model based on a deep learning algorithm, and uses the model to segment the RGBD image data to obtain a more accurate segmentation result. The deep learning model can automatically learn the features of the segmentation data, which is very effective for complex image segmentation tasks. By fusing the segmentation data of the historical image with the segmentation data of the corresponding RGBD image, more abundant and accurate segmentation data can be obtained, and the advantages of the two data can be fully utilized to improve the accuracy and completeness of the segmentation data.

[0021] Optionally, the image segmentation model further includes a first attention sub-model for allocating weights, and after performing the first acquisition step and before performing the model training step, further includes:

[0022] Assign weights: extract some samples from the training set, assign different weights to each part of the sample, and integrate all the samples after weight assignment into the first set;

[0023] Third training: Use the first set to train the first attention sub-model to obtain the second attention sub-model.

[0024] By adopting the above technical solution, some samples in the training set are extracted for weight assignment, and different degrees of importance can be given to different parts of the sample. This differential weight assignment helps the image segmentation model pay more attention to important information during the training process, thereby improving the accuracy of segmentation. Using the first set after weight assignment to train the first attention sub-model, the first attention sub-model can learn how to assign appropriate weights according to different parts of the sample, which helps the image segmentation model to more accurately identify key information in subsequent image segmentation tasks. By training the first attention sub-model, the robustness and generalization ability of the image segmentation model can be further enhanced, so that it can perform well in different scenarios and conditions.

[0025] Optionally, after performing the third training step and before performing the model training step, the method further includes:

[0026] Data update: including the first update and the second update;

[0027] First update: Use the remaining samples in the training set as the input of the second attention sub-model, assign different weights to each part of the samples in the training set, and integrate the training set samples with weights into a new training set;

[0028] Second update: Use the samples in the test set as the input of the second attention sub-model, assign different weights to each part of the samples in the test set, and integrate the test set samples with weights into a new test set.

[0029] By adopting the above technical solution and using the second attention sub-model to assign weights to the remaining samples in the training set, the training process can pay more attention to the sample parts that have an important impact on the performance improvement of the model, which helps the image segmentation model to learn key features faster, thereby improving the accuracy of segmentation. The updated training set contains weight information, which is a data enhancement method. Using the updated training set to train the first prediction sub-model and the first image segmentation sub-model of the image segmentation model helps to improve the generalization ability of the image segmentation model. Similarly, assigning weights to the samples in the test set can make the segmentation process more objective and accurate.

[0030] Optionally, after executing the step of updating the data and before executing the step of training the model, the method further includes:

[0031] Image pruning: including setting weight thresholds and modifying grayscale values;

[0032] Set weight threshold: Get the weight of the jth pixel in the i-th part of the current sample And gray value , and calculate the adaptive gray value threshold B, the calculation model of the adaptive gray value threshold B is as follows:

[0033] ;

[0034] Where p is the number of image blocks in the sample; q is the number of pixels contained in the i-th part;

[0035] Modify the grayscale value: obtain the part of the current sample whose grayscale value is less than the grayscale value threshold B, record it as the background, and set the grayscale value of the background to 0.

[0036] By adopting the above technical solution, the present application can dynamically adjust the threshold according to the weights and gray values ​​of different parts in the current sample by calculating the adaptive gray value threshold B, which helps to more accurately identify the background area, thereby reducing the interference of irrelevant information on model training. Setting the part with a gray value less than the gray value threshold B as the background, and setting the gray value of the background to 0 can simplify the image content, reduce the impact of noise and redundant information on model training, and help the image segmentation model focus more on the segmentation of key learning parts. By setting the gray value of the background to 0, the foreground (the target image block to be segmented) and the background in the image can be made clearer, which is beneficial to subsequent image segmentation tasks. By introducing the image pruning step and using the adaptive gray value threshold B to optimize the training data, the image segmentation model can focus more on learning key features during the training process while reducing the interference of noise and redundant information.

[0037] Optionally, the image segmentation model further includes a first image generation sub-model for generating an image, and after performing the first acquisition step and before performing the model training step, further includes:

[0038] Image generation: including image processing and fourth training;

[0039] Image processing: using image transformation technology to process the historical image to obtain a first image, and adding the first image to the training set;

[0040] Fourth training: using the historical image and the first image to train the first image generation sub-model to obtain the second image generation sub-model.

[0041] By adopting the above technical solution and processing the historical images with image transformation technology, more diverse image data can be generated, thereby increasing the richness and diversity of the training set. After that, the training set samples containing the first image are input into the first image segmentation sub-model for mask training, which helps to improve the recognition ability and generalization performance of the image segmentation model for different image features. Adding the processed first image to the training set can expand the scale of the training samples, provide more learning opportunities for the model, and help improve the accuracy of the image segmentation model for image segmentation. By training the first image generation sub-model with historical images and the first image, the image generation sub-model can learn how to generate more diverse image data, which helps to generate more challenging test samples in subsequent image segmentation tasks, thereby improving the performance of the image segmentation model. The trained first image generation sub-model is used to generate more diverse image data to support the execution of image segmentation tasks. By introducing the first image generation sub-model and performing the image generation step before training, the performance of the image segmentation model can be further enhanced. This enhancement can learn more diverse image features during the training process, thereby improving the accuracy of segmentation.

[0042] Optionally, after executing the step of image generation and before executing the step of model training, the method further includes:

[0043] Image deduplication: including similarity calculation, similarity judgment, deletion and retention;

[0044] Calculating similarity: Based on the hash algorithm, calculating the similarity between any two images in the historical image and / or the first image, and recording it as the first data;

[0045] Similarity judgment: judging whether the first data is greater than a preset similarity threshold, if so, executing the deletion step; if not, executing the retention step;

[0046] Deletion: deleting any one of the two images;

[0047] Keep: Keep the two images.

[0048] By adopting the above technical solution, the image similarity is calculated based on the hash algorithm, and the repeated or highly similar parts between images can be quickly and effectively identified. The hash algorithm has a low computational complexity and is suitable for processing large-scale image data. By calculating the similarity of any two images, the redundancy of the image data in the training set can be comprehensively evaluated, providing a reliable basis for subsequent deduplication operations. By presetting the similarity threshold, the strictness of deduplication can be flexibly controlled. A higher similarity threshold means a stricter deduplication standard, which helps to retain more diverse image data; a lower similarity threshold will cause more images to be misjudged as duplicates and deleted. By deleting duplicate or highly similar images, the data quality of the training set can be optimized, the interference of redundant information on model training can be reduced, and the generalization performance and accuracy of the image segmentation model can be improved. Retaining images with different features can maintain the diversity of the training set and provide more learning opportunities for the image segmentation model. By performing the image deduplication step before training, the quality and diversity of the training data can be further optimized, which helps to improve the performance of the image segmentation model, enabling it to more accurately identify the feature differences between different images, thereby improving the accuracy of segmentation.

[0049] Optionally, after executing the similarity determination step and before executing the deletion step, the method further includes:

[0050] Image acquisition: acquiring two images corresponding to the current first data, and recording the two images as a third image and a fourth image respectively;

[0051] Acquiring time: acquiring the timestamps of the third image and the fourth image respectively, and calculating the time difference;

[0052] Time judgment: judge whether the time difference is less than a preset time difference threshold, if so, execute the retaining step; if not, execute the deleting step.

[0053] By adopting the above technical solution, by explicitly obtaining the two images (the third image and the fourth image) involved in the similarity judgment, and respectively obtaining the timestamps of the third image and the fourth image (the timestamp information is one of the metadata of the image file, which is easy to obtain and does not require additional calculation or processing), the time sequence and relative interval in which they were created or captured can be reflected, and it is easy to judge whether the images are repeated or redundant. The present application presets a time difference threshold, which can flexibly control the timestamp-based deduplication strategy. If the time difference between the two images is less than the preset threshold, it means that they were captured or created at a similar time, so they are more likely to have duplicate content, but there is no need to delete the images. By introducing the two links of image acquisition and time judgment, the steps of image deduplication have been further refined and optimized. This optimization not only takes into account the similarity between images, but also combines the timestamp information of the images, thereby improving the accuracy and flexibility of the deduplication strategy.

[0054] Optionally, in the first training step, the mask blocks of the third image and the fourth image are at different positions.

[0055] By adopting the above-mentioned technical scheme, the third image and the fourth image of the present application are images acquired successively, and mask blocks at different positions are set in the third image and the fourth image, that is, mask blocks at different positions are set on the images of continuous frames. Through the change of the position of the mask blocks, the image segmentation model can be helped to identify the movement of the target object between different frames. During the training process, the image segmentation model continuously adapts and adjusts its internal parameters to accurately segment the masked areas at different positions. This training method helps to improve the generalization ability of the image segmentation model, so that it can handle a wider range and more complex image segmentation tasks.

[0056] In a second aspect, the present invention provides a multimodal image segmentation method, which adopts the following technical solution:

[0057] A multimodal image segmentation method comprises the following steps:

[0058] The third acquisition: acquiring the image to be segmented;

[0059] Image segmentation: inputting the image to be segmented into an image segmentation model to obtain an image segmentation result; wherein the image segmentation model is trained using the training method of the multimodal image segmentation model.

[0060] In summary, the present application includes at least one of the following beneficial technical effects:

[0061] 1. This application trains the first image prediction sub-model through masking, so that it can learn how to predict the information of the occluded area, which is suitable for processing occluded or missing image data. The masking training enhances the robustness of the first image prediction sub-model, so that it can better cope with complex scenes. This application adds mask blocks to the test set samples and inputs them into the first image prediction sub-model (i.e., the second image prediction sub-model) that has been trained with masking, so as to evaluate the prediction ability of the second image prediction sub-model under occlusion. The predicted image data also provides additional data support for the subsequent training of the first image segmentation sub-model. The first image segmentation sub-model is trained in combination with the training set and the predicted image data to improve the segmentation performance of the first image segmentation sub-model under occlusion. In this way, the first image segmentation sub-model can learn how to perform effective and accurate image segmentation under occlusion.

[0062] 2. By acquiring the radar point cloud data and LiDAR point cloud data of the target object in the historical image, three-dimensional spatial information can be introduced, which is helpful to understand the shape, size and position of the object in the image. After that, the depth information of the radar point cloud data is fused with the LiDAR point cloud data, which can make full use of the complementarity of the two data to improve the accuracy and completeness of the three-dimensional point cloud data. The three-dimensional point cloud data containing depth information is projected onto the imaging plane of the camera to generate RGBD image data. This data contains both color information and depth information, which helps to improve the accuracy of image segmentation. This application also establishes a deep learning model based on a deep learning algorithm, and uses the model to segment the RGBD image data to obtain more accurate segmentation results. The deep learning model can automatically learn the features of the segmentation data, which is very effective for complex image segmentation tasks. By fusing the segmentation data of the historical image with the segmentation data of the corresponding RGBD image, more abundant and accurate segmentation data can be obtained, and the advantages of the two data can be fully utilized to improve the accuracy and completeness of the segmentation data.

[0063] 3. The image transformation technology is used to process the historical images, which can generate more diverse image data, thereby increasing the richness and diversity of the training set. After that, the training set samples containing the first image are input into the first image segmentation sub-model for mask training, which helps to improve the recognition ability and generalization performance of the image segmentation model for different image features. Adding the processed first image to the training set can expand the scale of the training samples, provide more learning opportunities for the model, and help improve the accuracy of the image segmentation model for image segmentation. By using historical images and the first image to train the first image generation sub-model, the image generation sub-model can learn how to generate more diverse image data, which helps to generate more challenging test samples in subsequent image segmentation tasks, thereby improving the performance of the image segmentation model. The trained first image generation sub-model is used to generate more diverse image data to support the execution of image segmentation tasks. By introducing the first image generation sub-model and performing the image generation step before training, the performance of the image segmentation model can be further enhanced. This enhancement can learn more diverse image features during the training process, thereby improving the accuracy of segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a flowchart of S1 first acquisition and S2 model training in Example 1 of the present application;

[0065] Figure 2 is a flowchart of S11 multimodal data fusion of Example 1 of the present application;

[0066] Figure 3 is a flow chart of Example 2 of the present application;

[0067] Figure 4 This is a flowchart from S41 image generation to S42 image deduplication in Example 3 of the present application;

[0068] Figure 5 This is a flowchart of Example 3 of the present application with S422 similarity judgment to S423 omitted. DETAILED DESCRIPTION

[0069] The following combination Figures 1 to 5 This application is described in further detail.

[0070] Embodiment 1: This embodiment discloses a training method for a multimodal image segmentation model. In this embodiment, the image segmentation model includes: a first image prediction sub-model for predicting an occluded area, a first image segmentation sub-model for segmenting an image, the output end of the first image prediction sub-model is connected to the input end of the first image segmentation sub-model, and the first image segmentation sub-model is connected to the output end of the first image prediction sub-model. Figure 1The training method includes: S1 first acquisition and S2 model training, firstly, the historical image and the corresponding segmentation data are acquired, then the historical image and the segmentation data thereof are divided into a training set and a test set according to a certain ratio, then the training set is used to perform mask training on the first image prediction sub-model, then the mask block is added to the test set and input into the trained first image prediction sub-model to obtain the predicted image data, then the training set and the predicted image data are used to train the first image segmentation sub-model, and the training of the image segmentation model is completed. This embodiment includes the following steps:

[0071] S1 is the first acquisition, which involves acquiring historical images and corresponding segmentation data, and constructing a training set and a test set according to a certain ratio using the historical images and the corresponding segmentation data (in this embodiment, the historical images are divided into a training set and a test set according to a ratio of 7:3, and in other embodiments, they may also be divided according to other ratios). The samples in the training set and the test set include sample images and sample labels, and the sample labels mark the segmentation results of each part of the image. The test set is used to evaluate the performance of the second image prediction sub-model. When the difference between the predicted image data and the original sample labels exceeds expectations, S21 is re-executed to adjust the parameters of the first image prediction sub-model.

[0072] The segmentation data exists in the form of pixel-level labels, marking the boundaries and categories of different parts in the image, namely, sample labels.

[0073] In S1 first acquisition, the sample image is also preprocessed, and the preprocessing includes: adjusting the image size, normalizing pixel values, etc.

[0074] S2 model training includes S21 first training, S22 first prediction and S23 second training. This step aims to train the first image prediction sub-model and the first image segmentation sub-model respectively to obtain a trained image segmentation model.

[0075] S21 is the first training. The training set is used to perform mask training on the first image prediction sub-model. Mask training refers to masking some areas on the input image (sample image of the training set) randomly or according to certain rules, forcing the first image prediction sub-model to learn to infer the content of the masked part from the remaining information. This training method helps to improve the first image prediction sub-model's ability to understand and utilize image context information. After multiple iterative trainings, the second image prediction sub-model is obtained.

[0076] The first image prediction sub-model may adopt a generative adversarial network (GAN) sub-model or a convolutional neural network (CNN) sub-model. The first image segmentation sub-model may adopt a fully convolutional network (FCN) sub-model or a DeepLab series network sub-model.

[0077] S22 first prediction, adds a mask block to each sample image in the test set to simulate the occlusion situation that may be encountered in actual applications, and inputs these test sets with mask blocks into the second image prediction sub-model. The second image prediction sub-model will predict the content of the masked part based on the remaining information and generate predicted image data. These predicted image data not only include the second image prediction sub-model’s understanding of the original image, but also reflect the second image prediction sub-model’s prediction ability when facing occlusion.

[0078] The generation method of the mask block includes a masking strategy based on a Transformer model and a masking strategy based on a MaskedGAN.

[0079] The masking strategy based on the Transformer model is as follows:

[0080] Random sampling: Randomly select a portion of elements (such as image blocks in the image) from the input image as masking objects.

[0081] Masking operation: Replace the selected image patches with special masking tags (such as [MASK]), which are used to predict the values ​​of the original elements during model training.

[0082] Masking ratio: The masking ratio can be adjusted according to the specific task and data set. A higher masking ratio can increase the prediction difficulty of the second image prediction model, thereby forcing it to learn more comprehensive information. However, the proportion of masked blocks in the sample image needs to be less than 25%.

[0083] The masking strategy of MaskedGAN is as follows:

[0084] Spatial Masking: In image data, randomly selecting image patches to mask the spatial dimension helps force the MaskedGAN’s discriminator to learn information from all image locations, not just the easily distinguishable parts.

[0085] Spectral masking: Decomposing the image into multiple bands in the spectral space and masking part of the spectral bands with adaptive probability helps to encourage the discriminator of MaskedGAN to learn the difficult-to-distinguish bands, thereby improving the quality of the generated images.

[0086] Combination of masking strategies: Combining spatial masking and spectral masking can further improve the performance of MaskedGAN in masking with limited data.

[0087] S23 second training, using the training set and the predicted image data to train the first image segmentation sub-model to obtain the second image segmentation sub-model.

[0088] The purpose of this step is to use the additional information in the predicted image data to further refine the first image segmentation sub-model's understanding of image segmentation. By introducing the predicted image data, the first image segmentation sub-model can learn more about the image structure and context, thereby improving the accuracy and robustness of the segmentation.

[0089] This embodiment provides a training process for an image segmentation model, including collecting segmentation data from historical images and constructing a training set and a test set, followed by mask training to obtain preliminary image prediction capabilities, then adding mask blocks to the test set samples and using a second image prediction sub-model to generate predicted image data, and finally further training the first image segmentation sub-model in combination with the training set and the predicted image data to improve its segmentation accuracy and robustness, thereby achieving the training of the image segmentation model.

[0090] Reference Figure 2 In other embodiments, after performing S1 first acquisition and before performing S2 model training, it also includes:

[0091] S11 multimodal data fusion, including S111 second acquisition, S112 first fusion, S113 projection, S114 first modeling and S115 second fusion.

[0092] S111 is a second acquisition, in which point cloud data of the target object is extracted from the radar data corresponding to the historical image. The radar point cloud data is stored in a variety of formats, such as a two-dimensional radar scanning image or a three-dimensional point cloud.

[0093] This step also uses the spatial registration technology to obtain the LiDAR (Light Detection and Ranging) point cloud data of the target object based on these radar point cloud data. The process is as follows:

[0094] Point cloud segmentation: divide the radar point cloud data into different areas or objects for subsequent identification and positioning. The segmentation methods include distance-based segmentation, density-based segmentation, etc.

[0095] Target recognition: In the segmented point cloud data, the target object is identified through machine learning or deep learning algorithms. These algorithms can be based on the geometric properties of the point cloud (such as position, direction, curvature, etc.) or contextual information (such as relationship with other points, interaction with the environment, etc.). After obtaining the target object, its reflection signal is obtained based on the historical data of the LiDAR sensor on the autonomous vehicle, and the LiDAR point cloud data of the vehicle's surrounding environment is generated.

[0096] Point cloud filtering and noise reduction: In order to improve the quality of the target object LiDAR point cloud data, the LiDAR point cloud data also needs to be filtered and denoised.

[0097] Rigid body transformation: Due to the movement of the LiDAR device or the movement of the target object, the LiDAR point cloud data may be in different coordinate systems. Rigid body transformation aligns the LiDAR point cloud data to the same coordinate system through rotation and translation operations.

[0098] S112 is the first fusion, which fuses the depth information of the radar point cloud data with the LiDAR point cloud data to obtain three-dimensional point cloud data containing depth information, including: extracting features from the original data based on the ResNet autoencoder network and fusing the extracted features; projecting the LiDAR point cloud into the image coordinate system and fusing it with the image data.

[0099] The ResNet-based autoencoder network extracts features from the original data and fuses the extracted features. The specific steps are as follows:

[0100] Feature extraction: ResNet’s autoencoder network extracts the depth information of radar point cloud data and the features of LiDAR point cloud data to obtain depth features and LiDAR features.

[0101] Feature fusion: The depth feature and LiDAR feature are weighted and summed to obtain the fused feature. The calculation model is as follows:

[0102] ;

[0103] in, is the feature after the i-th fusion; is the i-th depth feature; is the weight coefficient of the i-th deep feature; m is the number of deep features; is the weight coefficient of the jth LiDAR feature; is the jth LiDAR feature; n is the number of LiDAR features.

[0104] For example: D=[1,2,3], E=[4,5,6], =[0.1,0.3,0.6], =[0.3,0.3,0.4];

[0105] Then C=[1*0.1+4*0.3, 2*0.3+5*0.3, 3*0.6+6*0.4]=[1.3,2.1,4.2].

[0106] Decoding: Based on the fused features, the decoder of the autoencoder network is used to reconstruct the point cloud data to obtain three-dimensional point cloud data containing depth information.

[0107] S113 projection, projecting the three-dimensional point cloud data containing depth information onto the imaging plane of the camera to obtain RGBD (Red, Green, Blue, and Depth) image data, the process is as follows:

[0108] S1131 Camera internal and external parameter calibration: The internal and external parameters of the camera are calibrated through camera calibration technology. The internal parameters include the focal length, optical center position, etc. of the camera, while the external parameters describe the position and posture of the camera in the world coordinate system, including rotation angle and translation parameters. The rotation angle constitutes a rotation matrix, which is used to realize the rotation transformation between different coordinate systems. The translation parameters constitute a translation matrix, which is used to realize the translation transformation between different coordinate systems.

[0109] In this embodiment, the intrinsic parameter matrix of the camera is , the intrinsic parameter matrix is ​​the key to realize the conversion between image coordinate system and pixel coordinate system. Indicates the length occupied by a pixel on the x-axis; Indicates the length occupied by a pixel on the y-axis; is the radial distortion parameter; f is the focal length of the camera; is the principal coordinate point of the pixel coordinate system.

[0110] S1132 sets the projection matrix: sets the projection matrix G based on the internal and external parameters of the camera.

[0111] ;

[0112] ;

[0113] ;

[0114] Where R is the rotation matrix; is the rotation angle; T is the translation matrix; is the horizontal coordinate of the target object in the world coordinate system; is the ordinate of the target object in the world coordinate system.

[0115] S1133 Projection of point cloud data to imaging plane: including S11331 coordinate transformation, S11332 perspective projection and S11333 depth information fusion.

[0116] S11331 Coordinate transformation: transform the 3D point cloud data from the world coordinate system to the camera coordinate system through rotation and translation transformation. The rotation matrix and translation matrix are both determined by the external parameters of the camera.

[0117] S11332 Perspective projection: Use the projection matrix to project points in the camera coordinate system into the two-dimensional pixel coordinate system. This process involves the principle of perspective projection, that is, the visual effect of things being larger when they are near and smaller when they are far.

[0118] S11333 Depth Information Fusion: By assigning a corresponding depth value to each pixel in the RGB image, the depth information (i.e., the distance from each point in the point cloud data to the camera) is combined with the RGB image data to generate RGBD image data.

[0119] S114 is the first modeling, which is based on the deep learning algorithm to establish a deep learning model, and use the deep learning model to segment the RGBD image data. Through the learning ability of the deep learning model, the segmentation information of the target object is extracted from the RGBD image data to obtain the RGBD image segmentation data.

[0120] In this embodiment, the deep learning model may adopt a CNN or RNN model.

[0121] S115 second fusion, fusing the segmentation data of the historical image with the corresponding RGBD image segmentation data to obtain the fused segmentation data, the process is as follows:

[0122] S1151 Feature extraction: Extract the shape, contour and other features of the object from the segmentation data of the historical image, and integrate them into a feature vector, recorded as the first feature vector; extract the color, texture and depth and other features from the RGBD image segmentation data, and integrate them into a feature vector, recorded as the second feature vector.

[0123] S1152 Feature fusion: concatenate the first feature vector and the second feature vector end to end to obtain the third feature vector.

[0124] S1153 adding labels: associating the segmentation data of the historical image with the third eigenvector, with the purpose of adding a corresponding category label (ie, the third eigenvector) to each pixel point of the historical image to obtain fused segmentation data.

[0125] The fused segmented data is divided into a new training set and a new test set according to the same proportion as in the first acquisition of S1, and the new training set and the new test set are used to perform S2 model training.

[0126] This embodiment integrates segmentation data of different modalities through S11 multimodal data fusion, enriches the training set and test set, improves the training effect of the image segmentation model, and builds a high-performance image segmentation model.

[0127] Example 2: Reference Figure 3The difference between this embodiment and Embodiment 1 is that the image segmentation model also includes a first attention sub-model for allocating weights, the output end of the first prediction sub-model of the image segmentation sub-model is connected to the input end of the first attention sub-model, and the output end of the first attention sub-model is connected to the input end of the first image segmentation sub-model.

[0128] After executing S1 first acquisition and before executing S2 model training, it also includes:

[0129] S31 assigns weights, extracts some samples from the training set, assigns different weights to each part of the sample, and integrates all the samples after the weights are assigned into a first set.

[0130] Each part refers to the image block that needs to be segmented and the remaining image blocks. One part corresponds to one image block. For example, if an object needs to be segmented in the image, then each part includes: this object and the remaining part of the image (if the remaining part is further divided into different image blocks, each image block will correspond to a part).

[0131] S32 is the third training, using the first set to train the first attention sub-model to obtain the second attention sub-model.

[0132] S33 data update, including S331 first update and S332 second update.

[0133] S331 is the first update, which takes the remaining samples in the training set as the input of the second attention sub-model, aiming to assign different weights to each part of the samples in the training set and integrate the training set samples with weights into a new training set.

[0134] The second update of S332 takes the test set as the input of the second attention sub-model, aiming to assign different weights to each part of the samples in the test set and integrate the test set samples with weights into a new test set.

[0135] S34 image pruning, including S341 setting a weight threshold and S342 modifying a gray value.

[0136] S341 sets a weight threshold, obtains a sample in the first update of S331 and / or the second update of S332, and extracts the weight of the jth pixel in the i-th part of the sample And gray value , and calculate the adaptive gray value threshold B, the calculation model of the adaptive gray value threshold B is as follows:

[0137] ;

[0138] Among them, p is the number of image blocks in the sample, that is, the sample is divided into several parts; q is the number of pixels contained in the i-th part.

[0139] The adaptive gray value threshold B can be dynamically adjusted according to the weight distribution and gray value distribution of the sample, so as to better distinguish the foreground and the background and perform image segmentation more accurately.

[0140] S342 modifies the grayscale value, obtains the part of the current sample whose grayscale value is less than the grayscale value threshold B, records it as the background, and sets the grayscale value of the background to 0, thereby removing noise and irrelevant information in the image and making the foreground object (the target object to be segmented) more prominent.

[0141] This embodiment can provide a more accurate and robust solution for the image segmentation task by introducing the first attention sub-model and executing the above steps. This method can focus on the important parts of the image and assign higher weights to them, thereby improving the performance of the image segmentation model in complex scenes. At the same time, through the image pruning step, the image data can be further optimized to improve the accuracy and clarity of the segmentation results.

[0142] Example 3: Reference Figure 4 The difference between this embodiment and Embodiment 1 is that the image segmentation model also includes a first image generation sub-model for generating an image, and the output end of the first image generation sub-model is connected to the input end of the first image prediction sub-model.

[0143] After executing S1 first acquisition and before executing S2 model training, it also includes:

[0144] S41 image generation, including S411 image processing and S412 fourth training.

[0145] S411 image processing, selecting a historical image as input, applying one or more image transformation techniques to process, and obtaining a first image.

[0146] The image transformation technology includes: rotation, translation, and scaling. When the image transformation technology is used to perform relevant operations on the historical image, the segmentation data corresponding to the historical image is also subjected to the above operations to generate the first image and the segmentation data corresponding to the first image.

[0147] The first image and its corresponding segmentation data are added to the training set to form a more diverse and larger training data set.

[0148] S412 The fourth training uses the historical image and the first image obtained by image transformation as training data to train the first image generation sub-model, aiming to enable the first image generation sub-model to learn how to generate new images that are similar to the real image but not exactly the same, and the trained first image generation sub-model is recorded as the second image generation sub-model.

[0149] S42 image deduplication includes S421 similarity calculation, S422 similarity judgment, S423 deletion and S424 retention.

[0150] S421 calculates the similarity, converts the image into a compact hash value based on a hash algorithm (such as perceptual hash, average hash, etc.), and then calculates the Hamming distance or Euclidean distance between these hash values ​​to obtain the similarity between any two images in the historical image and / or the first image, which is recorded as the first data.

[0151] Taking the perceptual hash algorithm as an example, the process is as follows:

[0152] Reduce image: Reduce the image to a fixed size (32x32 in this example).

[0153] Convert to Grayscale: Convert a color image to grayscale.

[0154] Calculate DCT: Perform discrete cosine transform on the grayscale image to obtain the DCT image.

[0155] Take the top left 8x8 region of the DCT: The top left region of the DCT image contains most of the energy of the image.

[0156] Calculate mean: Calculate the mean energy of the 8x8 region.

[0157] Generate a binary image: Compare each pixel with the mean, set pixels greater than the mean to 1, and pixels less than the mean to 0.

[0158] Generate hash value: Convert the binary image into a 64-bit hash value (treat each row as an 8-bit binary number and concatenate them).

[0159] Calculate the distance: Calculate the Hamming distance of the 64-bit hash values ​​corresponding to the two images through bit operations, and record the Hamming distance as the similarity (ie, the first data).

[0160] S422 similarity determination, determining whether the first data is greater than a preset similarity threshold, if so, it means that the two images are highly similar, and S423 is executed for deletion; if not, it means that the two images are not highly similar, and S424 is executed for retention.

[0161] S423: Deleting any one of the two images to reduce redundancy.

[0162] S424 retains the two images corresponding to the first data.

[0163] This embodiment can effectively improve the training efficiency and final performance of the image segmentation model through the above-mentioned steps of image generation and image deduplication. Image generation increases the diversity of training data, while image deduplication ensures the quality and effectiveness of training data.

[0164] Reference Figure 5 In other embodiments, after executing S422 similarity determination and before executing S423 deletion, the method further includes:

[0165] S4221 image acquisition, acquiring two images corresponding to the current first data, and marking the two images as a third image and a fourth image respectively.

[0166] S4222 obtains the time by reading metadata (such as EXIF ​​information) of the image, respectively obtaining the timestamps of the third image and the fourth image, and subtracting the timestamps of the two images to obtain the time difference between the two timestamps.

[0167] S4223 time judgment, judging whether the time difference is less than a preset time difference threshold. If so, it means that the two images were captured in a very short time and are continuous records of the same event or scene. In this case, even if their content is highly similar, S424 retention needs to be performed to retain the continuity or diversity in time by retaining the two images; if not, it means that the two images have a certain interval in time and represent different moments or events. In this case, if their content is highly similar, they are considered redundant and S423 deletion needs to be performed.

[0168] In the first training in S21, the mask blocks of the third image and the fourth image are at different positions.

[0169] By introducing the steps of image acquisition and time judgment, this embodiment can comprehensively consider the content and time attributes of the image for image deduplication, thereby making more accurate and reasonable decisions and ensuring the quality and diversity of the training set, which is of great significance for improving the training efficiency and final performance of the image segmentation model.

[0170] Embodiment 4: This embodiment also provides a multimodal image segmentation method, including:

[0171] The third acquisition: acquiring the image to be segmented;

[0172] Image segmentation: inputting the image to be segmented into an image segmentation model to obtain an image segmentation result; wherein the image segmentation model is trained using the training method of the multimodal image segmentation model.

[0173] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A multimodal image segmentation model training method, characterized in that: The image segmentation model includes: a first image prediction sub-model for predicting an occluded area, a first image segmentation sub-model for segmenting an image, and a first attention sub-model for assigning weights, and the training method includes: First acquisition: collect historical images and corresponding segmentation data, and build training sets and test sets; Multimodal data fusion: including second acquisition, first fusion, projection, first modeling and second fusion; Second acquisition: acquiring radar point cloud data of the target object in the historical image, and acquiring LiDAR point cloud data of the target object based on the radar point cloud data; First fusion: The depth information of radar point cloud data is fused with LiDAR point cloud data to obtain three-dimensional point cloud data containing depth information; Projection: Projecting the three-dimensional point cloud data containing depth information onto the imaging plane of the camera to obtain RGBD image data; First modeling: Based on the deep learning algorithm, a deep learning model is established, and the deep learning model is used to segment the RGBD image data to obtain the segmentation data of the RGBD image; Second fusion: fusing the segmentation data of the historical image with the segmentation data of the corresponding RGBD image to obtain fused segmentation data, and dividing the fused segmentation data into a new training set and a new test set according to a predetermined ratio; Assign weights: extract some samples from the training set, assign different weights to each part of the sample, and integrate all the samples after weight assignment into the first set. Each part refers to the image block to be segmented and the remaining image blocks, and one part corresponds to one image block. Third training: Use the first set to train the first attention sub-model to obtain the second attention sub-model; Data update: including the first update and the second update; First update: Use the remaining samples in the training set as the input of the second attention sub-model, assign different weights to each part of the samples in the training set, and integrate the training set samples with weights into a new training set; Second update: Use the samples in the test set as the input of the second attention sub-model, assign different weights to each part of the samples in the test set, and integrate the weighted test set samples into a new test set; Image pruning: including setting weight thresholds and modifying grayscale values; Set weight threshold: Get the weight of the jth pixel in the i-th part of the current sample And gray value , and calculate the adaptive gray value threshold B, the calculation model of the adaptive gray value threshold B is as follows: ； Where p is the number of image blocks in the sample; q is the number of pixels contained in the i-th part; Modify the gray value: obtain the part of the current sample whose gray value is less than the gray value threshold B, record it as the background, and set the gray value of the background to 0; Model training: including first training, first prediction and second training; First training: using the training set to perform mask training on the first image prediction sub-model to obtain the second image prediction sub-model; First prediction: add mask blocks to the samples in the test set, input the test set with the mask blocks added into the second image prediction sub-model, and obtain predicted image data; Second training: the first image segmentation sub-model is trained using the training set and the predicted image data to obtain a second image segmentation sub-model.

2. The multimodal image segmentation model training method according to claim 1, characterized in that: The image segmentation model further includes a first image generation sub-model for generating an image, and after performing the first acquisition step and before performing the model training step, further includes: Image generation: including image processing and fourth training; Image processing: using image transformation technology to process the historical image to obtain a first image, and adding the first image to the training set; Fourth training: using the historical image and the first image to train the first image generation sub-model to obtain the second image generation sub-model.

3. The multimodal image segmentation model training method according to claim 2, characterized in that: After performing the image generation step and before performing the model training step, it also includes: Image deduplication: including similarity calculation, similarity judgment, deletion and retention; Calculating similarity: Based on the hash algorithm, calculating the similarity between any two images in the historical image and / or the first image, and recording it as the first data; Similarity judgment: judging whether the first data is greater than a preset similarity threshold, if so, executing the deletion step; if not, executing the retention step; Deletion: deleting any one of the two images; Keep: Keep the two images.

4. The multimodal image segmentation model training method according to claim 3, characterized in that: After executing the similarity determination step and before executing the deletion step, the method further includes: Image acquisition: acquiring two images corresponding to the current first data, and recording the two images as a third image and a fourth image respectively; Acquiring time: acquiring the timestamps of the third image and the fourth image respectively, and calculating the time difference; Time judgment: judge whether the time difference is less than a preset time difference threshold, if so, execute the retaining step; if not, execute the deleting step.

5. The multimodal image segmentation model training method according to claim 4, characterized in that: In the first training step, the mask blocks of the third image and the fourth image are at different positions.

6. A multimodal image segmentation method, characterized in that: include: The third acquisition: acquiring the image to be segmented; Image segmentation: inputting the image to be segmented into an image segmentation model to obtain an image segmentation result; wherein the image segmentation model is trained using the training method of the multimodal image segmentation model as described in any one of claims 1 to 5.

Images