Multi-view semantic recognition method based on depth map assistance

Abstract

The invention discloses a multi-view semantic recognition method based on depth map assistance. Firstly, a multi-view depth map and a main view RGB image are acquired; then generating and processing a depth point cloud; semantic recognition and deep fusion semantic recognition of the single-view RGB image are carried out respectively; and finally, synthesizing output results of the single-view semantic network SVSN and the deep fusion semantic network DFSN through a semantic fusion module to generate a final semantic recognition result. According to the method, the three-dimensional geometric structure information is captured by using the depth point cloud, meanwhile, rich texture and semantic features are extracted through the RGB image, and weighted average is introduced in a semantic fusion stage to be combined with a rule-based dynamic adjustment strategy, so that the problem that a traditional method is easy to be interfered by illumination variation, shielding and noise in a complex scene is effectively solved. The method not only improves the comprehensiveness and accuracy of semantic recognition, but also enhances the ability of the model to cope with multi-modal data inconsistency and input abnormality, and has a wide application prospect.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and deep learning, especially to the field of multi-view semantic recognition. Specifically, the present invention relates to a multi-view semantic recognition method based on depth map assistance, which combines the network architecture of deep fusion semantic network and single view semantic network to achieve efficient fusion recognition of multi-view depth point cloud and main view RGB image. Background Art

[0002] With the development of computer vision technology, semantic recognition is increasingly used in autonomous driving, drone navigation, augmented reality (AR) and other fields. The goal of semantic recognition is to assign semantic labels to each pixel in the scene, such as roads, vehicles, pedestrians, etc. Traditional semantic recognition methods usually rely on single-view RGB images, but this method often faces challenges when dealing with complex scenes. For example, factors such as object occlusion, parallax between multiple perspectives, and lighting changes will affect the accuracy of recognition.

[0003] To solve these problems, researchers have proposed multi-view semantic recognition methods, which combine information from multiple perspectives in the hope of improving the robustness and accuracy of recognition. However, multi-view methods need to process a large amount of high-dimensional data, and how to effectively fuse multi-view information remains a challenge. In particular, when it comes to fusing data from different modalities, such as RGB images and depth maps, traditional methods usually face the following problems:

[0004] Difficulty in feature alignment: There is parallax between multiple views, resulting in different representations of the same object under different viewing angles. This difference is particularly evident for depth maps and RGB images. How to effectively align features in multi-view, multi-modal data to achieve meaningful fusion is a key issue.

[0005] Imperfect fusion strategy: Most traditional methods use simple feature splicing or weighted averaging for modal fusion, but this method is difficult to give full play to the complementary advantages of each modal data and may lead to information loss or redundancy.

[0006] High computing resource requirements: Processing multi-view and multi-modal data usually requires high computing resources. How to reduce the computational complexity while ensuring recognition accuracy is also an important research direction.

[0007] Therefore, in recent years, some researchers have proposed multimodal fusion networks combined with deep learning. For example, PointFusion uses the features of point clouds and RGB images, fuses the two through a simple connection layer, and finally realizes 3D object detection. MV3D projects the point cloud to multiple perspectives such as bird's-eye view and front view, and combines RGB images for multimodal feature extraction and fusion for 3D target detection. Frustum PointNets crops the corresponding point cloud blocks through the image detection frame, and uses PointNet to process the cropped point cloud to achieve 3D target detection. However, most of these networks have problems such as difficulty in feature alignment, strong dependence on data quality, and insufficient generalization ability.

[0008] In summary, traditional multi-view semantic recognition methods face significant challenges in feature alignment, modal fusion and computational complexity. The present invention aims to solve technical problems such as insufficient modal feature fusion, insufficient information redundancy and noise processing, sparsity of deep information, and insufficient real-time and consistency modeling in existing multi-view semantic recognition methods. These problems are effectively solved by designing a deep fusion semantic network (DFSN) and a single view semantic network (SVSN), and efficient fusion and high-precision semantic recognition of multi-view and multi-modal data are achieved. DFSN is responsible for processing the fusion of multi-view depth point clouds and RGB images, while SVSN focuses on the semantic recognition of the main view RGB image. The final recognition result is a combination of the outputs of these two networks, achieving higher recognition accuracy and robustness. Summary of the invention

[0009] In view of the shortcomings of the prior art, the present invention provides a multi-view semantic recognition method based on depth map assistance, which realizes high-precision semantic recognition by fusing multi-view depth point clouds with the RGB image information of the main view. In order to achieve this goal, the present invention designs a deep fusion algorithm including two main modules: a depth fusion semantic network (DFSN) module and a single view semantic network (SVSN) module.

[0010] The Deep Fusion Semantic Network (DFSN) is one of the core networks in this invention, which is specially used to process the fusion of multimodal data. It receives multi-view depth point cloud and main view RGB image, and completes fusion and recognition through the following steps:

[0011] Multimodal feature extraction: DFSN first performs independent feature extraction on the input depth point cloud and RGB image, and uses a convolutional neural network (CNN) to extract the spatial features and semantic features of the depth point cloud and RGB image respectively.

[0012] Feature alignment: After extracting features from the multi-view depth point cloud and the main view RGB image, DFSN aligns these features in the spatial dimension through a feature alignment module. This module uses an alignment algorithm based on an attention mechanism to ensure that features of data from different modalities are effectively fused in the same semantic space.

[0013] Feature fusion: The aligned features are fused in the feature fusion module. This module adopts an improved feature fusion strategy to fuse depth features with RGB features at multiple levels and scales to enhance the model's ability to recognize objects in complex scenes.

[0014] Semantic decoding: The fused features are used to generate semantic labels for multi-view depth point clouds and RGB images through a semantic decoder. DFSN also incorporates scene context information in this process to improve the recognition accuracy of occluded objects and boundary areas.

[0015] The Single View Semantic Network (SVSN) is used to perform preliminary semantic recognition on the main view RGB image. The network mainly includes the following modules:

[0016] Image feature extraction: SVSN uses a deep convolutional neural network to extract features from the input RGB image and generate a series of image features with different scales.

[0017] Multi-level semantic decoding: SVSN decodes the extracted image features layer by layer through a multi-level semantic decoder and outputs the semantic label of each pixel in the image.

[0018] Attention mechanism enhancement: During the feature decoding process, SVSN integrates an attention mechanism to enhance the recognition of small objects in complex scenes while reducing sensitivity to background noise.

[0019] Fusion recognition results: After SVSN and DFSN complete the recognition of RGB images and depth point clouds respectively, the semantic fusion module fuses the two recognition results. The fusion module uses weighted average or rule-based strategies to combine the single-view recognition results of SVSN and the multimodal fusion results of DFSN to generate the final global semantic recognition results. This fusion strategy can not only improve the accuracy of recognition, but also use the strong semantic information of RGB images to supplement when depth information is missing or the noise is large, thereby enhancing the robustness of the semantic fusion module.

[0020] Through the above steps, the depth map-assisted multi-view semantic recognition method of the present invention effectively combines the information of the multi-view depth point cloud and the main view RGB image, achieving higher recognition accuracy and robustness.

[0021] A multi-view semantic recognition method based on depth map assistance is implemented by the following steps:

[0022] Step 1: Acquire multi-view depth maps and main view RGB images.

[0023] Obtain multi-view depth maps and main view RGB images of the target scene from multiple cameras or sensors. These cameras can synchronously capture different view images and depth data of the same scene. Preprocess the acquired multi-view depth map data, including denoising, depth value normalization and other operations to ensure the quality of the depth map data. Select an RGB image of a main view, i.e., the main view, as the input image for single-view semantic recognition. The main view should have the best field of view, the least occlusion, and cover the main target objects in the scene.

[0024] Step 2: Generation and processing of depth point cloud.

[0025] The multi-view depth map is converted into multi-view depth point cloud data. Each pixel in the depth map is converted into a three-dimensional coordinate point according to its depth value to form point cloud data. The generated multi-view depth point cloud is processed, including point cloud alignment, denoising and sparse processing, to reduce data redundancy and improve computational efficiency. The alignment process ensures that the multi-view point cloud is represented in a unified coordinate system, so that data from different views can be effectively fused. The processed multi-view depth point cloud data is used as one of the inputs of the Deep Fusion Semantic Network (DFSN).

[0026] Step 3: Semantic recognition of single-view RGB images.

[0027] The RGB image of the main view is input into the Single View Semantic Network (SVSN). SVSN adopts a deep convolutional neural network (CNN) architecture and first extracts features from the RGB image. The convolution layer captures the spatial structural features of the image, and the multi-scale feature extraction layer captures the detail information in the image. Then SVSN decodes the extracted features layer by layer through a multi-level semantic decoder to generate a preliminary semantic label map. During the decoding process, the attention mechanism is used to dynamically focus on the key areas in the image and enhance the recognition ability of important targets. The output semantic label map of SVSN is stored as one of the inputs of the final fusion result.

[0028] Step 4: Deep fusion semantic recognition.

[0029] The preprocessed multi-view depth point cloud and the main view RGB image are input into the Deep Fusion Semantic Network (DFSN). DFSN first extracts features from the depth point cloud and RGB image independently. Then, the extracted depth point cloud and RGB image features are aligned. DFSN uses a feature alignment algorithm based on the attention mechanism to ensure that features of different modalities can be aligned in the same semantic space. The alignment process takes into account operations such as parallax compensation, feature rotation and scaling to overcome the parallax effect between multiple views.

[0030] The aligned features are sent to the feature fusion module for fusion. This module adopts an adaptive feature fusion strategy to dynamically adjust the weights of RGB image features and depth point cloud features to give full play to their respective advantages. The fused features are more robust and can better handle complex scenes and occlusion situations.

[0031] The fused features output by the feature fusion module are input into the semantic decoder to generate a fused semantic label map, which integrates the information of multi-view depth point cloud and main view RGB image, and has higher recognition accuracy than the single-modal method.

[0032] Step 5: Fusion of semantic recognition results.

[0033] The output results of the single view semantic network (SVSN) and the deep fusion semantic network (DFSN) are integrated through the semantic fusion module. The semantic fusion module uses weighted average combined with a rule-based fusion strategy to combine the semantic recognition results of the two networks. The specific fusion strategy is: under normal circumstances, the final prediction result is the weighted average result of the semantic recognition results of the two networks. However, in the case of missing input data, occlusion or light reflection, the semantic fusion module will correct the prediction result according to different situations, thereby improving the reliability of the final recognition. Generate the final semantic recognition result. This result combines multi-view and multi-modal information, has high robustness and accuracy, and can be applied to fields such as autonomous driving and robot navigation.

[0034] Step 6: Output and application.

[0035] The final semantic recognition results are output in the form of images or 3D models for subsequent analysis, processing or decision-making. The results can be used in application scenarios such as real-time environmental perception in autonomous driving systems, obstacle detection in drone navigation systems, and target tracking in intelligent monitoring systems.

[0036] Through the above steps, the depth map-assisted multi-view semantic recognition method provided by the present invention effectively solves the difficulties encountered by traditional methods in processing complex scenes, multi-view fusion, and the combination of depth maps and RGB images, and has high recognition accuracy and applicability.

[0037] The beneficial effects of the present invention are as follows:

[0038] Based on the existing technology, the present invention designs a multimodal fusion network based on deep learning by combining multi-view depth point clouds and main view RGB images, which significantly improves the accuracy and robustness of multi-view semantic recognition. Compared with traditional methods that only rely on information from a single modality, the present invention uses depth point clouds to capture three-dimensional geometric structure information, while extracting rich texture and semantic features through RGB images, and introduces weighted averaging combined with a rule-based dynamic adjustment strategy in the semantic fusion stage, thereby effectively solving the problem that traditional methods are susceptible to lighting changes, occlusions and noise interference in complex scenes. The present invention not only improves the comprehensiveness and accuracy of semantic recognition, but also enhances the model's ability to deal with multimodal data inconsistencies and input anomalies, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the network topology of the Deep Fusion Semantic Network (DFSN);

[0040] Figure 2 It is a schematic diagram of the network topology of Single View Semantic Network (SVSN);

[0041] Figure 3 It is a schematic diagram of data flow in the data fusion process. DETAILED DESCRIPTION

[0042] The present invention proposes a multi-view semantic recognition method based on depth map assistance. The method includes capturing depth maps and main view RGB images from multiple perspectives, using a deep fusion semantic network (DFSN) and a single view semantic network (SVSN) to process multimodal data respectively, and finally achieving high-precision semantic recognition by fusing the results of the two networks. The specific implementation method is as follows:

[0043] Step 1: Acquisition of multi-view depth map and main view RGB image;

[0044] 1.1 Multi-view data acquisition;

[0045] Use multiple cameras to synchronously capture depth images and main view RGB images from multiple perspectives of the same scene. Depth images can be obtained through technologies such as ToF (time of flight), structured light, or stereo vision. The main view RGB image usually selects the front view or the view with the least occlusion.

[0046] 1.2 Depth map preprocessing;

[0047] The preprocessing of the acquired depth image mainly includes the following steps:

[0048] (1) Denoising: Use filtering techniques (such as median filtering or bilateral filtering) to remove noise from the depth image.

[0049] (2) Depth value normalization: The depth value of the depth image is normalized to make it suitable for subsequent point cloud generation and feature extraction.

[0050] (3) Perspective alignment: Through geometric correction or other methods, ensure the alignment of depth images at different perspectives, thus laying the foundation for subsequent point cloud fusion.

[0051] 1.3 Selection of main view RGB image;

[0052] Select one of the collected multi-view RGB images as the main view input to SVSN. The main view should have the best field of view, the least occlusion, and cover the main target objects in the scene.

[0053] Step 2: Generation and processing of depth point cloud;

[0054] 2.1 Depth point cloud generation;

[0055] The preprocessed multi-view depth image is converted into depth point cloud data. The specific method is to triangulate each pixel in the depth image according to its depth value and the corresponding camera internal parameters (such as focal length, principal point coordinates, etc.) to generate the point coordinates (x, y, z) in three-dimensional space.

[0056] 2.2 Point cloud registration;

[0057] In order to make the point clouds generated under different viewing angles be represented in a unified coordinate system, the point clouds of each viewing angle are registered. The registration method can use the ICP (Iterative Closest Point) algorithm or perform rigid transformation alignment through known camera extrinsics.

[0058] 2.3 Point cloud sparseness and denoising;

[0059] Since the point cloud data generated by multiple perspectives may contain a lot of redundant information, it is necessary to perform sparse and denoise processing on the point cloud. Clustering methods such as DBSCAN or Voxel Grid can be used to simplify the point cloud and retain key feature points.

[0060] Step 3: Semantic recognition of single-view RGB images;

[0061] 3.1RGB image feature extraction;

[0062] The selected main view RGB image is input into the Single View Semantic Network (SVSN), which uses an encoder for feature extraction. The RGB image passes through multiple convolutional layers to extract spatial features of different scales, such as edges, textures, and shapes. Figure 2As shown in the figure, the encoder includes four encoder layers: the first encoder layer consists of a convolution block and a pooling layer; the second encoder layer includes two residual blocks; the third and fourth encoder layers consist of a residual block with strided convolution and a residual block. The convolution block includes a convolution layer, batch normalization, and ReLU activation function in sequence; the residual block consists of two convolution layers and a skip connection; the residual block with strided convolution consists of a strided convolution layer and a normal convolution layer combined with an internal skip connection. The main function of the strided convolution is to downsample and align the feature size, and the stride is set to 2.

[0063] As attached Figure 2 As shown in the figure, specifically, the input data is an RGB image, and the data is first subjected to preliminary feature extraction by the convolution block. The feature map is then downsampled through the pooling layer to reduce the resolution of the feature map, thereby reducing the computational complexity while retaining the main features. The residual block solves the problem of gradient vanishing in deep networks through skip connections and enhances feature learning capabilities. The residual block with stride convolution implements feature map downsampling through the convolution stride in the residual structure, further extracting features with more global semantics.

[0064] 3.2 Feature processing;

[0065] The output after encoder processing is processed through feature processing to enhance the accuracy of semantic expression. Figure 2 As shown in the figure, the feature processing part is composed of two residual blocks with dilated convolutions. In the residual block with dilated convolutions, the convolution kernel with holes filled after being stretched is used to replace the ordinary convolution kernel, which increases the receptive field of the convolution kernel without increasing the size of the convolution kernel and does not cause the spatial resolution of the feature map to decrease.

[0066] 3.3 Semantic decoding;

[0067] The processed features are input into the multi-level semantic decoder for semantic decoding. Figure 2 As shown in the figure, the multi-level semantic decoder includes four decoder layers, each of which receives the output of the corresponding encoder layer through a skip connection; the first encoder layer consists of two convolution blocks and a bilinear upsampling layer; the second and third encoder layers include a convolution block and a bilinear upsampling layer, and the last encoder layer includes two residual blocks. The bilinear upsampling layer is used to increase the resolution of the feature map and avoid blurring caused by interpolation. The skip connection directly connects the features of different stages of the encoder to the corresponding positions of the decoder, improves segmentation accuracy, and retains edge details. The multi-level semantic decoder decodes the features layer by layer, and each layer combines semantic information of different scales to finally generate a pixel-level semantic probability distribution matrix.

[0068] 3.4 Semantic label output;

[0069] In the SVSN network, the final output of the multi-level semantic decoder is the pixel-level semantic probability distribution matrix of the main view RGB image, which is generated by layer-by-layer decoding and feature fusion of the semantic decoder. The semantic probability distribution matrix is ​​finally generated by the output layer through the SoftMax function to generate the probability distribution of the category to which each pixel belongs, and further generates a semantic label map. This output is a semantic classification of each pixel in the main view RGB image, providing a preliminary pixel-level semantic prediction, laying the foundation for the subsequent multimodal fusion steps. The semantic label map assigns a semantic label to each pixel, such as "road", "building" or "pedestrian".

[0070] Step 4: Deep fusion semantic recognition;

[0071] 4.1 Multimodal feature extraction;

[0072] The processed multi-view depth point cloud data and multi-view RGB images are simultaneously input into the Deep Fusion Semantic Network (DFSN). Multi-view depth point cloud data refers to a set of point clouds generated by multi-view depth images, which integrates geometric information from different angles to make up for the occlusion or missing details that may exist in single-view point clouds. Unlike the geometric features and structural features extracted from point clouds, point cloud data itself emphasizes the complementarity of multi-view information, providing multi-angle feature support for fusion semantic analysis. By integrating the features of point clouds and RGB images, DFSN can realize the joint modeling of multimodal features and provide complete input for subsequent deep semantic analysis and semantic segmentation. DFSN first extracts features from these two types of input data respectively:

[0073] PointNet++ is used to extract geometric and structural features of multi-view depth point clouds. PointNet++ uses layered sampling and local aggregation methods to extract geometric and structural features of point clouds. By applying PointNet++ in local neighborhoods to extract geometric details and combining multi-scale and global features, the modeling capabilities of complex point cloud structures are significantly improved.

[0074] ResNet-50 is used to extract features from multi-view RGB images, and to extract color and texture features. ResNet-50 alleviates the gradient vanishing problem through residual connections, and can extract high-level semantic features while retaining low-level texture information. For the processing of multi-view RGB images, ResNet-50 with shared weights is used to extract features containing color and texture information for each RGB image. The features obtained from each image are spliced ​​and integrated to obtain multi-view features.

[0075] 4.2 Attention-based feature alignment;

[0076] After feature extraction, DFSN uses an attention-based feature alignment module to align the multi-view features of the depth point cloud and RGB image into the same semantic space. The alignment module considers factors such as parallax compensation, feature rotation and scaling to ensure that multimodal features can be effectively fused. The processing flow of the feature alignment module is as follows: (1) Construct query-key-value representation: The point cloud geometric and structural features obtained in step 4.1, collectively referred to as point cloud features, are used as queries (Query). The multi-view features obtained in step 4.1 are used as keys and values, and are mapped to a unified feature space through a fully connected layer with shared weights to ensure that the feature expression is consistent and aligned. (2) Calculate attention weights: The attention weights are calculated based on the similarity between the point cloud features and the multi-view features, and the weights are calculated dynamically. The weights are dynamically calculated using the relationship between the point cloud features and each view feature (i.e., the inner product similarity between the point cloud features and each multi-view feature), thereby obtaining the multi-view attention distribution for the point cloud. The calculation method is: the inner product similarity between the point cloud features and each multi-view feature is calculated after SoftMax normalization. (3) Fusion view features: Use the calculated attention weights to weight the multi-view features (Value) to generate multimodal fusion features aligned with the point cloud features. This process dynamically emphasizes the view features that are highly related to the point cloud geometry and structural information, while suppressing irrelevant features. (4) Output alignment features: The generated multimodal fusion features aligned with the point cloud features are input into the subsequent network for semantic analysis and classification. This alignment feature contains the geometry and structure information of the point cloud, as well as the detailed semantic features of the multi-view image, which can improve the robustness and accuracy of multimodal recognition.

[0077] 4.3 Feature fusion;

[0078] The point cloud features, multi-view attention distribution and multimodal fusion features are input into the feature fusion module to obtain the fusion features. The specific fusion process is as follows: (1) The point cloud features and multimodal fusion features are aligned and integrated and then input into the fully connected layer. The output of the fully connected layer is passed through the maximum pooling layer to obtain the point-single view fusion result. (2) The multimodal fusion features are used to obtain the three features with the highest correlation according to the multi-view attention distribution, which are recorded as v1, v2, and v3. v1, v2, and v3 are aligned and integrated with the point cloud feature p to form {p, v1}, {p,, v1, v2}, and {p, v1, v2, v3}, respectively, and input into the fully connected layer to obtain MF1, MF2, and MF3, respectively. MF1, MF2, and MF3 are averaged and pooled to obtain the point-multi-view fusion result. (3) The point-single view fusion result and the point-multi-view fusion result are aligned and integrated into the global feature. The fused global feature is more robust and can cope with complex scenes, such as lighting changes and occlusions.

[0079] 4.4 Semantic decoding;

[0080] The fused global features are passed through the semantic decoder of DFSN to generate a fused semantic label map. The semantic decoder contains the following structures: (1) The deconvolution module gradually restores the spatial resolution of the features so that the decoded feature map is consistent with the input data size. (2) The multi-scale fusion module combines the features at different levels in the decoding process to enhance the understanding of global and local semantics. (3) The convolution and activation functions further refine the decoded features and enhance the semantic representation ability of the output. (4) The generation of semantic labels uses Softmax to normalize the feature distribution of each pixel and generate a category probability distribution. Finally, the category with the highest probability is output as the prediction result of the semantic label map. The semantic decoder takes into account the contextual information of the scene to ensure high-precision recognition when processing complex scenes.

[0081] 4.5 Fusion semantic label output;

[0082] The final output of DFSN is the semantic segmentation result that combines multi-view depth point cloud and multi-view RGB image, thereby generating the final semantic label map.

[0083] Step 5: Fusion of semantic recognition results;

[0084] 5.1 Implementation of semantic fusion module;

[0085] After SVSN and DFSN generate semantic label maps respectively, the two results are combined through the semantic fusion module. The semantic fusion module uses weighted averaging combined with rule-based fusion strategy to weightedly fuse the results of DFSN and SVSN to obtain a weighted average result; the weights of weighting are: DFSN-0.7, SVSN-0.3. If the input data modality is missing, that is, the RGB image is missing or the number of valid points in the point cloud is less than 3000 points, the semantic recognition result of the non-missing data part is adopted. If the geometric features are incomplete due to occlusion or noise, the weights are adjusted to: DFSN-0.3, SVSN-0.7. If there is strong light or blur resulting in texture loss, the weights are adjusted to: DFSN-0.1, SVSN-0.9.

[0086] 5.2 Generation of final semantic recognition results;

[0087] After fusion, the prediction result with the highest probability will be taken as the final semantic segmentation result of the scene. This result combines multi-view and multi-modal information, has high robustness and accuracy, and is suitable for complex scenarios such as autonomous driving and robot navigation.

[0088] Step 6: Output and application of results;

[0089] 6.1 Result storage and output;

[0090] The final semantic recognition results are output in the form of images, 3D models or other suitable forms. These results can be stored in the local device or transmitted to the cloud via the network for further processing.

[0091] 6.2 Application scenarios;

[0092] The semantic segmentation results can be widely used in many fields:

[0093] (1) Autonomous driving: Real-time detection and identification of roads, pedestrians, vehicles and other elements to assist the decision-making of the autonomous driving system.

[0094] (2) Robot navigation: Provides environmental perception for autonomous mobile robots to avoid obstacles and plan paths.

[0095] (3) Intelligent monitoring: Identify abnormal behavior or specific targets in surveillance videos to improve safety.

[0096] Through the above steps, the depth map-assisted multi-view semantic recognition method provided by the present invention shows high recognition accuracy and practicality when processing complex scenes, multi-view fusion, and the combination of depth maps and RGB images.

[0097] The comparative test settings for this experiment are as follows:

[0098] The Princeton ModelNet40 dataset contains 12,311 3D CAD models of 40 categories. The point cloud is sampled from the surface of the CAD model, and the view data is captured by the camera. The experiment compares the voxel-based methods: 3D ShapeNets, VoxlNet. Deep learning models: MVCNN, PointNet, PointNet++.

[0099] The results are shown in the following table:

[0101]

[0102] Our method achieved a classification accuracy of 93.6% and an average retrieval precision of 90.5%, which is the best among the comparison methods.

[0103] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0104] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A multi-view semantic recognition method based on depth map assistance, characterized in that: This is accomplished by following these steps: Step 1: Acquisition of multi-view depth map and main view RGB image; Obtain a multi-view depth map and a main view RGB image of the target scene; pre-process the obtained multi-view depth map data to ensure the quality of the depth map data; select an RGB image of a main view, i.e., the main view, as the input image for single-view semantic recognition. The main view should have the best field of view, the least occlusion, and cover the main target objects in the scene; Step 2: Generation and processing of depth point cloud; The multi-view depth map is converted into multi-view depth point cloud data; each pixel in the depth map is converted into a three-dimensional coordinate point according to its depth value to form point cloud data; the generated multi-view depth point cloud is processed, including point cloud alignment, noise reduction and sparse processing; The processed multi-view depth point cloud data is used as one of the inputs of the deep fusion semantic network DFSN; Step 3: semantic recognition of single-view RGB images; The RGB image of the main view is input into the single view semantic network SVSN; SVSN adopts a deep convolutional neural network architecture to first extract features from the RGB image; Then SVSN decodes the extracted features layer by layer through a multi-level semantic decoder to generate a preliminary semantic label map; the output semantic label map of SVSN is stored as one of the inputs of the final fusion result; Step 4: Deep fusion semantic recognition; The preprocessed multi-view depth point cloud and the main view RGB image are input into the deep fusion semantic network DFSN. DFSN first extracts features from the depth point cloud and RGB image independently. Then, the extracted depth point cloud and RGB image features are aligned. DFSN uses a feature alignment algorithm based on the attention mechanism to ensure that features of different modalities can be aligned in the same semantic space. The aligned features are sent to the feature fusion module for fusion; the module adopts an adaptive feature fusion strategy to dynamically adjust the weights of RGB image features and depth point cloud features to give full play to their respective advantages; the fused features output by the feature fusion module are input into the semantic decoder to generate a fused semantic label map; Step 5: Fusion of semantic recognition results; The output results of the single-view semantic network SVSN and the deep fusion semantic network DFSN are integrated through the semantic fusion module; the semantic fusion module adopts weighted average combined with rule-based fusion strategy to combine the semantic recognition results of the two networks; the specific fusion strategy is: under normal circumstances, the final prediction result is the weighted average result of the semantic recognition results of the two networks; but in the case of missing input data, occlusion or light reflection, the semantic fusion module will correct the prediction result according to different situations, thereby improving the reliability of the final recognition; Generate the final semantic recognition results; Step 6: Output and application; The final semantic recognition results are output in the form of images or three-dimensional models for subsequent analysis, processing or decision-making; the results can be used for real-time environmental perception in autonomous driving systems, obstacle detection in drone navigation systems, or target tracking in intelligent monitoring systems.

2. The depth map-assisted multi-view semantic recognition method according to claim 1, characterized in that: Step 2: 2.1 Depth point cloud generation; Convert the preprocessed multi-view depth image into depth point cloud data; the specific method is to triangulate each pixel in the depth image according to its depth value and the corresponding camera internal parameter to generate the point coordinates (x, y, z) in three-dimensional space; 2.2 Point cloud registration; In order to make the point clouds generated under different perspectives be represented in a unified coordinate system, the point clouds of each perspective are registered; the registration method uses the ICP algorithm or rigid transformation alignment through known camera extrinsics; 2.3 Point cloud sparseness and denoising; The point cloud is simplified using clustering methods to retain key feature points.

3. The depth map-assisted multi-view semantic recognition method according to claim 1, characterized in that: Step 3: 3.1RGB image feature extraction; The selected main-view RGB image is input into the single-view semantic network SVSN, which uses an encoder for feature extraction; the RGB image is passed through multiple convolutional layers to extract spatial features of different scales; the encoder includes four encoder layers: the first encoder layer consists of a convolution block and a pooling layer; the second encoder layer includes two residual blocks; the third and fourth encoder layers consist of a residual block with strided convolution and a residual block; the convolution block includes a convolution layer, batch normalization and ReLU activation function in sequence; the residual block consists of two convolution layers and a skip connection; the residual block with strided convolution consists of a strided convolution layer and a normal convolution layer combined with an internal skip connection; the main function of the strided convolution is to downsample and align the feature size, and the stride is set to 2; 3.2 Feature processing; The output processed by the encoder is used to enhance the accuracy of semantic expression through the feature processing part. The feature processing part consists of two residual blocks with dilated convolutions. In the residual blocks with dilated convolutions, the convolution kernels that are stretched and filled with holes are used to replace the ordinary convolution kernels. 3.3 Semantic decoding; The processed features are input into the multi-level semantic decoder for semantic decoding; the multi-level semantic decoder includes four decoder layers, each of which receives the output of the corresponding encoder layer through a jump connection; the first encoder layer consists of two convolution blocks and a bilinear upsampling layer; the second and third encoder layers include a convolution block and a bilinear upsampling layer, and the last encoder layer includes two residual blocks; the bilinear upsampling layer is used to increase the resolution of the feature map to avoid blurring caused by interpolation; the jump connection directly connects the features of different stages of the encoder to the corresponding positions of the decoder to improve segmentation accuracy and retain edge details; the multi-level semantic decoder decodes the features layer by layer, and each layer combines semantic information of different scales to finally generate a pixel-level semantic probability distribution matrix; 3.4 Semantic label output; The semantic probability distribution matrix is ​​finally used by the output layer to generate the probability distribution of the category to which each pixel belongs through the SoftMax function, and further generates a semantic label map; the semantic label map assigns a semantic label to each pixel.

4. The depth map-assisted multi-view semantic recognition method according to claim 1, characterized in that: Step 4: 4.1 Multimodal feature extraction; The processed multi-view depth point cloud data and multi-view RGB images are simultaneously input into the deep fusion semantic network DFSN; DFSN first extracts features from the two input data respectively: pass PointNet++ extracts geometric and structural features of multi-view depth point clouds; ResNet-50 is used to extract features from multi-view RGB images, and color and texture features are extracted. For the processing of multi-view RGB images, ResNet-50 with shared weights is used to extract features containing color and texture information from each RGB image. The features obtained from each image are spliced ​​and integrated to obtain multi-view features. 4.2 Attention-based feature alignment; After feature extraction, DFSN uses an attention-based feature alignment module to align the multi-view features of the depth point cloud and RGB image into the same semantic space. The alignment module considers factors such as parallax compensation, feature rotation and scaling to ensure that multimodal features can be effectively fused. The processing flow of the feature alignment module is as follows: (1) Constructing a query-key-value representation: The point cloud geometric and structural features obtained in step 4.1, collectively referred to as point cloud features, are used as the query Query; the multi-view features obtained in step 4.1 are used as keys and values, and are mapped to a unified feature space through a fully connected layer with shared weights to ensure that the feature expression is consistent and aligned. (2) Calculating attention weights: Calculate attention weights based on the similarity between point cloud features and multi-view features, and dynamically calculate weights; dynamically calculate weights using the relationship between point cloud features and each view feature, thereby obtaining multi-view attention distribution for the point cloud; The calculation method is as follows: the inner product similarity between the point cloud feature and each multi-view feature is calculated after SoftMax normalization; (3) Fusion view feature: Use the calculated attention weight to weight the multi-view features to generate a multimodal fusion feature aligned with the point cloud feature; (4) Output alignment feature: Input the generated multimodal fusion feature aligned with the point cloud feature into the subsequent network for semantic analysis and classification; 4.3 Feature fusion; The point cloud features, multi-view attention distribution and multimodal fusion features are input into the feature fusion module to obtain fusion features. The specific fusion process is as follows: (1) The point cloud features and multimodal fusion features are aligned and integrated and then input into the fully connected layer. The output results of the fully connected layer are passed through the maximum pooling layer to obtain the point-single view fusion result. (2) The multimodal fusion features are used to obtain the three features with the highest correlation according to the multi-view attention distribution, which are recorded as v1, v2, and v3. v1, v2, and v3 are aligned and integrated with the point cloud feature p to form {p, v1}, {p,,v1,v2}, and {p,v1,v2,v3}, respectively, and input into the fully connected layer to obtain MF1, MF2, and MF3 respectively. MF1, MF2, and MF3 are subjected to the average pooling operation to obtain the point-multi-view fusion result. (3) The point-single view fusion result and the point-multi-view fusion result are aligned and integrated into the global feature. 4.4 Semantic decoding; The fused global features are passed through the semantic decoder of DFSN to generate a fused semantic label map; the semantic decoder contains the following structures: (1) a deconvolution module that gradually restores the spatial resolution of the features so that the decoded feature map is consistent with the input data size; (2) a multi-scale fusion module that combines features at different levels in the decoding process to enhance the understanding of global and local semantics; (3) convolution and activation functions that further refine the decoded features and enhance the semantic representation capability of the output; (4) semantic label generation that uses Softmax to normalize the feature distribution of each pixel and generate a category probability distribution; and finally outputs the category with the highest probability as the prediction result of the semantic label map. 4.5 Fusion semantic label output; The final output of DFSN is a semantic segmentation result that combines multi-view depth point cloud and multi-view RGB image, thereby generating the final semantic label map.

5. The depth map-assisted multi-view semantic recognition method according to any one of claims 1 to 4, characterized in that: Step 5: 5.1 Implementation of semantic fusion module; After SVSN and DFSN generate semantic label maps respectively, the two results are combined through the semantic fusion module; the semantic fusion module adopts weighted averaging combined with rule-based fusion strategy to perform weighted fusion on the results of DFSN and SVSN to obtain the weighted average result; the weights of weighting are: DFSN-0.7, SVSN-0.3; if the input data modality is missing, that is, the RGB image is missing or the number of valid points in the point cloud is less than 3000 points, the semantic recognition result of the non-missing data part is adopted; If the geometric features are incomplete due to occlusion or noise, the weights are adjusted to: DFSN-0.3, SVSN-0.7; if there is strong light or blur resulting in texture loss, the weights are adjusted to: DFSN-0.1, SVSN-0.9; 5.2 Generation of final semantic recognition results; After fusion, the prediction result with the highest probability will be taken as the final semantic segmentation result of the scene.

Images