A multi-view semantic recognition method based on depth map assistance

Abstract

The present invention discloses a multi-view semantic recognition method based on the assistance of depth map. First, a multi-view depth map and a main view RGB image are obtained; then, a depth point cloud is generated and processed; semantic recognition of a single view RGB image and a depth fusion semantic recognition are performed respectively; finally, the output results of the single view semantic network SVSN and the depth fusion semantic network DFSN are integrated through a semantic fusion module to generate a final semantic recognition result. The present invention uses a depth point cloud to capture three-dimensional geometric structure information, while extracting rich texture and semantic features through RGB images, and introduces a weighted average combined with a rule-based dynamic adjustment strategy in the semantic fusion stage, thereby effectively solving the problem that traditional methods are susceptible to illumination changes, occlusion and noise interference in complex scenes. The present invention not only improves the comprehensiveness and accuracy of semantic recognition, but also enhances the model's ability to cope with multimodal data inconsistencies and input anomalies, and has broad application prospects.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and deep learning, and in particular to the field of multi-view semantic recognition. Specifically, the present invention relates to a multi-view semantic recognition method assisted by a depth map. This method combines the network architecture of a deep fusion semantic network with a single-view semantic network to achieve efficient fusion recognition of multi-view depth point clouds and primary view RGB images. Background Art

[0002] With the advancement of computer vision technology, semantic recognition is increasingly being used in autonomous driving, drone navigation, augmented reality (AR), and other fields. The goal of semantic recognition is to assign semantic labels, such as road, vehicle, and pedestrian, to each pixel in a scene. Traditional semantic recognition methods typically rely on single-view RGB images, but this approach often faces challenges when dealing with complex scenes. For example, factors such as object occlusion, parallax between multiple viewpoints, and lighting variations can all affect recognition accuracy.

[0003] To address these issues, researchers have proposed multi-view semantic recognition methods that combine information from multiple perspectives in the hope of improving the robustness and accuracy of recognition. However, multi-view methods require processing large amounts of high-dimensional data, and effectively fusing multi-view information remains a challenge. In particular, when it comes to fusing data from different modalities, such as RGB images and depth maps, traditional methods often face the following problems:

[0004] Difficulty in feature alignment: Parallax exists between multiple views, resulting in different representations of the same object from different perspectives. This discrepancy is particularly pronounced for depth maps and RGB images. Effectively aligning features across multi-view, multi-modal data to achieve meaningful fusion is a key issue.

[0005] Imperfect fusion strategy: Traditional methods mostly use simple feature splicing or weighted averaging for modal fusion, but this approach cannot fully utilize the complementary advantages of each modal data and may lead to information loss or redundancy.

[0006] High computing resource requirements: Processing multi-view and multi-modal data usually requires high computing resources. How to reduce computational complexity while ensuring recognition accuracy is also an important research direction.

[0007] Therefore, in recent years, some researchers have proposed multimodal fusion networks that combine deep learning. For example, PointFusion uses the features of point clouds and RGB images, fuses the two through a simple connection layer, and ultimately achieves three-dimensional object detection. MV3D projects point clouds to multiple perspectives, such as bird's-eye views and front views, and combines them with RGB images for multimodal feature extraction and fusion for three-dimensional target detection. Frustum PointNets crops the corresponding point cloud blocks using image detection frames, and uses PointNet to process the cropped point clouds to achieve 3D target detection. However, most of these networks have problems such as difficulty in feature alignment, strong dependence on data quality, and insufficient generalization ability.

[0008] In summary, traditional multi-view semantic recognition methods face significant challenges in feature alignment, modal fusion and computational complexity. The present invention aims to solve technical problems existing in existing multi-view semantic recognition methods, such as insufficient modal feature fusion, insufficient information redundancy and noise processing, deep information sparsity, and insufficient real-time and consistency modeling. By designing a deep fusion semantic network (DFSN) and a single view semantic network (SVSN), these problems are effectively solved, and efficient fusion and high-precision semantic recognition of multi-view and multi-modal data are achieved. DFSN is responsible for processing the fusion of multi-perspective depth point clouds and RGB images, while SVSN focuses on the semantic recognition of the main view RGB image. The final recognition result is a combination of the outputs of these two networks, achieving higher recognition accuracy and robustness. Summary of the Invention

[0009] To address the shortcomings of existing technologies, the present invention provides a multi-view semantic recognition method assisted by depth maps. This method achieves high-precision semantic recognition by fusing multi-view depth point clouds with the RGB image information of the main view. To achieve this goal, the present invention designs a deep fusion algorithm consisting of two main modules: the Depth Fusion Semantic Network (DFSN) module and the Single View Semantic Network (SVSN) module.

[0010] The Deep Fusion Semantic Network (DFSN) is one of the core networks in this paper, which is specifically designed to handle the fusion of multimodal data. It receives multi-view depth point clouds and main view RGB images and completes fusion and recognition through the following steps:

[0011] Multimodal feature extraction: DFSN first performs independent feature extraction on the input depth point cloud and RGB image, and uses convolutional neural network (CNN) to extract spatial features and semantic features of depth point cloud and RGB image respectively.

[0012] Feature alignment: After extracting features from the multi-view depth point cloud and the primary view RGB image, DFSN uses a feature alignment module to align these features in the spatial dimension. This module uses an attention-based alignment algorithm to ensure that features from different modalities are effectively integrated within the same semantic space.

[0013] Feature Fusion: Aligned features are fused in the Feature Fusion module. This module uses an improved feature fusion strategy to fuse depth features with RGB features at multiple levels and scales to enhance the model's ability to recognize objects in complex scenes.

[0014] Semantic Decoding: The fused features are passed through a semantic decoder to generate semantic labels for the multi-view depth point cloud and RGB image. DFSN also incorporates scene context information during this process to improve the recognition accuracy of occluded objects and boundary areas.

[0015] The Single View Semantic Network (SVSN) is used to perform preliminary semantic recognition on the main view RGB image. The network mainly includes the following modules:

[0016] Image feature extraction: SVSN uses a deep convolutional neural network to extract features from the input RGB image and generate a series of image features with different scales.

[0017] Multi-level semantic decoding: SVSN decodes the extracted image features layer by layer through a multi-level semantic decoder and outputs the semantic label of each pixel in the image.

[0018] Attention mechanism enhancement: During the feature decoding process, SVSN integrates an attention mechanism to enhance the recognition ability of small objects in complex scenes while reducing sensitivity to background noise.

[0019] Fusion Recognition Results: After the SVSN and DFSN complete recognition of the RGB image and depth point cloud, respectively, the semantic fusion module fuses these two recognition results. Using a weighted average or rule-based strategy, the fusion module combines the single-view recognition results of the SVSN and the multimodal fusion results of the DFSN to generate the final global semantic recognition result. This fusion strategy not only improves recognition accuracy but also enhances the robustness of the semantic fusion module by leveraging the strong semantic information of the RGB image to supplement depth information when it is missing or noisy, thereby enhancing the depth information.

[0020] Through the above steps, the depth map-assisted multi-view semantic recognition method of the present invention effectively combines the information of multi-view depth point cloud and main view RGB image, achieving higher recognition accuracy and robustness.

[0021] A multi-view semantic recognition method based on depth map assistance is implemented by the following steps:

[0022] Step 1: Acquire multi-view depth maps and main view RGB images.

[0023] Acquire multi-view depth maps and a main view RGB image of the target scene from multiple cameras or sensors. These cameras can simultaneously capture images and depth data from different viewpoints of the same scene. Preprocess the acquired multi-view depth map data, including denoising and depth normalization, to ensure depth map data quality. Select an RGB image from a main view, or main view, as the input image for single-view semantic recognition. The main view should have the best field of view, minimal occlusion, and cover the main target objects in the scene.

[0024] Step 2: Generation and processing of depth point cloud.

[0025] The multi-view depth map is converted into multi-view depth point cloud data. Each pixel in the depth map is converted into a three-dimensional coordinate point based on its depth value, forming point cloud data. The generated multi-view depth point cloud is processed, including point cloud alignment, noise reduction, and sparsification, to reduce data redundancy and improve computational efficiency. The alignment process ensures that the multi-view point cloud is represented in a unified coordinate system, enabling effective fusion of data from different viewpoints. The processed multi-view depth point cloud data is used as one of the inputs to the Deep Fusion Semantic Network (DFSN).

[0026] Step 3: Semantic recognition of single-view RGB images.

[0027] The RGB image from the primary view is input into the Single View Semantic Network (SVSN). The SVSN employs a deep convolutional neural network (CNN) architecture and first extracts features from the RGB image. The convolutional layer captures the spatial structure of the image, while the multi-scale feature extraction layer captures detailed information within the image. The SVSN then decodes the extracted features layer by layer using a multi-layer semantic decoder to generate a preliminary semantic label map. During the decoding process, an attention mechanism is used to dynamically focus on key areas in the image, enhancing the recognition of important objects. The output semantic label map of the SVSN is stored as one of the inputs to the final fusion result.

[0028] Step 4: Deep fusion semantic recognition.

[0029] The preprocessed multi-view depth point cloud and the primary-view RGB image are fed into the Deep Fusion Semantic Network (DFSN). DFSN first performs independent feature extraction on the depth point cloud and RGB image. It then aligns the extracted depth point cloud and RGB image features. DFSN uses an attention-based feature alignment algorithm to ensure that features from different modalities are aligned within the same semantic space. This alignment process takes into account parallax compensation, feature rotation, and scaling to overcome the effects of parallax between multiple views.

[0030] The aligned features are then fed into the feature fusion module for fusion. This module uses an adaptive feature fusion strategy to dynamically adjust the weights of RGB image features and depth point cloud features to leverage their respective strengths. The fused features are more robust and can better handle complex scenes and occlusions.

[0031] The fused features output by the feature fusion module are input into the semantic decoder to generate a fused semantic label map. This label map integrates the information of multi-view depth point cloud and main view RGB image, and the recognition accuracy is higher than that of single-modal method.

[0032] Step 5: Fusion of semantic recognition results.

[0033] The output results of the Single View Semantic Network (SVSN) and the Deep Fusion Semantic Network (DFSN) are synthesized through the semantic fusion module. The semantic fusion module uses a weighted average combined with a rule-based fusion strategy to combine the semantic recognition results of the two networks. The specific fusion strategy is: under normal circumstances, the final prediction result is the weighted average of the semantic recognition results of the two networks. However, in the case of missing input data, occlusion or light reflection, the semantic fusion module will modify the prediction result according to different situations, thereby improving the reliability of the final recognition. The final semantic recognition result is generated. This result combines multi-view and multi-modal information, has high robustness and accuracy, and can be applied to fields such as autonomous driving and robot navigation.

[0034] Step 6: Output and application.

[0035] The resulting semantic recognition results are output as images or 3D models for subsequent analysis, processing, or decision-making. These results can be used in applications such as real-time environmental perception in autonomous driving systems, obstacle detection in drone navigation systems, and target tracking in intelligent monitoring systems.

[0036] Through the above steps, the depth map-assisted multi-view semantic recognition method provided by the present invention effectively solves the difficulties encountered by traditional methods in processing complex scenes, multi-view fusion, and the combination of depth maps and RGB images, and has high recognition accuracy and applicability.

[0037] The beneficial effects of the present invention are as follows:

[0038] Based on the existing technology, the present invention designs a multimodal fusion network based on deep learning by combining multi-view depth point clouds and main view RGB images, which significantly improves the accuracy and robustness of multi-view semantic recognition. Compared with traditional methods that only rely on information from a single modality, the present invention uses depth point clouds to capture three-dimensional geometric structure information, while extracting rich texture and semantic features through RGB images, and introduces weighted averaging combined with a rule-based dynamic adjustment strategy in the semantic fusion stage, thereby effectively solving the problem that traditional methods are susceptible to lighting changes, occlusion and noise interference in complex scenes. The present invention not only improves the comprehensiveness and accuracy of semantic recognition, but also enhances the model's ability to deal with multimodal data inconsistencies and input anomalies, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the network topology of Deep Fusion Semantic Network (DFSN);

[0040] Figure 2 It is a schematic diagram of the network topology of the Single View Semantic Network (SVSN);

[0041] Figure 3 This is a schematic diagram of data flow in the data fusion process. DETAILED DESCRIPTION

[0042] This paper proposes a multi-view semantic recognition method based on depth map assistance. The method involves capturing depth maps and main view RGB images from multiple perspectives, using a Deep Fusion Semantic Network (DFSN) and a Single View Semantic Network (SVSN) to process the multimodal data separately, and finally achieving high-precision semantic recognition by fusing the results of the two networks. The specific implementation method is as follows:

[0043] Step 1: Acquisition of multi-view depth maps and main view RGB images;

[0044] 1.1 Multi-view data acquisition;

[0045] Multiple cameras are used to synchronously capture depth images and main-view RGB images of the same scene from multiple perspectives. Depth images can be acquired using technologies such as Time of Flight (ToF), structured light, or stereo vision. The main-view RGB image is typically obtained from the orthographic perspective or the perspective with the least occlusion.

[0046] 1.2 Depth map preprocessing;

[0047] The preprocessing of the collected depth image mainly includes the following steps:

[0048] (1) Denoising: Use filtering techniques (such as median filtering or bilateral filtering) to remove noise from the depth image.

[0049] (2) Depth value normalization: The depth value of the depth image is normalized to make it suitable for subsequent point cloud generation and feature extraction.

[0050] (3) Perspective alignment: Through geometric correction or other methods, ensure the alignment of depth images under different perspectives, thus laying the foundation for subsequent point cloud fusion.

[0051] 1.3 Selection of main view RGB image;

[0052] From the collected multi-view RGB images, one is selected as the main view RGB image input to SVSN. The main view should have the best field of view, the least occlusion, and cover the main target objects in the scene.

[0053] Step 2: Generation and processing of depth point cloud;

[0054] 2.1 Depth point cloud generation;

[0055] The pre-processed multi-view depth image is converted into depth point cloud data. The specific method is to triangulate each pixel in the depth image based on its depth value and the corresponding camera intrinsic parameters (such as focal length, principal point coordinates, etc.) to generate the point coordinates (x, y, z) in three-dimensional space.

[0056] 2.2 Point cloud registration;

[0057] In order to represent the point clouds generated from different viewpoints in a unified coordinate system, the point clouds from each viewpoint are registered. The registration method can use the ICP (Iterative Closest Point) algorithm or rigid transformation alignment using known camera extrinsics.

[0058] 2.3 Point cloud sparseness and denoising;

[0059] Since point cloud data generated from multiple viewpoints may contain a large amount of redundant information, it is necessary to perform sparse and denoise processing on the point cloud. Clustering methods such as DBSCAN or Voxel Grid can be used to simplify the point cloud and retain key feature points.

[0060] Step 3: Semantic recognition of single-view RGB images;

[0061] 3.1RGB image feature extraction;

[0062] The selected main view RGB image is input into the Single View Semantic Network (SVSN), which uses an encoder for feature extraction. The RGB image passes through multiple convolutional layers to extract spatial features of different scales, such as edges, textures, and shapes. Figure 2As shown in the figure, the encoder includes four encoder layers: the first encoder layer consists of a convolutional block and a pooling layer; the second encoder layer includes two residual blocks; the third and fourth encoder layers consist of a residual block with strided convolution and a residual block. The convolutional block includes a convolutional layer, batch normalization, and a ReLU activation function; the residual block consists of two convolutional layers and skip connections; and the residual block with strided convolution consists of a strided convolution layer and a normal convolution layer with internal skip connections. The strided convolution is mainly used for downsampling and feature alignment, with a stride of 2.

[0063] As attached Figure 2 Specifically, the input data is an RGB image. It first passes through a convolutional block to perform preliminary feature extraction. The feature map is then downsampled through a pooling layer, reducing its resolution and computational complexity while retaining key features. Residual blocks use skip connections to address the vanishing gradient problem in deep networks and enhance feature learning. Residual blocks with strided convolutions downsample the feature map by adjusting the convolution stride within the residual structure, further extracting features with more global semantics.

[0064] 3.2 Feature processing;

[0065] The output after encoder processing is processed by feature processing to enhance the accuracy of semantic expression. Figure 2 As shown in the figure, the feature processing section consists of two residual blocks with dilated convolutions. In the residual blocks with dilated convolutions, a convolution kernel that has been stretched and then filled with holes is used instead of a normal convolution kernel. This increases the receptive field of the convolution kernel without increasing its size and without reducing the spatial resolution of the feature map.

[0066] 3.3 Semantic decoding;

[0067] The processed features are input into the multi-level semantic decoder for semantic decoding. Figure 2 As shown, the multi-level semantic decoder consists of four decoder layers, each of which receives the output of the corresponding encoder layer via skip connections. The first encoder layer consists of two convolutional blocks and a bilinear upsampling layer; the second and third encoder layers each consist of a convolutional block and a bilinear upsampling layer, and the final encoder layer includes two residual blocks. The bilinear upsampling layer is used to increase the resolution of the feature map and avoid blurring caused by interpolation. Skip connections directly connect features from different encoder stages to corresponding positions in the decoder, improving segmentation accuracy and preserving edge details. The multi-level semantic decoder decodes features layer by layer, combining semantic information at different scales at each layer to ultimately generate a pixel-level semantic probability distribution matrix.

[0068] 3.4 Semantic label output;

[0069] In the SVSN network, the final output of the multi-layer semantic decoder is a pixel-level semantic probability distribution matrix for the primary-view RGB image. This result is generated through layer-by-layer decoding and feature fusion by the semantic decoder. This semantic probability distribution matrix is ​​ultimately used by the output layer to generate a probability distribution for each pixel's category through a SoftMax function, which then generates a semantic label map. This output is a semantic classification of each pixel in the primary-view RGB image, providing preliminary pixel-level semantic predictions and laying the foundation for subsequent multimodal fusion steps. This semantic label map assigns a semantic label to each pixel, such as "road," "building," or "pedestrian."

[0070] Step 4: Deep fusion semantic recognition;

[0071] 4.1 Multimodal feature extraction;

[0072] The processed multi-view depth point cloud data and multi-view RGB images are simultaneously input into the Deep Fusion Semantic Network (DFSN). Multi-view depth point cloud data refers to a set of point clouds generated by multi-view depth images, which integrates geometric information from different angles to make up for the occlusion or missing details that may exist in single-view point clouds. Unlike the geometric features and structural features extracted from point clouds, point cloud data itself emphasizes the complementarity of multi-view information, providing multi-angle feature support for fusion semantic analysis. By integrating the features of point clouds and RGB images, DFSN can achieve joint modeling of multimodal features, providing complete input for subsequent deep semantic analysis and semantic segmentation. DFSN first extracts features from these two types of input data separately:

[0073] PointNet++ extracts geometric and structural features from multi-view depth point clouds. PointNet++ uses layered sampling and local aggregation to extract geometric and structural features from point clouds. By applying PointNet++ to local neighborhoods to extract geometric details while combining multi-scale and global features, it significantly improves the ability to model complex point cloud structures.

[0074] ResNet-50 is used to extract features from multi-view RGB images, extracting color and texture features. ResNet-50 uses residual connections to mitigate the vanishing gradient problem, extracting high-level semantic features while preserving low-level texture information. For multi-view RGB image processing, a ResNet-50 with shared weights is used to extract features containing color and texture information for each RGB image. The features obtained from each image are concatenated and integrated to generate multi-view features.

[0075] 4.2 Attention-based feature alignment;

[0076] After feature extraction, DFSN uses an attention-based feature alignment module to align the multi-view features of the depth point cloud and RGB image into the same semantic space. The alignment module takes into account factors such as parallax compensation, feature rotation and scaling to ensure that multimodal features can be effectively fused. The processing flow of the feature alignment module is as follows: (1) Construct query-key-value representation: The point cloud geometric and structural features obtained in step 4.1, collectively referred to as point cloud features, are used as queries (Query). The multi-view features obtained in step 4.1 are used as keys and values, and are mapped to a unified feature space through a fully connected layer with shared weights to ensure consistency and alignment of feature expressions. (2) Calculate attention weights: The attention weights are calculated based on the similarity between point cloud features and multi-view features, and the weights are calculated dynamically. The weights are dynamically calculated using the relationship between point cloud features and each view feature (i.e., the inner product similarity between point cloud features and each multi-view feature), thereby obtaining the multi-view attention distribution for the point cloud. The calculation method is: the inner product similarity between point cloud features and each multi-view feature is calculated after SoftMax normalization. (3) Fusion view features: Use the calculated attention weights to weight the multi-view features (Value) to generate multimodal fusion features aligned with the point cloud features. This process dynamically emphasizes view features that are highly correlated with the point cloud geometry and structural information, while suppressing irrelevant features. (4) Output alignment features: The generated multimodal fusion features aligned with the point cloud features are input into the subsequent network for semantic analysis and classification. This alignment feature contains the geometric and structural information of the point cloud, as well as the detailed semantic features of the multi-view image, which can improve the robustness and accuracy of multimodal recognition.

[0077] 4.3 Feature Fusion;

[0078] The point cloud features, multi-view attention distribution and multimodal fusion features are input into the feature fusion module to obtain the fusion features. The specific fusion process is as follows: (1) The point cloud features and multimodal fusion features are aligned and integrated and then input into the fully connected layer. The output of the fully connected layer is passed through the maximum pooling layer to obtain the point-single view fusion result. (2) The multimodal fusion features are used to obtain the three features with the highest correlation according to the multi-view attention distribution, which are recorded as v1, v2, and v3. v1, v2, and v3 are aligned and integrated with the point cloud feature p to form {p, v1}, {p,, v1, v2}, and {p, v1, v2, v3} respectively, and input into the fully connected layer to obtain MF1, MF2, and MF3 respectively. MF1, MF2, and MF3 are averaged and pooled to obtain the point-multi-view fusion result. (3) The point-single view fusion result and the point-multi-view fusion result are aligned and integrated into the global feature. The fused global feature is more robust and can cope with complex scenes such as lighting changes and occlusions.

[0079] 4.4 Semantic decoding;

[0080] The fused global features are passed through the semantic decoder of DFSN to generate a fused semantic label map. The semantic decoder contains the following structures: (1) Deconvolution module, which gradually restores the spatial resolution of the features so that the decoded feature map is consistent with the input data size. (2) Multi-scale fusion module, which combines features at different levels in the decoding process to enhance the understanding of global and local semantics. (3) Convolution and activation functions, which further refine the decoded features and enhance the semantic representation ability of the output. (4) Semantic label generation, using Softmax to normalize the feature distribution of each pixel and generate a category probability distribution. Finally, the category with the highest probability is output as the prediction result of the semantic label map. The semantic decoder takes into account the contextual information of the scene to ensure high-precision recognition when processing complex scenes.

[0081] 4.5 Fusion semantic label output;

[0082] The final output of DFSN is the semantic segmentation result that combines multi-view depth point cloud and multi-view RGB image, thereby generating the final semantic label map.

[0083] Step 5: Fusion of semantic recognition results;

[0084] 5.1 Implementation of semantic fusion module;

[0085] After SVSN and DFSN generate semantic label maps, the two results are combined using a semantic fusion module. This module uses a weighted average combined with a rule-based fusion strategy to combine the DFSN and SVSN results, yielding a weighted average result. Weights are set to -0.7 for DFSN and -0.3 for SVSN. If the input data modality is missing (i.e., the RGB image is missing or the point cloud has fewer than 3000 valid points), the semantic recognition results from the remaining data are used. If geometric features are incomplete due to occlusion or noise, the weights are adjusted to -0.3 for DFSN and -0.7 for SVSN. If texture is missing due to strong light or blur, the weights are adjusted to -0.1 for DFSN and -0.9 for SVSN.

[0086] 5.2 Generating the final semantic recognition results;

[0087] After fusion, the prediction with the highest probability is selected as the final semantic segmentation result for the scene. This result combines multi-view and multi-modal information, with high robustness and accuracy, making it suitable for complex scenarios such as autonomous driving and robot navigation.

[0088] Step 6: Output and application of results;

[0089] 6.1 Result storage and output;

[0090] The final semantic recognition results are output in the form of images, 3D models, or other suitable formats. These results can be stored locally or transmitted to the cloud via the network for further processing.

[0091] 6.2 Application scenarios;

[0092] The semantic segmentation results can be widely used in many fields:

[0093] (1) Autonomous driving: Real-time detection and identification of roads, pedestrians, vehicles and other elements to assist in the decision-making of the autonomous driving system.

[0094] (2) Robot navigation: Provides environmental perception for autonomous mobile robots to avoid obstacles and plan paths.

[0095] (3) Intelligent monitoring: Identify abnormal behavior or specific targets in surveillance videos to improve safety.

[0096] Through the above steps, the depth map-assisted multi-view semantic recognition method provided by the present invention demonstrates high recognition accuracy and practicality when processing complex scenes, multi-view fusion, and the combination of depth maps and RGB images.

[0097] The comparative test settings for this experiment are as follows:

[0098] The Princeton ModelNet40 dataset contains 12,311 3D CAD models from 40 categories. Point clouds are sampled from the CAD model surfaces, with view data captured by cameras. Experiments compare voxel-based methods: 3D ShapeNets and VoxlNet. Deep learning models: MVCNN, PointNet, and PointNet++.

[0099] The results are shown in the following table:

[0100]

[0101] Our method achieved a classification accuracy of 93.6% and an average retrieval precision of 90.5%, which is the best among the compared methods.

[0102] The above description is a further detailed description of the present invention in conjunction with specific / preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art of the present invention may make various substitutions or modifications to the described embodiments without departing from the scope of the present invention, and such substitutions or modifications should be considered to fall within the scope of protection of the present invention.

[0103] Parts of the present invention that are not described in detail belong to the common knowledge of those skilled in the art.

Claims

1. A multi-view semantic recognition method based on depth map assistance, characterized in that: This is achieved by following these steps: Step 1: Acquisition of multi-view depth maps and main view RGB images; Obtain a multi-view depth map and a main view RGB image of the target scene; preprocess the obtained multi-view depth map data to ensure the quality of the depth map data; select an RGB image of the main view, i.e., the main view, as the input image for single-view semantic recognition. The main view should have the best field of view, the least occlusion, and cover the main target objects in the scene; Step 2: Generation and processing of depth point cloud; Convert the multi-view depth map into multi-view depth point cloud data; each pixel in the depth map is converted into a three-dimensional coordinate point according to its depth value to form point cloud data; the generated multi-view depth point cloud is processed, including point cloud alignment, noise reduction and sparse processing; The processed multi-view depth point cloud data is used as one of the inputs of the deep fusion semantic network DFSN; Step 3: semantic recognition of single-view RGB images; The RGB image of the main view is input into the single-view semantic network SVSN; SVSN adopts a deep convolutional neural network architecture to first extract features from the RGB image; Then SVSN decodes the extracted features layer by layer through a multi-level semantic decoder to generate a preliminary semantic label map; the output semantic label map of SVSN is stored as one of the inputs of the final fusion result; Step 4: Deep fusion semantic recognition; The preprocessed multi-view depth point cloud and the main view RGB image are input into the deep fusion semantic network DFSN. DFSN first extracts features from the depth point cloud and RGB image independently. Then, the extracted depth point cloud and RGB image features are aligned. DFSN uses a feature alignment algorithm based on the attention mechanism to ensure that features from different modalities can be aligned in the same semantic space. The aligned features are fed into the feature fusion module for fusion. This module uses an adaptive feature fusion strategy to dynamically adjust the weights of RGB image features and depth point cloud features to give full play to their respective advantages. The fused features output by the feature fusion module are fed into the semantic decoder to generate a fused semantic label map. Step 5: Fusion of semantic recognition results; The semantic fusion module synthesizes the output results of the single-view semantic network (SVSN) and the deep fusion semantic network (DFSN). The semantic fusion module combines the semantic recognition results of the two networks using a weighted average combined with a rule-based fusion strategy. The specific fusion strategy is as follows: under normal circumstances, the final prediction result is the weighted average of the semantic recognition results of the two networks. However, in the case of missing input data, occlusion, or light reflection, the semantic fusion module will modify the prediction result according to different situations, thereby improving the reliability of the final recognition. Generate the final semantic recognition results; Step 6: Output and application; The final semantic recognition results are output in the form of images or three-dimensional models for subsequent analysis, processing or decision-making; the results can be used for real-time environment perception in autonomous driving systems, obstacle detection in drone navigation systems, or target tracking in intelligent monitoring systems.

2. The depth map-assisted multi-view semantic recognition method according to claim 1, characterized in that: Step 2: 2.1 Depth point cloud generation; Convert the preprocessed multi-view depth image into depth point cloud data; the specific method is to triangulate each pixel in the depth image according to its depth value and the corresponding camera intrinsic parameter to generate the point coordinates (x, y, z) in three-dimensional space; 2.2 Point cloud registration; In order to express the point clouds generated from different perspectives in a unified coordinate system, the point clouds from each perspective are registered. The registration method uses the ICP algorithm or rigid transformation alignment using known camera extrinsics. 2.3 Point cloud sparseness and denoising; Clustering method is used to simplify the point cloud and retain key feature points.

3. The depth map-assisted multi-view semantic recognition method according to claim 1, characterized in that: Step 3: 3.1RGB image feature extraction; The selected main-view RGB image is input into the single-view semantic network SVSN, which uses an encoder for feature extraction; the RGB image is passed through multiple convolutional layers to extract spatial features of different scales; the encoder includes four encoder layers: the first encoder layer consists of a convolution block and a pooling layer; the second encoder layer includes two residual blocks; the third and fourth encoder layers consist of a residual block with strided convolution and a residual block; the convolution block includes a convolution layer, batch normalization and ReLU activation function in sequence; the residual block consists of two convolution layers and a skip connection; the residual block with strided convolution consists of a strided convolution layer and a normal convolution layer combined with an internal skip connection; the main function of the strided convolution is to downsample and align the feature size, and the stride is set to 2; 3.2 Feature processing; The output after encoder processing is processed through the feature processing part to enhance the accuracy of semantic expression. The feature processing part consists of two residual blocks with dilated convolution. Among them, the residual blocks with dilated convolution use convolution kernels that are stretched and filled with holes instead of ordinary convolution kernels. 3.3 Semantic decoding; The processed features are input into the multi-level semantic decoder for semantic decoding; the multi-level semantic decoder includes four decoder layers, each of which receives the output of the corresponding encoder layer through a skip connection; the first encoder layer consists of two convolution blocks and a bilinear upsampling layer; the second and third encoder layers include a convolution block and a bilinear upsampling layer, and the last encoder layer includes two residual blocks; the bilinear upsampling layer is used to increase the resolution of the feature map to avoid blurring caused by interpolation; the skip connection directly connects the features of different stages of the encoder to the corresponding positions of the decoder, improving segmentation accuracy and retaining edge details; the multi-level semantic decoder decodes the features layer by layer, and each layer combines semantic information of different scales to finally generate a pixel-level semantic probability distribution matrix; 3.4 Semantic label output; The semantic probability distribution matrix is ​​finally used by the output layer to generate the probability distribution of the category to which each pixel belongs through the SoftMax function, and further generates a semantic label map; the semantic label map assigns a semantic label to each pixel.

4. The depth map-assisted multi-view semantic recognition method according to claim 1, characterized in that: Step 4: 4.1 Multimodal feature extraction; The processed multi-view depth point cloud data and multi-view RGB image are simultaneously input into the deep fusion semantic network DFSN; DFSN first performs feature extraction on these two input data respectively: pass PointNet++ extracts geometric and structural features of multi-view depth point clouds; Perform feature extraction on multi-view RGB images using ResNet-50 to extract color and texture features. For multi-view RGB images, use a weighted ResNet-50 to extract features containing color and texture information from each RGB image. The features obtained from each image are concatenated and integrated to obtain multi-view features. 4.2 Attention-based feature alignment; After feature extraction, DFSN uses an attention-based feature alignment module to align the multi-view features of the depth point cloud and RGB image into the same semantic space. The alignment module takes into account factors such as parallax compensation, feature rotation and scaling to ensure that multimodal features can be effectively fused. The processing flow of the feature alignment module is as follows: (1) Constructing a query-key-value representation: The point cloud geometric and structural features obtained in step 4.1, collectively referred to as point cloud features, are used as the query. The multi-view features obtained in step 4.1 are used as keys and values, and are mapped to a unified feature space through a fully connected layer with shared weights to ensure that the feature representation is consistent and aligned. (2) Calculating attention weights: Calculating attention weights based on the similarity between point cloud features and multi-view features, and dynamically calculating weights; dynamically calculating weights using the relationship between point cloud features and each view feature, thereby obtaining a multi-view attention distribution for the point cloud; The calculation method is as follows: the inner product similarity between the point cloud feature and each multi-view feature is calculated after SoftMax normalization; (3) Fusion view feature: Use the calculated attention weight to weight the multi-view features to generate a multimodal fusion feature aligned with the point cloud feature; (4) Output alignment feature: The generated multimodal fusion feature aligned with the point cloud feature is input into the subsequent network for semantic analysis and classification; 4.3 Feature Fusion; The point cloud features, multi-view attention distribution and multimodal fusion features are input into the feature fusion module to obtain fusion features; the specific fusion process is as follows: (1) the point cloud features and multimodal fusion features are aligned and integrated and then input into the fully connected layer, and the output results of the fully connected layer are passed through the maximum pooling layer to obtain the point-single view fusion results; (2) the multimodal fusion features are used to obtain the three features with the highest correlation according to the multi-view attention distribution, which are recorded as v1, v2, and v3; v1, v2, and v3 are aligned and integrated with the point cloud feature p to form {p, v1}, {p,, v1, v2}, and {p, v1, v2, v3}, respectively, and input into the fully connected layer to obtain MF1, MF2, and MF3 respectively; MF1, MF2, and MF3 are subjected to the average pooling operation to obtain the point-multi-view fusion results; (3) the point-single view fusion results and the point-multi-view fusion results are aligned and integrated into global features; 4.4 Semantic decoding; The fused global features are passed through the semantic decoder of DFSN to generate a fused semantic label map; the semantic decoder contains the following structures: (1) deconvolution module, which gradually restores the spatial resolution of the features so that the decoded feature map is consistent with the input data size; (2) multi-scale fusion module, which combines the features of different levels in the decoding process to enhance the understanding of global and local semantics; (3) convolution and activation functions, which further refine the decoded features and enhance the semantic representation ability of the output; (4) semantic label generation, which uses Softmax to normalize the feature distribution of each pixel and generate a category probability distribution; finally, the category with the highest probability is output as the prediction result of the semantic label map; 4.5 Fusion semantic label output; The final output of DFSN is the semantic segmentation result that combines multi-view depth point cloud and multi-view RGB image, thereby generating the final semantic label map.

5. The depth map-assisted multi-view semantic recognition method according to any one of claims 1 to 4, characterized in that: Step 5: 5.1 Implementation of semantic fusion module; After SVSN and DFSN generate semantic label maps respectively, the two results are combined through the semantic fusion module. The semantic fusion module uses weighted averaging combined with a rule-based fusion strategy to perform weighted fusion on the results of DFSN and SVSN to obtain a weighted average result. The weights are: DFSN - 0.7, SVSN - 0.

3. If the input data modality is missing, that is, the RGB image is missing or the number of valid points in the point cloud is less than 3000, the semantic recognition results of the non-missing data are used. If the geometric features are incomplete due to occlusion or noise, the weights are adjusted to: DFSN-0.3, SVSN-0.7; if there is strong light or blur resulting in texture loss, the weights are adjusted to: DFSN-0.1, SVSN-0.9; 5.2 Generating the final semantic recognition results; After fusion, the prediction result with the highest probability will be taken as the final semantic segmentation result of the scene.

Images