Room obstacle target detection method and system based on multi-modal information
Through the feature extraction and fusion of multimodal information, combined with the full convolutional network and the regional suggestion network, efficient and robust detection of room obstacles is achieved, and the inadequate information utilization and lack of weight adjustment mechanism in the existing technology is solved. Problem, improving the accuracy and automation of obstacle detection.
Patent Information
- Application Number
- CN202411546464.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-01
- Publication Date
- 2024-11-29
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing multimodal obstacle detection methods have not fully utilized the complementary characteristics of different mode data, lacked the adaptive adjustment mechanism for the weight of different mode data, and the research on room scenarios is not in-depth enough, making it difficult to achieve efficient and robust obstacles. Physical detection.
Using a room obstacle target detection method based on multimodal information, by acquiring RGB images and depth images, visual and depth features are extracted using convolutional neural networks, feature encoding and fusion, and pixels are performed by combining full convolutional networks and regional suggestion networks. Level segmentation and target detection, and the overlapping detection box is removed through a non-maximum suppression algorithm, outputting the location, size and category information of the obstacle target.
It significantly improves the accuracy and recall of obstacle detection, makes full use of the multimodal data of the image, provides richer feature representation, improves the comprehensiveness and robustness of detection, simplifies the post-processing process, and improves the Level of automation.
Smart Images

Figure CN119048747A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to target detection technology, and in particular to a room obstacle target detection method and system based on multimodal information. Background Art
[0002] Traditional obstacle detection methods are mainly based on single sensors, such as lidar, camera or depth camera. Although these methods can achieve good detection results under specific conditions, they still have some limitations. In order to overcome the limitations of a single sensor, researchers have begun to explore obstacle detection methods that fuse multimodal information. By fusing data from different sensors, their complementary characteristics can be fully utilized to improve the robustness and accuracy of obstacle detection. For example, by fusing lidar data with visual images, obstacle representations that have both three-dimensional geometric information and rich semantic information can be obtained; by fusing depth images with visual images, visual features can be used to complete and correct areas where depth information is missing.
[0003] However, the existing multimodal obstacle detection methods still have some shortcomings. First, most methods simply splice or cascade the data from different sensors without considering the semantic association and complementarity between different modal data, resulting in insufficient information utilization. Second, the existing methods lack an adaptive weight adjustment mechanism for different modal data during the information fusion process, making it difficult to dynamically adjust the fusion strategy according to different scenarios and task requirements. In addition, the research on multimodal obstacle detection methods for room scenes is not in-depth enough, and there is a lack of effective solutions that fully utilize the prior knowledge of room scenes.
[0004] Therefore, there is an urgent need for a room obstacle target detection method and system based on multimodal information, which can fully explore the complementary characteristics of different modal data, adaptively adjust the fusion weights, and combine the prior knowledge of the room scene to achieve efficient and robust obstacle detection. Summary of the invention
[0005] The embodiments of the present invention provide a room obstacle target detection method and system based on multimodal information, which can solve the problems in the prior art.
[0006] According to a first aspect of the embodiments of the present invention, A room obstacle target detection method based on multimodal information is provided, comprising: Obtain an RGB image and a depth image of the room to be detected, input the RGB image into the first convolutional neural network, extract visual features of the RGB image, input the depth image into the second convolutional neural network, extract depth features of the depth image, perform feature encoding on the depth image to obtain an encoded depth image, concatenate the extracted visual features and depth features in the channel dimension to obtain a fused feature map, and obtain a depth enhancement feature map based on the fused feature map and the encoded depth image; The depth enhancement feature map is input into the full convolutional network and the region proposal network respectively. The full convolutional network is used to perform pixel-level segmentation on the depth enhancement feature map to obtain the semantic segmentation result of the obstacle target. At the same time, the region proposal network is used to generate candidate regions and classify targets to obtain the initial obstacle target detection result. According to the obtained obstacle target semantic segmentation results and the initial obstacle target detection results, the overlapping detection frames are removed through the non-maximum suppression algorithm to obtain the final obstacle target detection results, and the position, size and category information of the obstacle targets in the room are output.
[0007] In an optional embodiment, The depth image is feature encoded, and the encoded depth image includes: Smoothing the acquired depth image, calculating the gradients of the smoothed depth image in the horizontal and vertical directions, obtaining the normal vector of each pixel in the depth image, and mapping the three components of the normal vector to the corresponding feature coding channels respectively, to obtain the extended surface normal vector feature coding channel; A plane is fitted by the least square method in the neighborhood of the pixel points of the depth image, and the flatness is obtained by calculating the average distance between the plane and all the pixel points in the neighborhood. The plane is mapped to a preset range through adaptive normalization, and a flatness channel of the extended feature coding is constructed according to the normalized result of the flatness. The flatness channel of the extended feature coding is combined with the original feature coding channel and the extended surface normal vector feature coding channel to obtain an encoded depth image.
[0008] In an optional embodiment, The flatness calculation formula is as follows: ; Where G(x, y) represents the flatness of the pixel point (x, y) in the depth image, V(·) represents the neighborhood of the pixel point, s represents the horizontal coordinate of a pixel point in the neighborhood, k represents the vertical coordinate of a pixel point in the neighborhood, a represents the x-axis parameter of the plane equation, b represents the y-axis parameter of the plane equation, c represents the z-axis parameter of the plane equation, d represents the constant parameter of the plane equation, and x represents the x-axis parameter of the plane equation. s Indicates the x-axis coordinate of the pixel point, y kRepresents the y-axis coordinate of the pixel point, z sk Indicates the z-axis coordinate of the pixel.
[0009] In an optional embodiment, The depth enhancement feature map is segmented at the pixel level through the fully convolutional network, and the semantic segmentation results of the obstacle target are obtained, including: Inputting the depth enhancement feature map into a fully convolutional network, performing semantic segmentation on the depth enhancement feature map through the fully convolutional network, and obtaining an initial segmentation result, wherein the initial segmentation result includes a probability distribution of each pixel in the depth enhancement feature map belonging to each pixel category; Based on the initial segmentation result, a conditional random field model is constructed, and a unary potential function and a pairwise potential function are introduced to construct the energy function of the conditional random field model. The conditional random field model is inferred using an approximate inference algorithm to obtain the optimal pixel category label assignment scheme when the energy function is minimized, and the optimal pixel category label assignment scheme is used as the second segmentation result; The initial segmentation result and the second segmentation result are fused to obtain the final semantic segmentation result of the obstacle target.
[0010] In an optional embodiment, The candidate regions are generated by the region proposal network and the target is classified. The initial obstacle target detection results include: The depth enhancement feature map is input into the pre-trained region proposal network, and multiple candidate regions of different scales and aspect ratios are preset. The binary classification layer is used to predict whether each candidate region contains an object, and the regression layer is used to predict the offset of each candidate region to generate multiple candidate regions. Filter the generated multiple candidate regions, use the non-maximum suppression algorithm to remove the candidate regions whose overlap with the highest confidence candidate region exceeds a preset threshold, and output the remaining candidate regions in order of confidence; Perform pooling operations on the output candidate regions to map candidate regions of different sizes into standard region features of fixed sizes; The standard region features are mapped to features of a specific dimension through a fully connected layer and normalized to obtain the probability that the candidate region belongs to each region category. The region category with the highest probability and exceeding the threshold is taken as the predicted category of the candidate region and used as the initial obstacle target detection result.
[0011] In an optional embodiment, Training a region proposal network involves: Calculate the classification loss and regression loss of the region proposal network, and use the back propagation algorithm to update the parameters of the region proposal network. The classification loss calculation formula of the region proposal network is as follows: ; Among them, p i represents the probability that the i-th candidate region contains a target, represents the true label of the i-th candidate region, where 1 represents the target, 0 represents the background, i represents the i-th candidate region, and N cls Represents the normalization factor of classification loss; The regression loss is calculated as follows: ; Among them, t i represents the bounding box offset of the i-th candidate region, represents the true bounding box offset corresponding to the i-th candidate region, N reg Represents the normalization factor of regression loss, Smooth L1 (·) means Smooth L1 Loss function.
[0012] In an optional embodiment, The non-maximum suppression algorithm is used to remove redundant and overlapping detection frames, and the final obstacle target detection results include: Construct a graph structure, consider all detection boxes in the initial obstacle target detection results as nodes in the graph, and use the overlap and semantic similarity between detection boxes as edges; Use the trained graph neural network to assign different weights to each node, adaptively aggregate the information of neighboring nodes, obtain the aggregated features of the node, and concatenate the aggregated features of the node with the original detection box confidence. Perform nonlinear transformation through a feedforward neural network to obtain the updated node features. The updated node features are concatenated with the original detection frame confidence to obtain the final score of the detection frame. The detection frames are sorted in descending order according to the final score of the detection frame, and the final obstacle target detection result is obtained by screening through the non-maximum suppression algorithm.
[0013] According to a second aspect of the embodiments of the present invention, A room obstacle target detection system based on multimodal information is provided, comprising: The first unit is used to obtain an RGB image and a depth image of the room to be detected, input the RGB image into the first convolutional neural network, extract visual features of the RGB image, input the depth image into the second convolutional neural network, extract depth features of the depth image, perform feature encoding on the depth image to obtain an encoded depth image, concatenate the extracted visual features and depth features in the channel dimension to obtain a fused feature map, and obtain a depth enhancement feature map based on the fused feature map and the encoded depth image; The second unit is used to input the depth enhancement feature map into the full convolutional network and the region proposal network respectively, and perform pixel-level segmentation on the depth enhancement feature map through the full convolutional network to obtain the semantic segmentation result of the obstacle target. At the same time, the candidate region is generated through the region proposal network and the target is classified to obtain the initial obstacle target detection result; The second unit is used to remove overlapping detection frames through a non-maximum suppression algorithm based on the obtained obstacle target semantic segmentation results and the initial obstacle target detection results, obtain the final obstacle target detection results, and output the position, size and category information of the obstacle targets in the room.
[0014] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.
[0015] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.
[0016] In this embodiment, two kinds of visual information, depth and RGB, are integrated to make full use of the multimodal data of the image. Through operations such as feature encoding, fusion and depth enhancement, a richer feature representation can be obtained, which provides strong support for subsequent obstacle target detection, thereby significantly improving the accuracy and recall of detection. The embodiment of the present invention includes two branches, semantic segmentation and target detection, which can verify and promote each other. The semantic segmentation results can provide effective prior constraints for target detection, and the target detection results can in turn optimize and refine the semantic segmentation. The synergy and complementarity of the two tasks help to improve the overall detection performance. The candidate regions of various scales and aspect ratios are preset through the region proposal network, which can adapt to obstacle targets of different sizes and shapes. At the same time, the full convolutional network branch provides pixel-level semantic segmentation results, and the two multi-scale and multi-perspective representations complement each other, improving the comprehensiveness and robustness of detection. Through the non-maximum suppression algorithm, a large number of redundant and overlapping detection frames can be automatically removed, simplifying the traditional complex post-processing process that requires manual participation, and improving the automation level of detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 Schematic diagram of a flow chart of a room obstacle target detection method based on multimodal information according to an embodiment of the present invention; Figure 2Schematic diagram of the structure of a room obstacle target detection system based on multimodal information according to an embodiment of the present invention. DETAILED DESCRIPTION
[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0019] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.
[0020] Figure 1 FIG. 1 is a flow chart of an embodiment of a method for detecting room obstacles based on multimodal information according to the present invention. Figure 1 As shown, the method includes: S101. Obtain an RGB image and a depth image of the room to be detected, input the RGB image into a first convolutional neural network, extract visual features of the RGB image, input the depth image into a second convolutional neural network, extract depth features of the depth image, perform feature encoding on the depth image to obtain an encoded depth image, concatenate the extracted visual features and depth features in the channel dimension to obtain a fused feature map, and obtain a depth enhancement feature map based on the fused feature map and the encoded depth image.
[0021] Among them, the first convolutional neural network adopts a deep residual network structure such as ResNet or DenseNet, and the second convolutional neural network adopts an encoding-decoding structure. During the encoding process, multi-scale convolution and pooling operations are used to extract the multi-scale features of the depth image, and during the decoding process, upsampling and jump connections are used to restore the spatial resolution of the feature map.
[0022] The feature encoding uses HHA (Height-Horizontal angle-Angle) encoding, which can highlight the geometric structure in the scene, emphasize the spatial relationship between different objects, and help improve the accuracy of obstacle segmentation. The depth image after HHA encoding includes three channels: height, horizontal disparity, and angle. Then, the feature pyramid of the depth image at different scales is extracted and point multiplied with the fused feature map to obtain the depth-enhanced feature map.
[0023] In an optional embodiment, The depth image is feature encoded, and the encoded depth image includes: Smoothing the acquired depth image, calculating the gradients of the smoothed depth image in the horizontal and vertical directions, obtaining the normal vector of each pixel in the depth image, and mapping the three components of the normal vector to the corresponding feature coding channels respectively, to obtain the extended surface normal vector feature coding channel; A plane is fitted by the least square method in the neighborhood of the pixel points of the depth image, and the flatness is obtained by calculating the average distance between the plane and all the pixel points in the neighborhood. The plane is mapped to a preset range through adaptive normalization, and a flatness channel of the extended feature coding is constructed according to the normalized result of the flatness. The flatness channel of the extended feature coding is combined with the original feature coding channel and the extended surface normal vector feature coding channel to obtain an encoded depth image.
[0024] Among them, the surface normal vector describes the direction of the plane where each pixel in the scene is located, and can provide important information about the local geometric structure.
[0025] For example, an edge-preserving filtering algorithm such as a bilateral filter can be used to smooth the depth value while maintaining the depth discontinuity. Then, the surface normal vector of each pixel can be obtained by calculating the gradient of the smoothed depth image.
[0026] In order to encode the surface normal vector into the HHA representation, it needs to be mapped to the range of [0, 255]. Specifically, the three components of the surface normal vector can be mapped to the corresponding channels respectively to form an extended surface normal vector feature encoding.
[0027] For each pixel in the depth image, determine the neighborhood and neighborhood radius of the point, fit a plane equation in the neighborhood using the least squares method, and calculate the average distance between the plane and all pixels in the neighborhood as the flatness. Adaptively normalize the calculated flatness and map it to a preset range, such as [0, 1] or [0, 255]. Based on the normalized flatness value, construct the flatness channel of the extended feature encoding.
[0028] The flatness channel of the extended feature coding is combined with the original feature coding channels (three channels of height, horizontal angle and angle coded by HHA) and the extended surface normal vector feature coding channel to obtain an encoded depth image, which includes multiple feature coding channels.
[0029] In an optional embodiment, The flatness calculation formula is as follows: ; Where G(x, y) represents the flatness of the pixel point (x, y) in the depth image, V(·) represents the neighborhood of the pixel point, s represents the horizontal coordinate of a pixel point in the neighborhood, k represents the vertical coordinate of a pixel point in the neighborhood, a represents the x-axis parameter of the plane equation, b represents the y-axis parameter of the plane equation, c represents the z-axis parameter of the plane equation, d represents the constant parameter of the plane equation, and x represents the x-axis parameter of the plane equation. s Indicates the x-axis coordinate of the pixel point, y k Represents the y-axis coordinate of the pixel point, z sk Indicates the z-axis coordinate of the pixel.
[0030] In this embodiment, by constructing an extended surface normal feature coding channel and a flatness feature coding channel, the geometric information contained in the depth image can be more comprehensively expressed, including not only basic height, horizontal angle and angle information, but also details such as the surface normal direction and local flatness. This rich expression of depth information helps subsequent computer vision tasks to better utilize depth data. Combining multiple different types of feature coding channels can greatly improve the feature expression capability of depth information. Features such as surface normals and flatness can reflect the geometric structure and spatial layout information of objects in the scene. By utilizing these features, the computer vision system can better understand the three-dimensional geometric structure of the scene, bringing significant performance improvements to tasks such as scene understanding and semantic segmentation.
[0031] S102. Input the depth enhanced feature map into the full convolutional network and the region proposal network respectively, perform pixel-level segmentation on the depth enhanced feature map through the full convolutional network to obtain the semantic segmentation result of the obstacle target, and generate candidate regions and perform target classification through the region proposal network to obtain the initial obstacle target detection result.
[0032] Among them, the fully convolutional network (FCN) classifies pixels and generates semantic segmentation results, and a region proposal network (RPN) is added to the last convolution layer of the fully convolutional network to generate candidate regions. RPN generates multiple rectangular candidate boxes of different sizes and aspect ratios at each position by sliding a small convolution kernel on the fused feature map. Then, these candidate boxes are subjected to binary classification (foreground / background) and bounding box regression to obtain the target detection results.
[0033] In an optional embodiment, The depth enhancement feature map is segmented at the pixel level through the fully convolutional network, and the semantic segmentation results of the obstacle target are obtained, including: Inputting the depth enhancement feature map into a fully convolutional network, performing semantic segmentation on the depth enhancement feature map through the fully convolutional network, and obtaining an initial segmentation result, wherein the initial segmentation result includes a probability distribution of each pixel in the depth enhancement feature map belonging to each pixel category; Based on the initial segmentation result, a conditional random field model is constructed, and a unary potential function and a pairwise potential function are introduced to construct the energy function of the conditional random field model. The conditional random field model is inferred using an approximate inference algorithm to obtain the optimal pixel category label assignment scheme when the energy function is minimized, and the optimal pixel category label assignment scheme is used as the second segmentation result; The initial segmentation result and the second segmentation result are fused to obtain the final semantic segmentation result of the obstacle target.
[0034] For example, the architecture of the fully convolutional network is first designed, usually using an encoder-decoder structure, such as U-Net or DeepLab series networks. The deep enhanced feature map is used as the input of the fully convolutional network, and multi-scale and multi-level feature representations are extracted through operations such as convolutional layers, pooling layers, and upsampling layers. In the output layer of the network, the probability distribution of each pixel belonging to each pixel category is calculated through the softmax activation function. The network is trained end-to-end using the cross entropy loss function to optimize the network parameters so that it can accurately predict the category probability distribution of pixels.
[0035] In the testing phase, the depth-enhanced feature map is input into the trained fully convolutional network to obtain the preliminary segmentation result, that is, the probability distribution of each pixel belonging to each pixel category.
[0036] A conditional random field model is constructed based on the preliminary segmentation results. Specifically, the depth enhancement feature map and the preliminary segmentation results are used as the observed variables and initial labels of the conditional random field model. The energy function of the conditional random field model is defined, including a unary potential function and a pairwise potential function.
[0037] Among them, the unary potential function measures the likelihood of each pixel being assigned to a specific category, and is usually designed based on the category probability distribution of the preliminary segmentation result. Specifically, the preliminary segmentation result is used as the input of the unary potential function to extract the probability distribution of each pixel belonging to each category. The preliminary segmentation result is normalized so that it satisfies the properties of the probability distribution, that is, the sum of the probabilities of all categories is 1. The normalized probability distribution is converted into an energy value, usually using a negative logarithmic function, that is, the energy value of the unary potential function is equal to the negative logarithmic probability. The energy values of the unary potential function of all pixels are calculated to obtain a complete unary potential function.
[0038] The pairwise potential function measures the compatibility of adjacent pixel labels and encourages pixels with similar features to be assigned to the same category. It usually considers the differences in features such as color, texture, and depth between pixels. Specifically, the scope of the pairwise potential function is determined, usually considering the 4-neighborhood or 8-neighborhood of the pixel. For each pair of adjacent pixels, their feature vectors are extracted, such as color, texture, depth, etc. The similarity between the feature vectors of adjacent pixels is calculated. Commonly used similarity metrics include Euclidean distance and Mahalanobis distance. The similarity is converted into an energy value, usually using an exponential function, that is, the energy value of the pairwise potential function is equal to the similarity of the exponential function. According to the compatibility of the pixel labels, the energy value of the pairwise potential function is adjusted to encourage pixels with similar features to be assigned to the same category. The energy values of the pairwise potential function of all adjacent pixel pairs are calculated to obtain a complete pairwise potential function.
[0039] The energy values of the unary potential function and the pairwise potential function are weighted and summed to obtain the complete energy function of the conditional random field model. The conditional random field model is constructed and the category label of the pixel is used as the hidden variable of the model.
[0040] Select an appropriate approximate inference algorithm, such as Mean Field Inference or Belief Propagation. Initialize the probability distribution of pixel category labels, usually using the preliminary segmentation results as the initial value. Iteratively update the probability distribution of pixel category labels, and calculate the message of each pixel based on the unary potential function and the pairwise potential function. In each iteration, the probability distribution of pixel category labels is updated according to the message until the convergence condition is reached or the preset number of iterations is reached. After convergence, the probability distribution of pixel category labels is converted into a determined label assignment scheme to obtain the second segmentation result of conditional random field inference.
[0041] The initial segmentation result and the second segmentation result are fused. First, a fusion strategy is designed, such as weighted average, voting, etc., to comprehensively consider the contribution of the initial segmentation result and the second segmentation result. According to the fusion strategy, the final category label of each pixel is calculated to obtain the fused semantic segmentation result. The fused semantic segmentation result is post-processed, such as removing small connected areas and smoothing segmentation boundaries, to improve the visual quality and consistency of the segmentation.
[0042] Optionally, use the samples on the test set to evaluate the semantic segmentation results and calculate various evaluation indicators, such as pixel accuracy, mean IoU, etc. Analyze the error samples of the semantic segmentation results and summarize the advantages and disadvantages of the algorithm and the direction of improvement.
[0043] In this embodiment, by combining the initial segmentation result with the conditional random field model, the accuracy of semantic segmentation can be significantly improved. The paired potential function in the conditional random field model can encourage adjacent pixels to have the same label, so that the segmentation result is smoother and more continuous, avoiding the presence of messy isolated pixels in the segmented area. This consistency is very beneficial for subsequent target recognition, tracking and other tasks. By integrating multiple visual cues such as depth information, color similarity, and spatial distance into the conditional random field model, the multimodal information in the image is fully utilized to produce more accurate and robust segmentation results. By dividing the fully convolutional network and the conditional random field model into two levels, this hierarchical structure makes the entire optimization process clearer and more controllable, and facilitates individual debugging and improvement of each module.
[0044] In an optional embodiment, The candidate regions are generated by the region proposal network and the target is classified. The initial obstacle target detection results include: The depth enhancement feature map is input into the pre-trained region proposal network, and multiple candidate regions of different scales and aspect ratios are preset. The binary classification layer is used to predict whether each candidate region contains an object, and the regression layer is used to predict the offset of each candidate region to generate multiple candidate regions. Filter the generated multiple candidate regions, use the non-maximum suppression algorithm to remove the candidate regions whose overlap with the highest confidence candidate region exceeds a preset threshold, and output the remaining candidate regions in order of confidence; Perform pooling operations on the output candidate regions to map candidate regions of different sizes into standard region features of fixed sizes; The standard region features are mapped to features of a specific dimension through a fully connected layer and normalized to obtain the probability that the candidate region belongs to each region category. The region category with the highest probability and exceeding the threshold is taken as the predicted category of the candidate region and used as the initial obstacle target detection result.
[0045] Exemplarily, a pre-trained region proposal network (RPN) is used to process the input depth enhancement feature map. Multiple reference boxes of different scales and aspect ratios are preset on the feature map. The binary classification layer of the RPN is used to predict whether each reference box contains an object. The regression layer of the RPN is used to predict the offset adjustment required for each reference box.
[0046] For all generated candidate regions, calculate their overlap with the highest confidence candidate region. Set a threshold and remove candidate regions whose overlap with the highest confidence candidate region exceeds the threshold. Repeat the previous step until there are no candidate regions to be removed. Sort the remaining candidate regions by confidence and output them.
[0047] For each candidate region output, a pooling operation is performed on the depth-enhanced feature map, mapping candidate regions of different sizes to standard region features of fixed size.
[0048] The standard region features are mapped into a feature vector of a specific dimension through a fully connected layer. The feature vector is normalized to obtain the probability score of the candidate region belonging to each obstacle category. By setting a probability threshold (such as 0.5), the obstacle category with the highest probability score that exceeds the preset threshold is used as the predicted category of the candidate region. The location, size, and predicted category of all candidate regions are output as the initial obstacle target detection result.
[0049] In an optional embodiment, Training a region proposal network involves: Calculate the classification loss and regression loss of the region proposal network, and use the back propagation algorithm to update the parameters of the region proposal network. The classification loss calculation formula of the region proposal network is as follows: ; Among them, p i represents the probability that the i-th candidate region contains a target, represents the true label of the i-th candidate region, where 1 represents the target, 0 represents the background, i represents the i-th candidate region, and N cls Represents the normalization factor of classification loss; The regression loss is calculated as follows: ; Among them, t i represents the bounding box offset of the i-th candidate region, represents the true bounding box offset corresponding to the i-th candidate region, N reg Represents the normalization factor of regression loss, Smooth L1 (·) means Smooth L1 Loss function.
[0050] In this embodiment, multiple modules such as region proposal network, candidate region feature extraction and classification prediction are integrated, which can accurately detect the position, size and category of obstacle targets in complex scenes, and provide important information support for subsequent navigation and decision-making. By presetting a variety of reference frames of different scales and aspect ratios, it can adapt to obstacle targets of different sizes and shapes, and improve the comprehensiveness and robustness of detection. The entire detection process is end-to-end, without manual participation, and automatically generates the final target detection result from the input depth enhancement feature map, which can be efficiently applied to real-time detection scenarios. Using the depth enhancement feature map as input, the depth information can be fully utilized to obtain more geometric and structural clues based on the traditional RGB image target detection, thereby further improving the detection performance. By combining the candidate region-based target detection framework with depth information processing, an efficient, accurate, robust and scalable obstacle target detection solution is formed.
[0051] S103. According to the obtained obstacle target semantic segmentation result and the initial obstacle target detection result, the overlapping detection frames are removed by the non-maximum suppression algorithm to obtain the final obstacle target detection result, and the position, size and category information of the obstacle target in the room are output.
[0052] In an optional embodiment, The non-maximum suppression algorithm is used to remove redundant and overlapping detection frames, and the final obstacle target detection results include: Construct a graph structure, consider all detection boxes in the initial obstacle target detection results as nodes in the graph, and use the overlap and semantic similarity between detection boxes as edges; Use the trained graph neural network to assign different weights to each node, adaptively aggregate the information of neighboring nodes, obtain the aggregated features of the node, and concatenate the aggregated features of the node with the original detection box confidence. Perform nonlinear transformation through a feedforward neural network to obtain the updated node features. The updated node features are concatenated with the original detection frame confidence to obtain the final score of the detection frame. The detection frames are sorted in descending order according to the final score of the detection frame, and the final obstacle target detection result is obtained by screening through the non-maximum suppression algorithm.
[0053] Exemplarily, first, all detection boxes in the obstacle detection results are regarded as nodes in the graph. In order to construct the edges between the nodes, the overlap and semantic similarity between the detection boxes are calculated. For the overlap, the traditional intersection over union (IoU) calculation method is used, that is, the intersection area of the two detection boxes is divided by their union area. For semantic similarity, the obstacle category information to which the detection boxes belong is used to calculate their distance in the semantic space. Specifically, each obstacle category is represented as a vector, which is learned by word embedding (such as Word2Vec) and other methods. Then, the cosine similarity between the category vectors to which the two detection boxes belong is calculated as their semantic similarity.
[0054] The edges between detection boxes can be constructed based on the calculated overlap and semantic similarity. The overlap and semantic similarity are normalized separately, and then weighted and summed to obtain the final edge weight.
[0055] After constructing the graph structure, a graph neural network (GNN) is used to update and propagate node features and introduce global context information. Specifically, a graph attention network (GAT) is used as the basic GNN module. GAT assigns different weights to each node through the attention mechanism, and can adaptively aggregate the information of neighboring nodes. In the method in this paper, the input of GAT is the feature vector of the detection box (such as the features extracted by CNN), and the output is the updated node feature vector. By stacking multiple layers of GAT, the recursive propagation and update of node features on the graph can be realized, capturing context information of different ranges.
[0056] In each layer of GAT, first, the attention weights between the node and its neighboring nodes are calculated. Then, the features of the neighboring nodes are weighted and summed using the attention weights to obtain the aggregated features of node $i$. Finally, the aggregated features are concatenated with the features of the node itself and nonlinearly transformed through a feedforward neural network (FFN) to obtain the updated node features. Through the recursive update of multiple layers of GAT, contextual information of different ranges can be gradually integrated to obtain a more comprehensive and accurate node representation.
[0057] After obtaining the updated node features, they are concatenated with the original detection frame confidence as the final score of each detection frame. Then, all detection frames are sorted in descending order according to the scores and screened using the non-maximum suppression (NMS) algorithm. Specifically, starting from the detection frame with the highest score, other detection frames with an overlap higher than the threshold are continuously removed until all detection frames are processed. The final set of detection frames is the optimized obstacle detection result.
[0058] In this embodiment, the graph neural network can make full use of the mutual relationship between detection frames, including overlap and semantic similarity. By aggregating the information of neighbor nodes on the graph structure, the context of each detection frame can be better understood, thereby improving the recognition accuracy of obstacle targets. Since the graph neural network can perform global reasoning on the entire detection result, it can promote the consistency of the detection results. Similar obstacle targets will receive similar detection scores, thereby reducing the drastic changes in the detection results of the previous and next frames or adjacent areas, making the detection smoother and more coherent. By aggregating the information of multiple neighbor nodes, the limitations of a single detection frame can be compensated. Even if the confidence of some detection frames is low, they can be corrected with the help of the surrounding contextual information, thereby improving the overall robustness of the detection results. Since the graph neural network can learn the relationship between detection frames, it can better identify repeated and redundant detection frames. By appropriately reducing the scores of these detection frames and cooperating with the non-maximum suppression algorithm, a large number of redundant detection results can be effectively filtered out. By introducing the graph neural network module, the relationship information between the detection frames can be effectively utilized, thereby significantly improving the accuracy, consistency, robustness and efficiency of the detection, while maintaining good adaptability and interpretability, providing an advanced solution for obstacle detection tasks in complex scenarios.
[0059] Figure 2 FIG. 1 is a schematic diagram of the structure of a room obstacle target detection embodiment system based on multimodal information of the present invention. Figure 2 As shown, the system comprises: The first unit is used to obtain an RGB image and a depth image of the room to be detected, input the RGB image into the first convolutional neural network, extract visual features of the RGB image, input the depth image into the second convolutional neural network, extract depth features of the depth image, perform feature encoding on the depth image to obtain an encoded depth image, concatenate the extracted visual features and depth features in the channel dimension to obtain a fused feature map, and obtain a depth enhancement feature map based on the fused feature map and the encoded depth image; The second unit is used to input the depth enhancement feature map into the full convolutional network and the region proposal network respectively, and perform pixel-level segmentation on the depth enhancement feature map through the full convolutional network to obtain the semantic segmentation result of the obstacle target. At the same time, the candidate region is generated through the region proposal network and the target is classified to obtain the initial obstacle target detection result; The second unit is used to remove overlapping detection frames through a non-maximum suppression algorithm based on the obtained obstacle target semantic segmentation results and the initial obstacle target detection results, obtain the final obstacle target detection results, and output the position, size and category information of the obstacle targets in the room.
[0060] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.
[0061] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.
[0062] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.
[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A room obstacle target detection method based on multimodal information, It is characterized in that include: Obtain an RGB image and a depth image of the room to be detected, input the RGB image into the first convolutional neural network, extract visual features of the RGB image, input the depth image into the second convolutional neural network, extract depth features of the depth image, perform feature encoding on the depth image to obtain an encoded depth image, concatenate the extracted visual features and depth features in the channel dimension to obtain a fused feature map, and obtain a depth enhancement feature map based on the fused feature map and the encoded depth image; The depth enhancement feature map is input into the full convolutional network and the region proposal network respectively. The full convolutional network is used to perform pixel-level segmentation on the depth enhancement feature map to obtain the semantic segmentation result of the obstacle target. At the same time, the region proposal network is used to generate candidate regions and classify targets to obtain the initial obstacle target detection result. According to the obtained obstacle target semantic segmentation results and the initial obstacle target detection results, the overlapping detection frames are removed through the non-maximum suppression algorithm to obtain the final obstacle target detection results, and the position, size and category information of the obstacle targets in the room are output.
2. The method according to claim 1, It is characterized in that The depth image is feature encoded, and the encoded depth image includes: Smoothing the acquired depth image, calculating the gradients of the smoothed depth image in the horizontal and vertical directions, obtaining the normal vector of each pixel in the depth image, and mapping the three components of the normal vector to the corresponding feature coding channels respectively, to obtain the extended surface normal vector feature coding channel; A plane is fitted by the least square method in the neighborhood of the pixel points of the depth image, and the flatness is obtained by calculating the average distance between the plane and all the pixel points in the neighborhood. The plane is mapped to a preset range through adaptive normalization, and a flatness channel of the extended feature coding is constructed according to the normalized result of the flatness. The flatness channel of the extended feature coding is combined with the original feature coding channel and the extended surface normal vector feature coding channel to obtain an encoded depth image.
3. The method according to claim 2, It is characterized in that The flatness calculation formula is as follows: ; Where G(x, y) represents the flatness of the pixel point (x, y) in the depth image, V(·) represents the neighborhood of the pixel point, s represents the horizontal coordinate of a pixel point in the neighborhood, k represents the vertical coordinate of a pixel point in the neighborhood, a represents the x-axis parameter of the plane equation, b represents the y-axis parameter of the plane equation, c represents the z-axis parameter of the plane equation, d represents the constant parameter of the plane equation, and x represents the x-axis parameter of the plane equation. s Indicates the x-axis coordinate of the pixel point, y k Represents the y-axis coordinate of the pixel point, z sk Indicates the z-axis coordinate of the pixel.
4. The method according to claim 1, It is characterized in that The depth enhancement feature map is segmented at the pixel level through the fully convolutional network, and the semantic segmentation results of the obstacle target are obtained, including: Inputting the depth enhancement feature map into a fully convolutional network, performing semantic segmentation on the depth enhancement feature map through the fully convolutional network, and obtaining an initial segmentation result, wherein the initial segmentation result includes a probability distribution of each pixel in the depth enhancement feature map belonging to each pixel category; Based on the initial segmentation result, a conditional random field model is constructed, and a unary potential function and a pairwise potential function are introduced to construct the energy function of the conditional random field model. The conditional random field model is inferred using an approximate inference algorithm to obtain the optimal pixel category label assignment scheme when the energy function is minimized, and the optimal pixel category label assignment scheme is used as the second segmentation result; The initial segmentation result and the second segmentation result are fused to obtain the final semantic segmentation result of the obstacle target.
5. The method according to claim 1, It is characterized in that The candidate regions are generated by the region proposal network and the target is classified. The initial obstacle target detection results include: The depth enhancement feature map is input into the pre-trained region proposal network, and multiple candidate regions of different scales and aspect ratios are preset. The binary classification layer is used to predict whether each candidate region contains an object, and the regression layer is used to predict the offset of each candidate region to generate multiple candidate regions. Filter the generated multiple candidate regions, use the non-maximum suppression algorithm to remove the candidate regions whose overlap with the highest confidence candidate region exceeds a preset threshold, and output the remaining candidate regions in order of confidence; Perform pooling operations on the output candidate regions to map candidate regions of different sizes into standard region features of fixed sizes; The standard region features are mapped to features of a specific dimension through a fully connected layer and normalized to obtain the probability that the candidate region belongs to each region category. The region category with the highest probability and exceeding the threshold is taken as the predicted category of the candidate region and used as the initial obstacle target detection result.
6. The method according to claim 5, It is characterized in that Training a region proposal network involves: Calculate the classification loss and regression loss of the region proposal network, and use the back propagation algorithm to update the parameters of the region proposal network. The classification loss calculation formula of the region proposal network is as follows: ; Among them, p i represents the probability that the i-th candidate region contains a target, represents the true label of the i-th candidate region, where 1 represents the target, 0 represents the background, i represents the i-th candidate region, and N cls Represents the normalization factor of classification loss; The regression loss is calculated as follows: ; Among them, t i represents the bounding box offset of the i-th candidate region, represents the true bounding box offset corresponding to the i-th candidate region, N reg Represents the normalization factor of regression loss, Smooth L1 (·) means Smooth L1 Loss function.
7. The method according to claim 1, It is characterized in that The non-maximum suppression algorithm is used to remove redundant and overlapping detection frames, and the final obstacle target detection results include: Construct a graph structure, consider all detection boxes in the initial obstacle target detection results as nodes in the graph, and use the overlap and semantic similarity between detection boxes as edges; Use the trained graph neural network to assign different weights to each node, adaptively aggregate the information of neighboring nodes, obtain the aggregated features of the node, and concatenate the aggregated features of the node with the original detection box confidence. Perform nonlinear transformation through a feedforward neural network to obtain the updated node features. The updated node features are concatenated with the original detection frame confidence to obtain the final score of the detection frame. The detection frames are sorted in descending order according to the final score of the detection frame, and the final obstacle target detection result is obtained by screening through the non-maximum suppression algorithm.
8. A room obstacle target detection system based on multimodal information, used to implement the method described in any one of claims 1 to 7. It is characterized in that include: The first unit is used to obtain an RGB image and a depth image of the room to be detected, input the RGB image into the first convolutional neural network, extract visual features of the RGB image, input the depth image into the second convolutional neural network, extract depth features of the depth image, perform feature encoding on the depth image to obtain an encoded depth image, concatenate the extracted visual features and depth features in the channel dimension to obtain a fused feature map, and obtain a depth enhancement feature map based on the fused feature map and the encoded depth image; The second unit is used to input the depth enhancement feature map into the full convolutional network and the region proposal network respectively, and perform pixel-level segmentation on the depth enhancement feature map through the full convolutional network to obtain the semantic segmentation result of the obstacle target. At the same time, the candidate region is generated through the region proposal network and the target is classified to obtain the initial obstacle target detection result; The second unit is used to remove overlapping detection frames through a non-maximum suppression algorithm based on the obtained obstacle target semantic segmentation results and the initial obstacle target detection results, obtain the final obstacle target detection results, and output the position, size and category information of the obstacle targets in the room.
9. An electronic device, It is characterized in that include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.
10. A computer readable storage medium having computer program instructions stored thereon, It is characterized in that When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
An image semantic segmentation method based on deep learning
CN109711413A
Anchor-frame-free 3D target detection method based on multi-sensor fusion
CN114118247A
Cascade regression target detection method and device and computer readable storage medium
CN114241250A
RGB-D saliency target detection method based on depth quality weighting
CN116310396A
RGB-D saliency target detection method based on lightweight cross-modal fusion network
CN116486112A
Cited By
Early warning processing method and device for remote monitoring
CN119649614A
Unmanned aerial vehicle routing inspection line adaptive obstacle detection method and system based on monocular camera
CN119672577A
Heavy-load large holding tool driving wheel multi-mode damage detection method based on machine vision
CN119760646A
Multi-modal information system construction method for multi-tuple unmanned equipment
CN119942292A
State indicator lamp positioning method and device for complex electrical instrument equipment
CN119992039A
