Indoor robot navigation method based on multi-modal feature fusion

Abstract

The invention relates to an indoor robot navigation method based on multi-modal feature fusion. The method comprises the following steps: constructing a semantic map based on visual observation environment information; the method comprises the following steps: acquiring an RGB image of an indoor scene object, converting the RGB image into point cloud data, and preprocessing the RGB image and the point cloud data; image multilayer semantic features and point cloud features in the RGB image and the point cloud data are extracted respectively, and initial fusion is carried out; performing weighted fusion on the multi-layer semantic features and the point cloud features of the image by adopting space-channel-cross-modal multi-attention dynamic cooperation; and predicting a long-term target in a map space from top to bottom based on the fused feature map and the semantic map, and performing navigation path planning based on the current position and the long-term target. Under low-cost hardware configuration, the robustness of environment perception, the real-time performance of decision response and the usability of system integration in a dynamic complex environment are comprehensively improved, and the method is particularly suitable for application scenes such as indoor service robots.

Description

Technical Field

[0001] The present application relates to the technical fields of robots and artificial intelligence, and particularly to an indoor robot navigation method based on multi-modal feature fusion. Background Art

[0002] In the field of indoor robot navigation, accurate environmental perception and real-time autonomous decision-making in complex dynamic scenarios are the core challenges restricting the implementation of technologies.

[0003] Current mainstream systems mostly rely on a single sensor (such as lidar or vision camera), but face significant limitations. On the one hand, although lidar can provide centimeter-level geometric accuracy through dense point clouds, it is prone to interference due to reflections in scenarios such as weak-texture corridors and glass curtain walls, and cannot identify semantic targets, resulting in a lack of scene understanding ability in navigation strategies. On the other hand, although vision cameras can capture rich semantic information, their positioning accuracy is easily affected by sudden changes in illumination (such as direct sunlight, shadow occlusion), motion blur, and weak-texture environments, and the positioning error can reach 10% - 30%.

[0004] In addition, existing fusion methods mostly adopt shallow data splicing (such as directly aligning point clouds and image pixels) or simple weighted fusion, ignoring the heterogeneous characteristics of multi-modal data: the sparsity of point clouds, the high-dimensional semantics of noise distribution in images, and the significant feature differences between dense pixels. Direct fusion is prone to information redundancy and modal interference.

[0005] Therefore, in the related technologies, there is an urgent need for a method that can solve the problems of low efficiency of multi-modal sensor information fusion and poor environmental adaptability in indoor robot navigation. Summary of the Invention

[0006] Based on this, in view of the above technical problems, it is necessary to provide an indoor robot navigation method based on multi-modal feature fusion that can solve the problems of low efficiency of multi-modal sensor information fusion and poor environmental adaptability in indoor robot navigation.

[0007] In a first aspect, the present application provides an indoor robot navigation method based on multi-modal feature fusion. The method includes:

[0008] Construct a semantic map based on visual observation of environmental information;

[0009] Obtain the RGB image of an indoor scene object, convert it into point cloud data, and preprocess the RGB image and the point cloud data;

[0010] Extract the multi-layer semantic features of the RGB image and the point cloud features of the point cloud data respectively, and perform initial fusion;

[0011] Use spatial-channel-cross-modal multi-attention dynamic collaboration to perform weighted fusion on the multi-layer semantic features of the image and the point cloud features;

[0012] Predict long-term goals in the top-down map space based on the fused feature map and the semantic map, and plan a navigation path based on the current position and the long-term goals.

[0013] Optionally, in an embodiment of the present application, the constructing a semantic map based on visual observation environment information includes:

[0014] Obtain an RGB-D depth image and convert it into point cloud data;

[0015] Predict the semantic category corresponding to the point cloud based on the RGB-D depth image using the Mask R-CNN model;

[0016] Project the point cloud with semantic labels into a unified world coordinate system and extract multi-scale spatial features;

[0017] Reconstruct a semantic map based on the multi-scale spatial features.

[0018] Optionally, in an embodiment of the present application, the separately extracting the multi-layer semantic features of the image and the point cloud features from the RGB image and the point cloud data includes:

[0019] Extract the multi-layer semantic features of the RGB image based on attention weight calculation, global context aggregation, and feature modulation and enhancement;

[0020] Perform key point detection on the point cloud data and perform inter-channel feature recombination to obtain point cloud features.

[0021] Optionally, in an embodiment of the present application, the using spatial-channel-cross-modal multi-attention dynamic collaboration to perform weighted fusion on the multi-layer semantic features of the image and the point cloud features includes:

[0022] Enhance the multi-layer semantic features of the image and the point cloud features respectively to highlight important information;

[0023] Use channel attention to learn the dependencies between channels and use spatial attention to focus on key regions;

[0024] Use bilinear interpolation for cross-modal feature alignment;

[0025] Use the attention mechanism to determine the modal interaction weights and perform weighted fusion by combining spatial attention and channel attention.

[0026] Optionally, in an embodiment of the present application, the method further includes:

[0027] The feature map obtained by weighted fusion is concatenated with the direction encoding and the target encoding along the feature dimension, and non-linear transformation and feature enhancement are performed.

[0028] Optionally, in an embodiment of the present application, predicting long-term goals in the top-down map space based on the fused feature map and the semantic map, and the navigation path planning based on the current position and the long-term goals includes:

[0029] Predicting reachable long-term goal positions based on spatial semantic relationships and generating coarse-grained navigation instructions;

[0030] Determining a navigation feasible path based on the coarse-grained navigation instructions using the fast marching algorithm.

[0031] In a second aspect, the present application also provides an indoor robot navigation device based on multi-modal feature fusion. The device includes:

[0032] A semantic map construction module for constructing a semantic map based on visual observation environment information;

[0033] An object image preprocessing module for obtaining the RGB images of indoor scene objects, converting them into point cloud data, and preprocessing the RGB images and the point cloud data;

[0034] A feature extraction module for respectively extracting the multi-layer semantic features of the images and the point cloud features in the RGB images and the point cloud data, and performing initial fusion;

[0035] A multi-modal feature fusion module for performing weighted fusion on the multi-layer semantic features of the images and the point cloud features by using spatial-channel-cross-modal multi-attention dynamic collaboration;

[0036] A navigation planning module for predicting long-term goals in the top-down map space based on the fused feature map and the semantic map, and performing navigation path planning based on the current position and the long-term goals.

[0037] In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor executes the steps of the methods in the above various embodiments.

[0038] In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the methods in the above various embodiments are implemented.

[0039] The above-mentioned indoor robot navigation method based on multi-modal feature fusion first constructs a semantic map based on visual observation of environmental information. Then, it acquires the RGB images of indoor scene objects, converts them into point cloud data, and preprocesses the RGB images and point cloud data. After that, it extracts the multi-layer semantic features of the RGB images and the point cloud features respectively, and performs initial fusion. Then, it uses spatial-channel-cross-modal multi-attention dynamic collaboration to perform weighted fusion on the multi-layer semantic features of the images and the point cloud features. Finally, it predicts the long-term goals in the top-down map space based on the fused feature map and the semantic map, and plans the navigation path based on the current position and the long-term goals. That is to say, by dynamically fusing visual semantics and point cloud geometric information, a geometric-semantic collaborative perception mechanism is constructed, which significantly improves the positioning accuracy and recognition accuracy in complex scenarios. Based on the feature screening architecture of the attention mechanism, the navigation accuracy of the multi-modal system is realized by using a monocular RGB camera, effectively reducing the hardware cost and adapting to the needs of large-scale deployment. By adopting the cross-modal attention weight real-time generation technology, a millisecond-level dynamic response ability is formed, and the navigation path can be quickly adjusted in the face of sudden obstacles, synchronously ensuring safety and task continuity. Combining with an adaptive multi-source data fusion framework, the limitation of traditional manual calibration is broken through, and the geometric-semantic feature alignment and collaborative optimization are automatically realized, significantly shortening the deployment cycle. With low-cost hardware configuration, the robustness of environmental perception, the real-time performance of decision-making response and the ease of use of system integration in dynamic complex environments are comprehensively improved, especially suitable for application scenarios such as indoor service robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is an application environment diagram of an indoor robot navigation method based on multi-modal feature fusion in an embodiment;

[0041] Figure 2 It is a flowchart of an indoor robot navigation method based on multi-modal feature fusion in an embodiment;

[0042] Figure 3 It is a flowchart of semantic map construction in an embodiment;

[0043] Figure 4 It is a flowchart of multi-modal feature fusion in an embodiment;

[0044] Figure 5 It is a schematic diagram of experimental trajectory comparison in a test set scenario in an embodiment;

[0045] Figure 6 It is a structural block diagram of an indoor robot navigation device based on multi-modal feature fusion in an embodiment;

[0046] Figure 7It is the internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0047] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] A method for indoor robot navigation based on multi-modal feature fusion provided by an embodiment of the present application can be applied to, for example, Figure 1 the application environment shown. Among them, the terminal communicates with the server through the network. The data storage system can store the data that the server needs to process. The data storage system can be integrated on the server, or placed in the cloud or other network servers. Among them, the terminal can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0049] In one embodiment, as Figure 2 shown, a method for indoor robot navigation based on multi-modal feature fusion is provided. Taking the server in Figure 1 as an example, the method includes the following steps:

[0050] S201: Construct a semantic map based on visual observation of environmental information.

[0051] In the embodiment of the present application, first, the semantic categories and segmentations of the objects seen in the visual observation are predicted, the semantic subdivision in the first-person view is performed, and the first-person prediction is converted into a top-down map using distinguishable projections.

[0052] Specifically, in an embodiment of the present application, the constructing a semantic map based on visual observation of environmental information includes:

[0053] S301: Obtain an RGB-D depth image and convert it into point cloud data.

[0054] S303: Predict the semantic category corresponding to the point cloud based on the RGB-D depth image using the Mask R-CNN model.

[0055] S305: Project the point cloud with semantic labels into a unified world coordinate system and extract multi-scale spatial features.

[0056] S307: Reconstruct the semantic map based on the multi-scale spatial features.

[0057] In an embodiment of the present application, as Figure 3 shown, first, the depth image obtained by the RGB-D camera is converted into point cloud data, and the verified Mask R-CNN model is used to predict the semantic category of each point (such as objects, building structures, etc.) in the RGB image. Subsequently, the point cloud with semantic labels is projected into a unified world coordinate system through a differentiable geometric transformation module, and the voxelization method is used to discretize the three-dimensional space into regular grids, and multi-scale spatial features are extracted in the height dimension. The voxel data is deconstructed into three feature dimensions of obstacle distribution, semantic classification, and vertical structure through channel separation technology, and multi-view feature alignment is achieved through the spatial transformation layer in combination with temporal observation data. These multi-channel semantic features are processed by a denoising deep neural network with an encoder-decoder structure, and a high-resolution semantic probability map is reconstructed through deconvolution operations. The entire process is trained through end-to-end supervised learning, and the cross-entropy loss function (two-dimensional semantic segmentation and three-dimensional semantic map prediction) is used to simultaneously and synchronously optimize the two-dimensional semantic segmentation accuracy and the three-dimensional map reconstruction quality, and the differentiable projection module is used to achieve the backpropagation of semantic prediction errors to the geometric calculation link, forming a synergistic optimization semantic mapping process.

[0058] The specific cross-entropy loss function is as follows:

[0059] Two-dimensional semantic segmentation loss:

[0060]

[0061] where N is the total number of pixels, C is the total number of semantic categories, y i,c ∈{0, 1} is the true label of the i-th pixel, and p i,c is the predicted probability that the i-th pixel belongs to the category c by the model.

[0062] Three-dimensional semantic map prediction loss:

[0063]

[0064] where M is the total number of valid voxels, K = 2 for binary occupancy classification (obstacle presence: 0 = free, 1 = occupied), C is the total number of semantic categories, is the true label of whether the j-th voxel is occupied, is the predicted occupancy probability by the model, is the true semantic label of the j-th voxel, is the predicted semantic distribution by the model.

[0065] S203: Obtain the RGB image of the indoor scene objects, convert it into point cloud data, and preprocess the RGB image and the point cloud data.

[0066] In the embodiment of the present application, synchronize the RGB image of the indoor scene collected by the RGB camera, convert it into point cloud data, and perform cross-modal data preprocessing through voxel filtering denoising and adaptive histogram equalization respectively.

[0067] S205: Extract the multi-layer semantic features of the image and the point cloud features from the RGB image and the point cloud data respectively, and perform initial fusion.

[0068] In the embodiment of the present application, afterwards, perform multi-modal feature extraction. As Figure 4 shown, extract the multi-layer semantic features of the image and the point cloud features respectively. After the extraction is completed, directly add the features for initial fusion, which is expressed by the formula as follows:

[0069]

[0070] Among them, is the initial multi-layer semantic feature of the image, is the initial point cloud feature.

[0071] In an embodiment of the present application, the extracting the multi-layer semantic features of the image and the point cloud features from the RGB image and the point cloud data respectively includes:

[0072] S401: Extract the multi-layer semantic features of the RGB image based on attention weight calculation, global context aggregation, feature modulation and enhancement.

[0073] S403: Perform key point detection and inter-channel feature recombination based on the point cloud data to obtain point cloud features. In an embodiment of the present application, for the RGB image, first generate the query / key:

[0074] Q = Conv2D 1×1 (X) ∈ R B×D×H×W

[0075] K = Conv2D 1×1 (X) ∈ R B×D×H×W

[0076] Among them, Conv2D 1×1 is the convolution operation, X is the input data, B is the batch size, D is the number of output channels, H is the height of the feature map, and W is the width of the feature map.

[0077] Afterwards, perform attention weight calculation, and first perform vector normalization:

[0078]

[0079] Among them, L2 is normalization, and Normalize(Q, dim=-1) means to perform it on the tensor Q along the specified dimension (dim=-1).

[0080] Calculate the additive attention score: Among them, W g is a learnable parameter, N = H×W is the number of spatial positions, and d = token_dim.

[0081] After that, perform global context aggregation and context vector generation: Among them, A i is the attention weight matrix, is the projected feature vector.

[0082] Spatial replication: Among them, G is the result of context vector generation, and the Repeat operation replicates G N times along the spatial dimension N.

[0083] After that, perform feature modulation and enhancement, feature modulation: Among them, is the result of spatial replication, is the result of normalization, and ⊙ represents element-wise multiplication.

[0084] Projection and residual connection:

[0085]

[0086] Among them, Reshape(M) means to restore the spatial dimension of M, X represents the residual connection, and Conv2D 1×1 is the convolution operation.

[0087] Feature enhancement mechanism: Among them, Identity is the residual, is the attention modulation, which can retain both the original gradient and the attention-guided semantic gradient at the same time.

[0088] For point cloud data, first perform keypoint detection through spatial dimension retention and hierarchical feature learning. conv1 expands the channel dimension from 3 to the target dimension to capture local geometric patterns. Use a 1x1 convolution kernel for channel expansion:

[0089] F (1) = W1*X ′ +b1

[0090] Among them, W1 is the convolution kernel weight, X ′ is the result of dimension conversion, X ′= Permute(X, [0, 3, 1, 2]), where b1 is the bias term, and F (1) is the output shape, and F (1) ∈R N×D×W×H .

[0091] For the k-th channel at the position (i, j) of the output feature map:

[0092]

[0093] where X ′ n,c,i,j is the input feature map (n is the batch index, c is the input channel index, j is the spatial coordinate, W1[k, c, 0, 0] is the convolutional kernel weight (k is the output channel index, c is the input channel index, 0, 0 is the fixed position of the 1x1 convolutional kernel), and b1[k] is the bias term (k represents the bias value corresponding to the k-th output channel).

[0094] After that, conv2 performs inter-channel feature recombination to enhance the response output of high-dimensional feature maps for key regions such as edges and corners. Each spatial position corresponds to the geometric context of the original point cloud. Nonlinear recombination is performed in the feature channel dimension:

[0095] F (2) = W2 * F (1) + b2

[0096] where W2 is the convolutional kernel weight, b2 is the bias term, and F (2) is the output shape, and F (2) ∈R N×D×W×H .

[0097] Element-wise formula:

[0098]

[0099] where is the result of the previous channel expansion, W2 is the convolutional kernel weight, and b2[k] is the bias term.

[0100] Spatial resolution invariance. Since the convolutional kernel size = 1 and the stride = 1, the feature map size satisfies:

[0101]

[0102] where H in is the height or width of the input feature map, padding is the number of pixels padded at the edges of the input feature map, and stride is the stride at which the convolutional kernel slides on the input feature map. The entire process maintains spatial resolution invariance, and the feature vector at each spatial position contains both the original geometric information and the fused context semantics.

[0103] S207: Use spatial-channel-cross-modal multi-attention dynamic collaboration to perform weighted fusion on the multi-layer semantic features and point cloud features of the image.

[0104] In the embodiment of the present application, after obtaining the multi-layer semantic features and point cloud features of the image, use spatial-channel-cross-modal multi-attention dynamic collaboration to perform weighted fusion on the multi-layer semantic features and point cloud features of the image.

[0105] Specifically, in an embodiment of the present application, the use of spatial-channel-cross-modal multi-attention dynamic collaboration to perform weighted fusion on the multi-layer semantic features and point cloud features of the image includes:

[0106] S501: Enhance the multi-layer semantic features and point cloud features of the image respectively to highlight important information.

[0107] S503: Use channel attention to learn the dependencies between channels, and use spatial attention to focus on key regions.

[0108] S505: Use bilinear interpolation for cross-modal feature alignment.

[0109] S507: Use the attention mechanism to determine the modal interaction weights, and perform weighted fusion by combining spatial attention and channel attention.

[0110] In an embodiment of the present application, as Figure 4 shown, first, perform single-modal feature optimization, and enhance the image feature (F img ∈R N×C×H×W ) and the point cloud feature (F pc ∈R N×C×H×W ) respectively to highlight important information. For the image feature, introduce a multi-scale context enhancement module to extract a hybrid representation of local details and global semantics. For the point cloud feature, optimize the geometric structure perception through a voxel-based sparse convolutional network, and combine a density-aware sampling strategy to strengthen the point cloud distribution features in key regions.

[0111] After that, perform channel attention optimization, apply channel attention to the features of each modality to learn the dependencies between channels. Calculate the channel weight vector for each modality respectively:

[0112] W ca =σ(W2·δ(W1·GAP(F)))

[0113] where GAP(·) is global average pooling, W1 and W2 are the weight of the convolution kernels for dimensionality reduction-dimension increase, δ is the ReLU activation function, and σ is the Sigmoid function.

[0114] The optimized feature representation is:

[0115]

[0116] Among them, the symbol represents channel-by-channel multiplication.

[0117] The lightweight multi-scale channel attention (MS-CAM) is adopted. The spatial dimension is compressed through adaptive average pooling to generate global channel weights. 1×1 convolution is used to capture local channel relationships, avoiding the computational redundancy of large convolution kernels. After adding the global and local weights and normalizing, the adaptability to cross-scale features is enhanced.

[0118] After that, spatial attention optimization is carried out to further optimize the spatial dimension and focus on key regions. A spatial weight map is generated through channel statistics:

[0119]

[0120] Among them, Avg(·) and Max(·) are the mean and maximum values in the channel dimension, and [·;·] represents the concatenation operation. W conv is the weight of the 7×7 convolution kernel.

[0121] The optimized feature representation is:

[0122]

[0123] Calculate the mean and maximum values respectively in the channel dimension, generate a spatial weight map through 7×7 convolution after concatenation, upsample the high-level features and concatenate them with the low-level features to enhance the sensitivity to small targets.

[0124] After that, cross-modal feature alignment is carried out, and cross-modal alignment from the point cloud feature resolution to the image feature size is achieved through bilinear interpolation. The problem of inconsistent heterogeneous feature map sizes is solved to ensure spatial alignment.

[0125] Let the input feature map be F in ∈R^{H in ×W in}, which needs to be interpolated to the target size (H out , W out ). First, normalize the coordinate mapping:

[0126]

[0127] Among them, H out , W out are the height and width of the output image, and H in , W in are the height and width of the input image.

[0128] Then, determine the four points of the neighborhood:

[0129]

[0130] Among them, is the integer coordinate on the left side of the floating-point coordinate x, and x1 is the integer coordinate on the right side.

[0131] Calculate the interpolation weight:

[0132]

[0133] The bilinear interpolation formula is:

[0134]

[0135] Among them, F in [x a , x b is the pixel value of the input image at the coordinate (x a , x b ). x a and x b need to be processed at the boundary (take the nearest pixel value when out of range).

[0136] Adjust the receptive field of the image features by deformable convolution, match the spatial resolution of the point cloud features through sparse interpolation, construct a cross-modal graph network, and establish a feature mapping relationship between modalities using the cosine similarity between nodes.

[0137] After that, perform cross-modal dynamic fusion, learn the complementary relationship between modalities through the attention mechanism, and dynamically allocate weights.

[0138] Concatenate the features of the two modalities and generate the interaction weight:

[0139]

[0140] Among them, σ is the Sigmoid function, W is the parameter matrix, δ is the Softmax function, represents the concatenation of the image and point cloud features.

[0141] Combine spatial attention and channel attention:

[0142] F attn = σ(F ca ) · σ(F sa )

[0143] Among them, σ is the Sigmoid function, F ca is the optimized result of channel attention, and F sa is the optimized result of spatial attention.

[0144] The weighted fused feature is:

[0145]

[0146] Among them, F initial is the initial fusion result, F attn is the result of combining spatial attention and channel attention, and W cross is the interaction weight.

[0147] After that, it is adjusted through the main convolutional network:

[0148]

[0149] Among them, F fused is the weighted fusion feature, is the convolutional conv operation.

[0150] In this embodiment, through the three-level attention linkage of space-channel-cross-modal, local details, channel dependencies, and modal complementarity are captured simultaneously to solve the problems of traditional single-modal or linear fusion. Automatically adjust the weights of each modality based on task requirements, making it more robust in complex scenarios (such as occlusion and lighting changes), with an accuracy improvement of 5-8% compared to the fixed-weight fusion model. The lightweight design reduces the number of parameters by 40%, making it suitable for deployment on edge devices. The multi-scale spatial attention and cross-modal alignment mechanism work together to significantly improve the detection accuracy of small targets.

[0151] In an embodiment of the present application, the method further includes:

[0152] Concatenate the feature map obtained by weighted fusion with the direction encoding and target encoding along the feature dimension, and perform non-linear transformation and feature enhancement.

[0153] In an embodiment of the present application, the features after fusion are concatenated with the direction encoding and target encoding along dimension 1 (feature dimension) to utilize various information (such as current state, direction, target) to improve the decision-making or prediction ability. And the concatenated data is subjected to non-linear transformation through a fully connected layer and the ReLU activation function to provide the relationship between features for model learning. And perform feature enhancement to finally optimize the fused features and improve the discriminability. Specifically, iterative attention feature fusion is adopted, and channel / space attention and feature superposition are alternately executed multiple times to gradually eliminate fusion noise.

[0154] S209: Predict long-term goals in the top-down map space based on the fused feature map and the semantic map, and plan a navigation path based on the current position and long-term goals.

[0155] In the embodiment of the present application, the fused feature map and the semantic map are used to predict long-term goals in the top-down map space, so as to plan a path from the current position to the long-term goal to complete the navigation decision, and finally complete the target navigation task.

[0156] Specifically, in an embodiment of the present application, predicting long-term goals in the top-down map space based on the fused feature map and the semantic map, and performing navigation path planning based on the current position and the long-term goals includes:

[0157] S601: Predict reachable long-term goal positions based on spatial semantic relationships and generate coarse-grained navigation instructions.

[0158] S603: Use the fast marching algorithm to determine a navigable path based on the coarse-grained navigation instructions.

[0159] In an embodiment of the present application, first, at the global planning layer, a top-down environmental perception is constructed by fusing multi-modal feature maps and semantic maps, reachable long-term goal positions are predicted based on spatial semantic relationships, and coarse-grained navigation instructions are generated. Then, at the local execution layer, the fast marching algorithm is used as a deterministic local policy to update the obstacle channel information of the semantic map at a high frequency of 10 Hz, and continuously optimize the navigable path from the current position to the global goal.

[0160] As Figure 5 shown, it is a schematic diagram of the comparison of experimental trajectories in the test set scenario. It can be seen that this hybrid architecture combining global goal prediction with semantic reasoning and real-time replanning can not only ensure the rationality of the long-term navigation direction through feature fusion in a dynamic environment, but also achieve centimeter-level obstacle avoidance accuracy through high-frequency local path optimization. Its navigation stability is better than that of pure learning-based strategies, and finally, the goal navigation task in complex scenarios is completed through the global-local decision-making closed loop.

[0161] In the above indoor robot navigation method based on multi-modal feature fusion, first, a semantic map is constructed based on visual observation of environmental information; then, an RGB image of indoor scene objects is obtained, converted into point cloud data, and the RGB image and point cloud data are preprocessed; then, multi-layer image semantic features and point cloud features in the RGB image and point cloud data are respectively extracted and initially fused; then, spatial-channel-cross-modal multi-attention dynamic collaboration is used to perform weighted fusion on the multi-layer image semantic features and point cloud features; finally, a long-term goal in the top-down map space is predicted based on the fused feature map and the semantic map, and a navigation path is planned based on the current position and the long-term goal. That is to say, by dynamically fusing visual semantics and point cloud geometric information, a geometric-semantic collaborative perception mechanism is constructed to significantly improve the positioning accuracy and recognition accuracy in complex scenarios; based on the attention mechanism feature screening architecture, the navigation accuracy of the multi-modal system is achieved using a monocular RGB camera, effectively reducing the hardware cost and adapting to the requirements of large-scale deployment; the cross-modal attention weight real-time generation technology is adopted to form a millisecond-level dynamic response ability, and the navigation path can be quickly adjusted in the face of sudden obstacles, synchronously ensuring safety and task continuity; combined with an adaptive multi-source data fusion framework, the limitation of traditional manual calibration is broken through, and geometric-semantic feature alignment and collaborative optimization are automatically realized, significantly shortening the deployment cycle. With low-cost hardware configuration, the robustness of environmental perception, the real-time nature of decision-making response, and the ease of system integration in dynamic and complex environments are comprehensively improved, especially suitable for application scenarios such as indoor service robots.

[0162] It should be understood that although the steps in the flowcharts involved in the above embodiments are sequentially shown according to the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless there is a clear indication in this article, there is no strict order limit for the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages do not necessarily have to be executed at the same moment, but can be executed at different moments, and the execution order of these steps or stages does not necessarily have to be sequential, but can be executed alternately or in turn with at least some of the steps or stages in other steps or other steps.

[0163] Based on the same inventive concept, an embodiment of the present application further provides an indoor robot navigation device based on multi-modal feature fusion for implementing the indoor robot navigation method based on multi-modal feature fusion involved above. The implementation solutions provided by this device to solve problems are similar to those recorded in the above method. Therefore, the specific limitations in one or more embodiments of the indoor robot navigation device based on multi-modal feature fusion provided below can refer to the limitations on the indoor robot navigation method based on multi-modal feature fusion in the above text, and will not be elaborated here.

[0164] In one embodiment, as Figure 6 shown, an indoor robot navigation device 600 based on multi-modal feature fusion is provided, including: a semantic map construction module 601, an object image preprocessing module 603, a feature extraction module 605, a multi-modal feature fusion module 607, and a navigation planning module 609, where:

[0165] The semantic map construction module 601 is used to construct a semantic map based on visual observation environment information;

[0166] The object image preprocessing module 603 is used to obtain the RGB image of indoor scene objects, convert it into point cloud data, and preprocess the RGB image and the point cloud data;

[0167] The feature extraction module 605 is used to extract the image multi-layer semantic features and point cloud features in the RGB image and the point cloud data respectively, and perform initial fusion;

[0168] The multi-modal feature fusion module 607 is used to perform weighted fusion on the image multi-layer semantic features and the point cloud features by using spatial-channel-cross-modal multi-attention dynamic collaboration;

[0169] The navigation planning module 609 is used to predict long-term goals in the top-down map space based on the fused feature map and the semantic map, and perform navigation path planning based on the current position and the long-term goals.

[0170] In an embodiment of the present application, the semantic map construction module is further used for:

[0171] Obtain an RGB-D depth image and convert it into point cloud data;

[0172] Based on the RGB-D depth image, use the Mask R-CNN model to predict the semantic category corresponding to the point cloud;

[0173] Project the point cloud with semantic labels into a unified world coordinate system, and extract multi-scale spatial features;

[0174] Reconstruct the semantic map based on the multi-scale spatial features.

[0175] In one embodiment of the present application, the feature extraction module is further configured to:

[0176] Extract the multi-layer semantic features of the RGB image based on attention weight calculation, global context aggregation, feature modulation, and enhancement;

[0177] Perform key point detection on the point cloud data and perform inter-channel feature recombination to obtain point cloud features.

[0178] In one embodiment of the present application, the multi-modal feature fusion module is further configured to:

[0179] Enhance the multi-layer semantic features of the image and the point cloud features respectively to highlight important information;

[0180] Adopt channel attention to learn the inter-channel dependence relationship, and adopt spatial attention to focus on key regions;

[0181] Perform cross-modal feature alignment using bilinear interpolation;

[0182] Adopt an attention mechanism to determine the modal interaction weight, and perform weighted fusion by combining spatial attention and channel attention.

[0183] In one embodiment of the present application, the method further includes:

[0184] Concatenate the feature map obtained by weighted fusion with the direction encoding and the target encoding along the feature dimension, and perform non-linear transformation and feature enhancement.

[0185] In one embodiment of the present application, the navigation planning module is further configured to:

[0186] Predict reachable long-term target positions based on spatial semantic relationships and generate coarse-grained navigation instructions;

[0187] Determine a navigation feasible path based on the coarse-grained navigation instructions using a fast marching algorithm.

[0188] Each module in the above indoor robot navigation device based on multi-modal feature fusion can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so as to facilitate the processor to call and execute the operations corresponding to the above-mentioned modules.

[0189] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as Figure 7As shown in the figure. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be achieved through WIFI, mobile cellular networks, NFC (Near Field Communication) or other technologies. When the computer program is executed by the processor, it realizes an indoor robot navigation method based on multi-modal feature fusion. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, a trackball or a touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0190] Those skilled in the art can understand that Figure 7 the structure shown in the figure is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0191] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are realized.

[0192] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the steps in the above method embodiments are realized.

[0193] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, the steps in the above method embodiments are realized.

[0194] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0195] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include Read-Only Memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., and are not limited thereto.

[0196] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0197] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. An indoor robot navigation method based on multi-modal feature fusion, characterized in that, The method includes: Constructing a semantic map based on visual observation environment information; Obtaining the RGB image of indoor scene objects, converting it into point cloud data, and preprocessing the RGB image and the point cloud data; Respectively extracting the multi-layer image semantic features and point cloud features in the RGB image and the point cloud data, and performing initial fusion; Using spatial-channel-cross-modal multi-attention dynamic collaboration to perform weighted fusion on the multi-layer image semantic features and point cloud features; Predicting long-term goals in the top-down map space based on the fused feature map and the semantic map, and performing navigation path planning based on the current position and the long-term goals.

2. The indoor robot navigation method based on multi-modal feature fusion according to claim 1, characterized in that, The constructing a semantic map based on visual observation environment information includes: Obtaining an RGB-D depth image and converting it into point cloud data; Predicting the semantic category corresponding to the point cloud based on the RGB-D depth image using the Mask R-CNN model; Projecting the point cloud with semantic labels into a unified world coordinate system and extracting multi-scale spatial features; Reconstructing a semantic map based on the multi-scale spatial features.

3. A method for indoor robot navigation based on multi-modal feature fusion according to claim 1, characterized in that, The respectively extracting the multi-layer image semantic features and point cloud features in the RGB image and the point cloud data includes: Extracting the multi-layer image semantic features of the RGB image based on attention weight calculation, global context aggregation, feature modulation and enhancement; Performing key point detection and inter-channel feature recombination on the point cloud data to obtain point cloud features.

4. A method for indoor robot navigation based on multi-modal feature fusion according to claim 1, characterized in that, The using spatial-channel-cross-modal multi-attention dynamic collaboration to perform weighted fusion on the multi-layer image semantic features and point cloud features includes: Respectively enhancing the multi-layer image semantic features and point cloud features to highlight important information; Using channel attention to learn the inter-channel dependence relationship, and using spatial attention to focus on key regions; Performing cross-modal feature alignment using bilinear interpolation; Using an attention mechanism to determine the modal interaction weight, and performing weighted fusion by combining spatial attention and channel attention.

5. The indoor robot navigation method based on multi-modal feature fusion according to claim 4, wherein, The method further includes: Concatenating the feature map obtained by weighted fusion with direction encoding and target encoding along the feature dimension, and performing non-linear transformation and feature enhancement.

6. The indoor robot navigation method based on multi-modal feature fusion according to claim 1, characterized in that, The predicting long-term goals in the top-down map space based on the fused feature map and the semantic map, and performing navigation path planning based on the current position and the long-term goals includes: Predicting the reachable long-term goal position based on the spatial semantic relationship and generating a coarse-grained navigation instruction; Determining a navigation feasible path based on the coarse-grained navigation instruction using the fast marching algorithm.

7. An indoor robot navigation device based on multi-modal feature fusion, characterized in that, The device includes: A semantic map construction module for constructing a semantic map based on visual observation environment information; An object image preprocessing module for obtaining the RGB image of indoor scene objects, converting it into point cloud data, and preprocessing the RGB image and the point cloud data; A feature extraction module for respectively extracting the multi-layer image semantic features and point cloud features in the RGB image and the point cloud data, and performing initial fusion; A multi-modal feature fusion module for using spatial-channel-cross-modal multi-attention dynamic collaboration to perform weighted fusion on the multi-layer image semantic features and point cloud features; The navigation planning module is used to predict long-term goals in the top-down map space based on the fused feature map and the semantic map, and perform navigation path planning based on the current position and the long-term goals.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 6.

Images