Robot sensing and decision-making method based on lightweight multi-modal large model

Abstract

The invention relates to a robot sensing and decision-making method based on a lightweight multi-modal large model. The method comprises the following steps: constructing a semantic voxel map; collecting multi-modal data based on the semantic voxel map and preprocessing the multi-modal data, wherein the multi-modal data comprises visual data, point cloud data and a target semantic tag; performing feature extraction on the preprocessed multi-modal data, and performing dynamic cross-modal attention fusion to obtain multi-modal fusion features; inputting the multi-modal data into a lightweight multi-modal large model at the same time, and performing semantic analysis to obtain global space object semantic description; and based on the global space object semantic description, the target and direction embedding vector and the multi-modal fusion feature, a reinforcement learning algorithm is adopted to carry out hierarchical navigation decision making to obtain a target decision, and the target and direction embedding vector is a preprocessed target semantic tag. And the accuracy, timeliness and adaptability of robot perception and navigation decision making in a complex scene are improved.

Description

Technical Field

[0001] The present application relates to the field of robotics and artificial intelligence technology, and in particular to a robot perception and decision-making method based on a lightweight multimodal large model. Background Art

[0002] Target navigation technology enables intelligent agents (such as robots and autonomous vehicles) to autonomously perceive and understand their surroundings in complex environments, plan optimal paths based on pre-defined goals, and achieve safe and efficient movement from a starting point to a destination. This technology has broad application prospects in a variety of fields, including logistics and transportation, intelligent security, and unmanned delivery.

[0003] At present, target navigation technology mainly includes the following three implementation methods: traditional visual navigation technology uses monocular or binocular cameras to collect environmental images, builds scene maps through visual SLAM algorithms, and realizes navigation in combination with path planning algorithms; multimodal sensor fusion technology integrates data from multiple sensors such as lidar and RGB-D cameras, and uses Kalman filtering or early feature stitching technology to enhance environmental perception capabilities; semantically enhanced navigation technology constructs scene semantic maps through object detection, and combines target semantic coding to guide navigation decisions.

[0004] However, the above existing technologies all have certain defects: traditional visual navigation technology has poor adaptability in dynamic scenes, and lighting changes and interference from dynamic objects can easily cause visual SLAM failure and trigger positioning drift, and the lack of semantic understanding ability can easily cause path planning to violate physical rules; multimodal sensor fusion technology faces the problem of modal alignment, and it is difficult to handle feature conflicts caused by the resolution difference between point clouds and visual data. At the same time, the multi-branch convolutional network has high computational complexity, and the inference delay on the embedded platform exceeds 200ms, which cannot meet real-time requirements; semantically enhanced navigation technology relies too much on manually preset object categories and attributes, and the system fails when facing unknown categories. The semantic understanding method based on large models occupies more than 8GB of video memory on edge devices, making it difficult to achieve large-scale deployment.

[0005] Therefore, in related technologies, there is an urgent need for a method that can improve the accuracy, timeliness and adaptability of robot perception and navigation decisions in complex scenarios. Summary of the Invention

[0006] Based on this, it is necessary to address the above technical problems and provide a robot perception and decision-making method based on a lightweight multimodal large model that can improve the accuracy, timeliness and adaptability of robot perception and navigation decisions in complex scenarios.

[0007] In a first aspect, the present application provides a robot perception and decision-making method based on a lightweight multimodal large model. The method comprises: Construct semantic voxel maps; Collecting and preprocessing multimodal data based on the semantic voxel map, the multimodal data including visual data, point cloud data and target semantic labels; Perform feature extraction on the preprocessed multimodal data and perform dynamic cross-modal attention fusion to obtain multimodal fusion features; The multimodal data are simultaneously input into a lightweight multimodal large model for semantic analysis to obtain a semantic description of global spatial objects; Based on the global spatial object semantic description, target and direction embedding vectors and multimodal fusion features, a reinforcement learning algorithm is used to perform hierarchical navigation decision-making to obtain a target decision, wherein the target and direction embedding vectors are target semantic labels after preprocessing.

[0008] Optionally, in one embodiment of the present application, constructing a semantic voxel map includes: A lightweight real-time semantic segmentation network is used to perform pixel-by-pixel semantic analysis and map the semantically labeled depth point cloud to a global coordinate system to construct an initial 3D semantic representation. The initial 3D semantic representation is decomposed into three subspaces using a multi-resolution voxel grid partitioning strategy, and temporal pose estimation, 3D non-rigid registration, and spatiotemporal feature fusion are performed. The 3D voxel features are compressed into a 2D bird's-eye view representation and an online map updating mechanism is integrated to obtain a semantic voxel map.

[0009] Optionally, in one embodiment of the present application, extracting features from the preprocessed multimodal data includes: The query-key attention mechanism, channel separation and normalization are used on the visual feature map to generate multi-layer semantic features of the image; The point cloud feature map is voxelized or projected, and dimension adjustment and convolution feature extraction are performed to generate high-dimensional point cloud features.

[0010] Optionally, in one embodiment of the present application, the dynamic cross-modal attention fusion includes: Initial fusion of the extracted image multi-layer semantic features and point cloud features; Bilinear interpolation is used for cross-modal feature alignment; Channel attention is used to learn inter-channel dependencies, and spatial attention is used to focus on key areas; The attention mechanism is used to determine the modal interaction weights, and spatial attention and channel attention are combined for weighted fusion.

[0011] Optionally, in one embodiment of the present application, the simultaneously inputting the multimodal data into the lightweight multimodal large model for semantic parsing to obtain a semantic description of the global spatial object includes: Preprocessing the multimodal data; A differentiable bijective projection layer is designed to perform cross-modal feature fusion on the pre-processed multimodal data to obtain fused multimodal features. Constructing a hierarchical knowledge distillation framework to determine the structure of a three-dimensional scene graph based on the fused multimodal features; A global spatial object semantic description is generated based on the three-dimensional scene graph structure through a relational graph reasoning network.

[0012] Optionally, in one embodiment of the present application, the layered navigation decision is performed using a reinforcement learning algorithm based on the global spatial object semantic description, the target and direction embedding vector, and the multimodal fusion feature, and the target decision includes: Performing action space modeling based on the global space object semantic description and target embedding vector; Determining local real-time obstacle avoidance constraints based on the multimodal fusion features and the direction embedding vector; A hierarchical navigation decision is made in the action space in combination with the real-time obstacle avoidance constraint.

[0013] Optionally, in one embodiment of the present application, performing a hierarchical navigation decision in the action space in combination with the real-time obstacle avoidance constraint includes: Determine the total loss function based on PPO policy gradient loss, value function loss and entropy regularization term; Performing end-to-end optimization of the policy network based on the total loss function; The path length weighted success rate indicator is used to evaluate the efficiency and success rate of hierarchical navigation decisions.

[0014] In a second aspect, the present application also provides a robot perception and decision-making device based on a lightweight multimodal large model. The device comprises: Semantic voxel map construction module, used to construct semantic voxel map; A multimodal data acquisition and preprocessing module, configured to acquire and preprocess multimodal data based on the semantic voxel map, wherein the multimodal data includes visual data, point cloud data, and target semantic labels; The multimodal feature extraction and fusion module is used to extract features from the preprocessed multimodal data and perform dynamic cross-modal attention fusion to obtain multimodal fusion features; A structured spatial semantic modeling module is used to simultaneously input the multimodal data into a lightweight multimodal large model, perform semantic analysis, and obtain a semantic description of global spatial objects; The hierarchical navigation decision module is used to make hierarchical navigation decisions based on the semantic description of the global spatial object, the target and direction embedding vectors and the multimodal fusion features, using a reinforcement learning algorithm to obtain a target decision, wherein the target and direction embedding vectors are the target semantic labels after preprocessing.

[0015] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program and the processor executes the steps of the method described in each of the above embodiments.

[0016] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in each of the above embodiments.

[0017] The aforementioned robot perception and decision-making method based on a lightweight multimodal large model first constructs a semantic voxel map. Based on the semantic voxel map, multimodal data is collected and preprocessed. The multimodal data includes visual data, point cloud data, and target semantic labels. Feature extraction is then performed on the preprocessed multimodal data, followed by dynamic cross-modal attention fusion to obtain multimodal fusion features. The multimodal data is then simultaneously input into the lightweight multimodal large model for semantic parsing to obtain a global spatial object semantic description. Finally, a reinforcement learning algorithm is used to perform hierarchical navigation decisions based on the global spatial object semantic description, target and direction embedding vectors, and multimodal fusion features, resulting in a target decision. The target and direction embedding vectors represent the preprocessed target semantic labels. In other words, the dynamic cross-modal attention mechanism adaptively fuses point cloud, visual, and semantic features, eliminating resolution and semantic-level differences in multimodal data and significantly improving real-time decision-making efficiency. Through a lightweight, multimodal large model, semantic parsing of visual and point cloud data is achieved, generating structured descriptions containing object attributes and physical constraints. While deepening the depth and accuracy of scene understanding, the system complexity is reduced by optimizing computational logic, achieving efficient computing power allocation. Furthermore, the physical constraint encoding and object attributes output by the large model are used as inputs for action decisions, guiding the strategy to generate paths that conform to real-world rules. This can accurately avoid invalid collisions and significantly enhance the physical rationality of the decision-making process. This improves the accuracy, timeliness, and adaptability of the robot's perception and navigation decisions in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 1. A diagram of an application environment of a robot perception and decision-making method based on a lightweight multimodal large model in one embodiment; Figure 21 is a flow chart of a robot perception and decision-making method based on a lightweight multimodal large model in one embodiment; Figure 3 A schematic diagram of a robot navigation process in one embodiment; Figure 4 A schematic diagram showing a comparison of success rates of different target objects in a Gibson test set according to an embodiment; Figure 5 A schematic diagram showing a comparison of the path length weighted success rates (SPLs) of different target objects in a Gibson test set according to one embodiment; Figure 6 A schematic diagram of the change in object path length weighted success rate (SPL) for different numbers of verification rounds in a Gibson test set in one embodiment; Figure 7 Schematic diagram of experimental trajectory comparison results in a Gibson test set scenario in one embodiment; Figure 8 1 is a structural block diagram of a robot perception and decision-making device based on a lightweight multimodal large model in one embodiment; Figure 9 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0020] The embodiment of the present application provides a robot perception and decision-making method based on a lightweight multimodal large model, which can be applied to Figure 1 In the application environment shown, the terminal communicates with the server through the network. The data storage system can store data that the server needs to process. The data storage system can be integrated on the server or placed on the cloud or other network servers. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0021] In one embodiment, Figure 2 As shown in the figure, a robot perception and decision-making method based on a lightweight multimodal large model is provided. Figure 1 The following steps are used as an example to illustrate the server in the example: S201: Constructing a semantic voxel map.

[0022] In the examples of this application, semantic segmentation and 3D point cloud mapping, 3D spatial feature extraction and decomposition, spatiotemporal feature fusion and registration, and map projection are performed through environmental interaction to create a semantic voxel map. While maintaining end-to-end differentiability, collaborative optimization from first-person perception to global semantic maps is achieved, demonstrating significant advantages in the three key dimensions of obstacle detection, scene understanding, and spatial modeling.

[0023] Specifically, in one embodiment of the present application, constructing a semantic voxel map includes: S301: Use a lightweight real-time semantic segmentation network to perform pixel-by-pixel semantic analysis, and map the depth point cloud with semantic labels to the global coordinate system to construct an initial 3D semantic representation.

[0024] S303: Decomposing the initial three-dimensional semantic representation into three subspaces by adopting a multi-resolution voxel grid partitioning strategy, and performing temporal pose estimation, three-dimensional space non-rigid registration and spatiotemporal feature fusion.

[0025] S305: compressing the 3D voxel features into a 2D bird's-eye view representation, integrating an online map update mechanism, and obtaining a semantic voxel map.

[0026] In one embodiment of this application, a lightweight real-time semantic segmentation network achieves pixel-by-pixel semantic parsing. A differentiable geometric transformation module is used to map semantically labeled depth point clouds to a global coordinate system, constructing a 3D semantic representation of a dynamic environment. For 3D feature extraction, a multi-resolution voxel gridding strategy is employed. A hierarchical feature pyramid structure is constructed in the height dimension to effectively capture spatial semantic information, from ground-level details to high-altitude structures. The raw voxel features are decomposed into three orthogonal subspaces: an obstacle probability field, a semantic category distribution field, and a vertical structure field. The obstacle field generates a binary occupancy representation using a learnable threshold function, the semantic field uses a softmax probability distribution to describe multi-category affiliation, and the structure field captures spatial continuity in the vertical dimension using a direction encoder. An implicit pose estimation network then predicts temporal view transformation parameters, and a 3D spatial transformer is used to perform non-rigid registration of historical observation features. Finally, a gated recurrent unit achieves spatiotemporal feature fusion. At the output level, a differentiable map projection layer is proposed to compress 3D voxel features into a 2D bird's-eye view representation. An online map update mechanism is integrated to ensure real-time map performance in dynamic environments.

[0027]

[0028] in, is the center coordinate of a voxel, cloud is the point cloud set, is the voxelized kernel width, are the features of all points in the point cloud, is a single point in the point cloud.

[0029] S203: Collecting and preprocessing multimodal data based on the semantic voxel map, wherein the multimodal data includes visual data, point cloud data, and target semantic labels.

[0030] In an embodiment of the present application, visual data RGB images, point cloud data and target semantic labels are collected and preprocessed based on the semantic voxel map. Specifically, standardized preprocessing is performed on the RGB image, and spatial alignment between multi-resolution images is achieved through bilinear interpolation, and a deep separable convolutional network is used to extract multi-scale feature maps. For point cloud data, the original point cloud is rasterized into a 2D feature map, retaining the three-dimensional spatial distribution information. Coordinate normalization and noise filtering are performed to obtain a visual feature map and a point cloud feature map. Target encoding and direction encoding are performed for the target semantic label, and the text target is mapped to an 8-dimensional vector through word embedding and the robot orientation is encoded using a 72-dimensional direction embedding table to obtain the target and direction embedding vectors. In this way, while maintaining the characteristics of each modal data, the scale consistency of the cross-modal feature space is achieved.

[0031] S205: Extract features from the pre-processed multimodal data and perform dynamic cross-modal attention fusion to obtain multimodal fusion features.

[0032] In an embodiment of the present application, after multimodal data preprocessing, visual feature extraction and point cloud feature extraction are performed respectively, and dynamic cross-modal attention fusion is performed to obtain the fused semantic and geometric features of different modalities, and the target category is mapped to the semantic space through the Embedding layer, combined with the semantic projection of the pre-trained language model to form a multimodal feature input.

[0033] Specifically, in one embodiment of the present application, extracting features from the pre-processed multimodal data includes: S401: Apply query-key attention mechanism, channel separation and normalization to the visual feature map to generate multi-layer semantic features of the image.

[0034] S403: voxelize or project the point cloud feature map, perform dimension adjustment and convolution feature extraction to generate high-dimensional point cloud features.

[0035] In one embodiment of the present application, for visual features, a lightweight convolutional network is used to output multi-scale feature maps, and a query-key attention mechanism, channel separation and normalization are used to generate global context, improve the stability of attention weights, and enhance visual features. The query-key attention mechanism is expressed as:

[0036]

[0037]

[0038] in, is the input feature, B is the batch size, D is the embedding dimension, H and W are the height and width of the feature map, is a 1×1 convolution kernel, HW is to flatten the spatial dimension of the feature map into a one-dimensional sequence, Calculate the similarity between the query at each position and the keys at all positions, To normalize along the dimension. Then perform residual connection: .

[0039] For point cloud features, after voxelization or projection transformation, the point cloud is converted into a regular two-dimensional grid structure. , where B is the batch size, HxW is the grid resolution, and each grid cell stores the XYZ mean of the point cloud in the area. The dimensions are adjusted from [B, H, W, 3] to [B, 3, H, W], and the 1×1 convolution feature extraction is performed ( and ), the final output is a high-dimensional point cloud feature map , where the eigenvector corresponding to each spatial position (i, j) is [:,:,i,j] Encodes the geometric properties of the grid area.

[0040] In one embodiment of the present application, the dynamic cross-modal attention fusion includes: S501: Initially fuse the extracted multi-layer semantic features and point cloud features of the image.

[0041] S503: Use bilinear interpolation to perform cross-modal feature alignment.

[0042] S505: Use channel attention to learn the dependencies between channels and use spatial attention to focus on key areas.

[0043] S507: Use the attention mechanism to determine the modal interaction weight, and combine spatial attention and channel attention for weighted fusion.

[0044] In one embodiment of the present application, first, the multi-layer semantic features of the image and the point cloud features are directly added together for initial fusion, which can be expressed as follows:

[0045] in, is the multi-layer semantic feature of the initial image, is the initial point cloud feature.

[0046] Afterwards, cross-modal feature alignment is performed, using bilinear interpolation to achieve cross-modal alignment from point cloud feature resolution to image feature size. This solves the problem of inconsistent sizes of heterogeneous feature maps and ensures spatial alignment.

[0047] Assume the input feature map is , need to be interpolated to the target size ( , ), first normalize the coordinate mapping:

[0048] in, 、 are the height and width of the output image, 、 are the height and width of the input image.

[0049] Then determine four areas:

[0050] in, is the integer coordinate to the left of the floating point coordinate x, are integer coordinates on the right.

[0051] Calculate the interpolation weights:

[0052] The bilinear interpolation formula is:

[0053] in, For the input image at coordinates ( ), and Boundary processing is required (take the nearest pixel value when out of range).

[0054] After that, channel attention optimization is performed to apply channel attention to the features of each modality and learn the dependencies between channels. The channel weight vector is calculated for each modality separately:

[0055] in, is the global average pooling, 、 is the convolution kernel weight of dimensionality reduction-dimensionality increase, is the ReLU activation function, is the Sigmoid function.

[0056] The optimized feature representation is:

[0057] Among them, the symbol Represents channel-by-channel multiplication.

[0058] Lightweight Multi-Scale Channel Attention (MS-CAM) is employed to compress spatial dimensions through adaptive average pooling and generate global channel weights. 1×1 convolution is used to capture local channel relationships, avoiding computational redundancy associated with large convolution kernels. Global and local weights are summed and normalized to enhance adaptability to cross-scale features.

[0059] After that, spatial attention optimization is performed to further optimize the spatial dimension and focus on key areas. The spatial weight map is generated through channel statistics:

[0060] in, and is the mean and maximum value of the channel dimension, Represents a splicing operation is the 7×7 convolution kernel weight.

[0061] The optimized feature representation is:

[0062] The mean and maximum values ​​are calculated in the channel dimension respectively, and after splicing, a 7×7 convolution is performed to generate a spatial weight map. The high-level features are upsampled and spliced ​​with the low-level features to enhance the sensitivity to small targets.

[0063] Finally, cross-modal dynamic fusion is performed, the complementary relationship between modalities is learned through the attention mechanism, and weights are dynamically assigned.

[0064] Concatenate the two modal features and generate interaction weights:

[0065] in, for function, W is the parameter matrix, for function, Indicates the stitching of image and point cloud features.

[0066] Combining spatial attention and channel attention:

[0067] in, for function, For channel attention optimization results, Optimizing results for spatial attention.

[0068] The features after weighted fusion are:

[0069] in, Initial fusion results, The result of combining spatial attention and channel attention, is the interaction weight.

[0070] S207: Input the multimodal data into the lightweight multimodal large model simultaneously, perform semantic analysis, and obtain a semantic description of the global spatial object.

[0071] In the embodiment of the present application, a lightweight multimodal large model is optimized based on the Qwen architecture. Through a lightweight design with a small number of parameters, the point cloud is first denoised, voxelized and semantically segmented, and the texture features of the RGB image are extracted simultaneously. Cross-modal feature alignment is achieved through a differentiable projection layer. A dynamic sparse attention mechanism is adopted to retain only 5% of the key attention connections when fusing the geometric features of the point cloud with the visual semantic features, significantly reducing the computational complexity. Through hierarchical feature distillation technology, multimodal data is mapped into a three-dimensional scene graph containing object attributes (size, position) and spatial relationships (distance, topology). At the same time, physical constraints such as non-passable walls and furniture movement rules are injected to generate an interpretable structured semantic description of global spatial objects.

[0072] Specifically, in one embodiment of the present application, the step of simultaneously inputting the multimodal data into a lightweight multimodal large model for semantic parsing to obtain a semantic description of a global spatial object includes: S601: Preprocess the multimodal data.

[0073] S603: Design a differentiable bijective projection layer to perform cross-modal feature fusion on the pre-processed multimodal data to obtain fused multimodal features.

[0074] S605: Construct a hierarchical knowledge distillation framework to determine the three-dimensional scene graph structure based on the fused multimodal features.

[0075] S607: Generate a global space object semantic description based on the three-dimensional scene graph structure through a relational graph reasoning network.

[0076] In one embodiment of the present application, first, a hybrid denoising process is implemented on the original point cloud data, a non-uniform voxelization algorithm is used to construct a hierarchical spatial index, and three-dimensional geometric features are extracted through a multi-scale semantic segmentation network. In two-dimensional modal processing, a deformable convolutional network is used to extract multi-level texture features of RGB images. Afterwards, a differentiable bijective projection layer is designed to establish a three-dimensional-two-dimensional feature space mapping relationship, and a cross-modal alignment loss function based on optimal transmission theory is introduced to ensure the metric consistency of geometric features and visual semantic features. The pre-processed multi-modal data is subjected to cross-modal feature fusion to obtain fused multi-modal features. In the feature fusion stage, a dynamic sparse attention mechanism is adopted to adaptively screen key feature interaction nodes through a learnable gated network, and the attention connection density is controlled within 5%, reducing the computational complexity. Afterwards, a hierarchical knowledge distillation framework is constructed, and the multi-modal features are encoded into a three-dimensional scene graph structure through a graph neural network. Among them, the node attributes include geometric descriptors such as object size and posture parameters, and the edge attributes encode spatial topological relationships. Specifically, a symbolic logic-based physical constraint module is embedded in the decoding layer to define a wall penetration cost function and a furniture mobility matrix, ensuring that the output conforms to real-world physical laws. Finally, a relational graph reasoning network is used to generate a structured semantic description. The output conforms to the formal language specification and consists of <subject, predicate, object> triples and constraints, such as ("sofa-01", "located at", "TV wall - front", constraint: spacing = 1.2m ± 0.1m). This representation system supports the knowledge reasoning requirements of downstream tasks.

[0077] S209: Based on the global spatial object semantic description, the target and direction embedding vector and the multimodal fusion feature, a reinforcement learning algorithm is used to perform hierarchical navigation decision to obtain a target decision, wherein the target and direction embedding vector is a target semantic label after preprocessing.

[0078] In the embodiment of this application, Figure 3 The figure shows the robot navigation process. After map construction and multimodal data collection and processing in the early stage, the robot locks in long-term goals and builds a lightweight and robust strategy network. Through global semantic planning and local obstacle avoidance decisions, it combines reinforcement learning and semantic reasoning to make hierarchical navigation decisions, achieve optimal decisions in dynamic environments, and convert multimodal perception information into executable navigation action sequences.

[0079] In one embodiment of the present application, the target decision is obtained by using a reinforcement learning algorithm to perform hierarchical navigation decision based on the semantic description of the global space object, the target and direction embedding vector, and the multimodal fusion feature. S701: Perform action space modeling based on the global space object semantic description and target embedding vector.

[0080] S703: Determine local real-time obstacle avoidance constraints based on the multimodal fusion features and the direction embedding vector.

[0081] S705: Performing hierarchical navigation decision in the action space in combination with the real-time obstacle avoidance constraint.

[0082] In one embodiment of the present application, action space modeling is first performed. The role of the action space is to generate coarse-grained navigation instructions based on semantic features, guiding the robot to a macroscopic path towards the target area. The semantic description of the global spatial object is concatenated with the target embedding vector and mapped into a discrete action space probability distribution through a fully connected layer. Subsequently, local real-time obstacle avoidance constraints are determined based on the multimodal fusion features and the directional embedding vector, and corresponding fine-grained motion control instructions are generated, enabling the robot to avoid obstacles in real time while executing the global planned path.

[0083] Reinforcement learning algorithms specifically include: Discrete action space implementation:

[0084] in, is the action space dimension, is the action space matrix, is the transpose operator, is the discrete action selected, is the hidden state vector, is the state at time t.

[0085] Continuous Action Space Implementation:

[0086]

[0087]

[0088] in, 、 is the weight matrix, is the hidden state vector, is the action mean, is the action variance.

[0089] State value estimation:

[0090] in, 、 is the weight matrix, Indicates belonging to the value network, Represents the first layer of the value network, is the activation function, is the hidden state vector.

[0091] In one embodiment of the present application, performing hierarchical navigation decision in the action space in combination with the real-time obstacle avoidance constraint includes: S801: Determine the total loss function based on the PPO policy gradient loss, value function loss and entropy regularization term.

[0092] S803: Perform end-to-end optimization of the policy network based on the total loss function.

[0093] S805: Using the path length weighted success rate indicator to evaluate the efficiency and success rate of the hierarchical navigation decision.

[0094] In one embodiment of the present application, a total loss function is determined based on the PPO policy gradient loss, the value function loss, and the entropy regularization term, and the end-to-end optimization of the policy network is performed based on the total loss function. The total loss function expression is as follows:

[0095]

[0096]

[0097]

[0098] in, is the PPO policy gradient loss, is the loss of value function, is the entropy regularization term, 、 and is a hyperparameter, is the importance sampling ratio, is the advantage function, clip(·) is the clipping function, is the clipping threshold, Target value, is the current value network output, H(·) is the entropy of the strategy, The action probability output by the current policy network parameters.

[0099] The optimization process is based on the gradient update of the total loss function, which is expressed as:

[0100] in, is the adaptive learning rate.

[0101] In addition, the path length weighted success rate indicator is used to evaluate the efficiency and success rate of hierarchical navigation decisions. The calculation formula is:

[0102] in, Is the success of the i-th task (1 for success, 0 for failure), is the optimal path length of task i, is the actual path length of the agent in task i.

[0103] The general success rate only counts whether the target is reached, and cannot distinguish between "efficient arrival" and "detour arrival". The path length weighted success rate can make up for the limitation of the simple success rate. In specific applications, the Gibson test set scenario is used to test the robot navigation, such as Figure 4 As shown, this is a schematic diagram comparing the success rates of different target objects. Figure 5 Schematic diagram of the comparison of the weighted success rate (SPL) of different target object path lengths. Figure 6 This is a diagram showing the change in the weighted success rate (SPL) of the object path length for different verification rounds. Figure 7 Schematic diagram of the experimental trajectory comparison results.

[0104] In the aforementioned robot perception and decision-making method based on a lightweight multimodal large model, a semantic voxel map is first constructed. Multimodal data, including visual data, point cloud data, and target semantic labels, is then collected and preprocessed based on the semantic voxel map. Feature extraction is then performed on the preprocessed multimodal data, followed by dynamic cross-modal attention fusion to obtain multimodal fusion features. The multimodal data is then simultaneously input into the lightweight multimodal large model for semantic parsing, resulting in a global spatial object semantic description. Finally, a reinforcement learning algorithm is used to perform hierarchical navigation decisions based on the global spatial object semantic description, target and direction embedding vectors, and multimodal fusion features, resulting in a target decision. The target and direction embedding vectors represent the preprocessed target semantic labels. In other words, the dynamic cross-modal attention mechanism adaptively fuses point cloud, visual, and semantic features, eliminating resolution and semantic-level differences in multimodal data and significantly improving real-time decision-making efficiency. Through a lightweight, multimodal large model, semantic parsing of visual and point cloud data is achieved, generating structured descriptions containing object attributes and physical constraints. While deepening the depth and accuracy of scene understanding, the system complexity is reduced by optimizing computational logic, achieving efficient computing power allocation. Furthermore, the physical constraint encoding and object attributes output by the large model are used as inputs for action decisions, guiding the strategy to generate paths that conform to real-world rules. This can accurately avoid invalid collisions and significantly enhance the physical rationality of the decision-making process. This improves the accuracy, timeliness, and adaptability of the robot's perception and navigation decisions in complex scenarios.

[0105] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0106] Based on the same inventive concept, the embodiments of the present application also provide a robot perception and decision-making device based on a lightweight multimodal large model for implementing the aforementioned robot perception and decision-making method based on a lightweight multimodal large model. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of a robot perception and decision-making device based on a lightweight multimodal large model provided below can be found in the above-mentioned limitations of a robot perception and decision-making method based on a lightweight multimodal large model, and will not be repeated here.

[0107] In one embodiment, Figure 8 As shown, a robot perception and decision-making device 800 based on a lightweight multimodal large model is provided, comprising: a semantic voxel map construction module 801, a multimodal data acquisition and preprocessing module 803, a multimodal feature extraction and fusion module 805, a structured spatial semantic modeling module 807, and a hierarchical navigation decision module 809, wherein: A semantic voxel map construction module 801 is used to construct a semantic voxel map; A multimodal data acquisition and preprocessing module 803 is used to acquire and preprocess multimodal data based on the semantic voxel map, wherein the multimodal data includes visual data, point cloud data, and target semantic labels; The multimodal feature extraction and fusion module 805 is used to extract features from the pre-processed multimodal data and perform dynamic cross-modal attention fusion to obtain multimodal fusion features; The structured spatial semantic modeling module 807 is used to simultaneously input the multimodal data into a lightweight multimodal large model, perform semantic analysis, and obtain a semantic description of global spatial objects; The layered navigation decision module 809 is used to make layered navigation decisions based on the global spatial object semantic description, the target and direction embedding vector and the multimodal fusion feature using a reinforcement learning algorithm to obtain a target decision, wherein the target and direction embedding vector is a target semantic label after preprocessing.

[0108] In one embodiment of the present application, the semantic voxel map construction module is further configured to: A lightweight real-time semantic segmentation network is used to perform pixel-by-pixel semantic analysis and map the semantically labeled depth point cloud to a global coordinate system to construct an initial 3D semantic representation. The initial 3D semantic representation is decomposed into three subspaces using a multi-resolution voxel grid partitioning strategy, and temporal pose estimation, 3D non-rigid registration, and spatiotemporal feature fusion are performed. The 3D voxel features are compressed into a 2D bird's-eye view representation and an online map updating mechanism is integrated to obtain a semantic voxel map.

[0109] In one embodiment of the present application, the multimodal feature extraction and fusion module is further used to: The query-key attention mechanism, channel separation and normalization are used on the visual feature map to generate multi-layer semantic features of the image; The point cloud feature map is voxelized or projected, and dimension adjustment and convolution feature extraction are performed to generate high-dimensional point cloud features.

[0110] In one embodiment of the present application, the multimodal feature extraction and fusion module is further used to: Initial fusion of the extracted image multi-layer semantic features and point cloud features; Bilinear interpolation is used for cross-modal feature alignment; Channel attention is used to learn inter-channel dependencies, and spatial attention is used to focus on key areas; The attention mechanism is used to determine the modal interaction weights, and spatial attention and channel attention are combined for weighted fusion.

[0111] In one embodiment of the present application, the structured spatial semantic modeling module is further used to: Preprocessing the multimodal data; A differentiable bijective projection layer is designed to perform cross-modal feature fusion on the pre-processed multimodal data to obtain fused multimodal features. Constructing a hierarchical knowledge distillation framework to determine the structure of a three-dimensional scene graph based on the fused multimodal features; A global spatial object semantic description is generated based on the three-dimensional scene graph structure through a relational graph reasoning network.

[0112] In one embodiment of the present application, the layered navigation decision module is further configured to: Performing action space modeling based on the global space object semantic description and target embedding vector; Determining local real-time obstacle avoidance constraints based on the multimodal fusion features and the direction embedding vector; A hierarchical navigation decision is made in the action space in combination with the real-time obstacle avoidance constraint.

[0113] In one embodiment of the present application, the layered navigation decision module is further configured to: Determine the total loss function based on PPO policy gradient loss, value function loss and entropy regularization term; Performing end-to-end optimization of the policy network based on the total loss function; The path length weighted success rate indicator is used to evaluate the efficiency and success rate of hierarchical navigation decisions.

[0114] Each module in the aforementioned lightweight multimodal large model-based robot perception and decision-making device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0115] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown. The computer device includes a processor, memory, a communication interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal via wired or wireless communication. The wireless communication can be achieved via Wi-Fi, a mobile cellular network, NFC (near-field communication), or other technologies. When executed by the processor, the computer program implements a robot perception and decision-making method based on a lightweight multimodal large model. The display screen of the computer device can be a liquid crystal display or an electronic ink display. The input device of the computer device can be a touch layer covering the display screen, keys, a trackball, or a touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse.

[0116] Those skilled in the art will understand that Figure 9The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0117] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0118] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0119] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0120] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0121] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0122] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A robot perception and decision-making method based on a lightweight multimodal large model, characterized by: The method comprises: Construct semantic voxel maps; Collecting and preprocessing multimodal data based on the semantic voxel map, the multimodal data including visual data, point cloud data and target semantic labels; Perform feature extraction on the preprocessed multimodal data and perform dynamic cross-modal attention fusion to obtain multimodal fusion features; The multimodal data are simultaneously input into a lightweight multimodal large model for semantic analysis to obtain a semantic description of global spatial objects; Based on the global spatial object semantic description, target and direction embedding vectors and multimodal fusion features, a reinforcement learning algorithm is used to perform hierarchical navigation decision-making to obtain a target decision, wherein the target and direction embedding vectors are target semantic labels after preprocessing.

2. The robot perception and decision-making method based on a lightweight multimodal large model according to claim 1 is characterized in that: The constructing of the semantic voxel map comprises: A lightweight real-time semantic segmentation network is used to perform pixel-by-pixel semantic analysis and map the semantically labeled depth point cloud to a global coordinate system to construct an initial 3D semantic representation. The initial 3D semantic representation is decomposed into three subspaces using a multi-resolution voxel grid partitioning strategy, and temporal pose estimation, 3D non-rigid registration, and spatiotemporal feature fusion are performed. The 3D voxel features are compressed into a 2D bird's-eye view representation and an online map updating mechanism is integrated to obtain a semantic voxel map.

3. The robot perception and decision-making method based on a lightweight multimodal large model according to claim 1 is characterized in that: The feature extraction of the pre-processed multimodal data includes: The query-key attention mechanism, channel separation and normalization are used on the visual feature map to generate multi-layer semantic features of the image; The point cloud feature map is voxelized or projected, and dimension adjustment and convolution feature extraction are performed to generate high-dimensional point cloud features.

4. The robot perception and decision-making method based on a lightweight multimodal large model according to claim 1 is characterized in that: The dynamic cross-modal attention fusion includes: Initial fusion of the extracted image multi-layer semantic features and point cloud features; Bilinear interpolation is used for cross-modal feature alignment; Channel attention is used to learn inter-channel dependencies, and spatial attention is used to focus on key areas; The attention mechanism is used to determine the modal interaction weights, and spatial attention and channel attention are combined for weighted fusion.

5. The robot perception and decision-making method based on a lightweight multimodal large model according to claim 1 is characterized in that: The step of simultaneously inputting the multimodal data into a lightweight multimodal large model and performing semantic parsing to obtain a semantic description of a global spatial object includes: Preprocessing the multimodal data; A differentiable bijective projection layer is designed to perform cross-modal feature fusion on the pre-processed multimodal data to obtain fused multimodal features. Constructing a hierarchical knowledge distillation framework to determine the structure of a three-dimensional scene graph based on the fused multimodal features; A global spatial object semantic description is generated based on the three-dimensional scene graph structure through a relational graph reasoning network.

6. The robot perception and decision-making method based on a lightweight multimodal large model according to claim 1 is characterized in that: The target decision is obtained by using a reinforcement learning algorithm to perform hierarchical navigation decision based on the semantic description of the global space object, the target and direction embedding vector and the multimodal fusion feature: Performing action space modeling based on the global space object semantic description and target embedding vector; Determining local real-time obstacle avoidance constraints based on the multimodal fusion features and the direction embedding vector; A hierarchical navigation decision is made in the action space in combination with the real-time obstacle avoidance constraint.

7. The robot perception and decision-making method based on a lightweight multimodal large model according to claim 6 is characterized in that: The performing of hierarchical navigation decision in the action space in combination with the real-time obstacle avoidance constraint includes: Determine the total loss function based on PPO policy gradient loss, value function loss and entropy regularization term; Performing end-to-end optimization of the policy network based on the total loss function; The path length weighted success rate indicator is used to evaluate the efficiency and success rate of hierarchical navigation decisions.

8. A robot perception and decision-making device based on a lightweight multimodal large model, characterized in that: The device comprises: Semantic voxel map construction module, used to construct semantic voxel map; A multimodal data acquisition and preprocessing module, configured to acquire and preprocess multimodal data based on the semantic voxel map, wherein the multimodal data includes visual data, point cloud data, and target semantic labels; The multimodal feature extraction and fusion module is used to extract features from the preprocessed multimodal data and perform dynamic cross-modal attention fusion to obtain multimodal fusion features; A structured spatial semantic modeling module is used to simultaneously input the multimodal data into a lightweight multimodal large model, perform semantic analysis, and obtain a semantic description of global spatial objects; The hierarchical navigation decision module is used to make hierarchical navigation decisions based on the semantic description of the global spatial object, the target and direction embedding vectors and the multimodal fusion features, using a reinforcement learning algorithm to obtain a target decision, wherein the target and direction embedding vectors are the target semantic labels after preprocessing.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images