Robot navigation method, device and equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a robot navigation method and device, equipment and a storage medium. The method comprises the following steps: determining a visual feature vector according to a first video frame at the current moment; according to the first video frame, navigation task information for the robot at the current moment is determined and coded, and a prompt vector is determined; inputting the visual feature vector and the prompt vector into a multi-modal model to carry out robot track and action prediction processing, and determining track information and action information corresponding to the robot at the current moment; the multi-modal model comprises a visual language action model; middle action information corresponding to the robot at the current moment is determined according to the track information; according to the intermediate action information and the action information, target motion information corresponding to the robot at the current moment is determined, and a navigation instruction executed by the robot at the current moment is determined according to the target motion information. The robot navigation method provided by the invention can ensure the accuracy of the navigation trajectory of the robot executing the action.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a robot navigation method, apparatus, device, and storage medium. Background Technology

[0002] Currently, in order for robots to replace humans in performing tedious and repetitive tasks, robots generally need to have autonomous capabilities such as perception, planning, navigation, and movement. That is, the robot needs to plan and navigate its own path based on the information it perceives, and control the robot to move according to the navigation.

[0003] In related technologies, the input data is generally processed by a multimodal pre-trained model to output the robot's action sequence or action instructions, and the robot is controlled to move according to the action sequence or action instructions to achieve robot navigation.

[0004] However, the aforementioned techniques are prone to causing the accumulation of errors in the action sequence, which ultimately leads to deviations in the navigation trajectory of the robot's actions. Summary of the Invention

[0005] This application provides a robot navigation method, apparatus, device, and storage medium to solve the technical problem in the prior art that errors in the action sequence are easily accumulated, ultimately causing deviations in the navigation trajectory of the robot's actions.

[0006] In a first aspect, embodiments of this application provide a robot navigation method, including: Obtain the first video frame at the current moment, and determine the visual feature vector corresponding to the first video frame based on the first video frame; The navigation task information for the robot at the current moment is determined based on the first video frame, and the navigation task information is encoded to determine the prompt vector corresponding to the navigation task information. The visual feature vector and cue vector are input into the multimodal model for robot trajectory and motion prediction to determine the first trajectory information and first motion information of the robot at the current moment; the multimodal model mentioned above includes a visual-language-action model. Determine the intermediate action information of the robot at the current moment based on the first trajectory information; Based on the intermediate motion information and the first motion information, the target motion information corresponding to the robot at the current moment is determined, and the navigation instructions to be executed by the robot at the current moment are determined based on the target motion information.

[0007] In one embodiment, the multimodal model further includes a trajectory decoder and an action decoder. The process of inputting visual feature vectors and cue vectors into the multimodal model for robot trajectory and action prediction, and determining the first trajectory information and first action information corresponding to the robot at the current moment, includes: The visual feature vector and cue vector are input into the visual language action model to perform robot trajectory and action prediction processing to determine the prediction vector; the prediction vector includes trajectory information and action information predicted for the robot. The predicted vector is input into the trajectory decoder for decoding to determine the first trajectory information of the robot at the current moment. The predicted vector is input into the action decoder for decoding to determine the robot's first action information at the current moment.

[0008] In one embodiment, the multimodal model further includes a memory module, which comprises a memory bank and a memory cross-attention module. The process of inputting the prediction vector into a trajectory decoder for decoding to determine the first trajectory information corresponding to the robot at the current moment includes: Retrieve the historical trajectory vector corresponding to the robot's historical trajectory information from the memory bank; The robot's historical trajectory vector and predicted vector are input into the memory cross-attention module for cross-attention processing to determine the trajectory assistance information output by the memory cross-attention module; The trajectory auxiliary information and prediction vector are both input into the trajectory decoder for decoding processing to determine the first trajectory information.

[0009] In one embodiment, the memory module further includes a memory encoding module, and the method further includes: The first trajectory information is input into the memory encoding module for encoding processing to determine the first trajectory vector corresponding to the first trajectory information; Update the first trajectory vector to the memory.

[0010] In one embodiment, the training method for the above-mentioned multimodal model includes: The system acquires teaching action information and predicted action information of the robot at various moments during navigation, as well as the robot's actual trajectory information and predicted trajectory information in the world coordinate system during navigation. The teaching action information is obtained by manually operating the robot to navigate and recording the robot's actions during navigation. The predicted action information and the predicted trajectory information are obtained by using sample video frames collected by the robot during navigation, and by determining the sample visual feature vector and sample cue vector through the sample video frames, which are then input into the initial multimodal model to predict the robot's trajectory and actions. Based on the taught action information and predicted action information, the action decoder in the initial multimodal model is trained using imitation learning. Based on the real trajectory information and predicted trajectory information, the trajectory decoder and memory in the initial multimodal model are trained using supervised learning to obtain the multimodal model.

[0011] In one embodiment, the above-mentioned training of the action decoder in the initial multimodal model using imitation learning based on taught action information and predicted action information, and the training of the trajectory decoder and memory in the initial multimodal model using supervised learning based on real trajectory information and predicted trajectory information, to obtain a multimodal model, includes: Obtain the predicted action information and predicted trajectory information of a single sample video frame, freeze the model parameters of the memory module in the initial multimodal model, and train the trajectory decoder and action decoder in the initial multimodal model based on the taught action information, the predicted action information of a single sample video frame, the real trajectory information, and the predicted trajectory information of a single sample video frame to obtain the initially trained trajectory decoder and the initially trained action decoder. Obtain the predicted action information and predicted trajectory information of sample video frames in consecutive frames, unfreeze the model parameters of the memory module in the initial multimodal model, and train the trajectory decoder, action decoder and memory module in the initial multimodal model based on the taught action information, the predicted action information of sample video frames in consecutive frames, the real trajectory information and the predicted trajectory information of sample video frames in consecutive frames, to obtain the multimodal model.

[0012] In one embodiment, the training method for the above-mentioned multimodal model includes: Obtain a data sample sequence of a preset sequence length; the data sample sequence includes multiple data samples, each of which includes a sample video frame and sample navigation task information at a given moment. Randomly sample the i-th data sample from the data sample sequence, and divide the data sample sequence into a first data sample sequence and a second data sample sequence at the i-th data sample position; the first sequence length of the first data sample sequence and the second sequence length of the second data sample sequence may be the same or different; Based on the first and second data sample sequences, the visual language action model, trajectory decoder, action decoder, and memory module in the initial multimodal model are jointly trained to obtain the multimodal model.

[0013] In one embodiment, the first video frame includes a first depth map and a first RGB map at the current moment, and determining the visual feature vector corresponding to the first video frame based on the first video frame includes: Obtain the preliminary calibration parameters corresponding to the image acquisition device; the aforementioned image acquisition device is a device for acquiring the first depth map and the first RGB image. The projection matrix of the depth map onto the RGB image is determined based on the preliminary calibration parameters, and the projection matrix is ​​iteratively calculated to determine the visual fusion parameters between the first depth map and the first RGB image. The first depth map and the first RGB map are fused according to the visual fusion parameters to determine the visual feature map corresponding to the first video frame; The visual feature map is encoded to determine the visual feature vector.

[0014] In one embodiment, the first video frame includes a first depth map and a first RGB map at the current moment, and determining the visual feature vector corresponding to the first video frame based on the first video frame includes: The first depth map and the first RGB map are input into a convolutional neural network for fusion processing to determine the visual feature map corresponding to the first video frame; The visual feature map is encoded to determine the visual feature vector.

[0015] In one embodiment, the first video frame includes a first RGB image at the current moment, and determining the navigation task information for the robot at the current moment based on the first video frame includes: The scene description information in the first RGB image is extracted and processed to determine the scene description information at the current moment; Based on the robot's current pose and scene description information, determine the initial navigation task information; Based on the scene description information and the initial navigation task information, determine the navigation task information for the robot at the current moment.

[0016] Secondly, embodiments of this application provide a robot navigation device, comprising: The visual vector acquisition module is used to acquire the first video frame at the current moment and determine the visual feature vector corresponding to the first video frame based on the first video frame. The cue vector determination module is used to determine the navigation task information for the robot at the current moment based on the first video frame, encode the navigation task information, and determine the cue vector corresponding to the navigation task information. The prediction module is used to input visual feature vectors and cue vectors into the multimodal model for robot trajectory and motion prediction processing, and to determine the first trajectory information and first motion information of the robot at the current moment; the multimodal model mentioned above includes a visual language action model; The intermediate processing module is used to determine the intermediate action information of the robot at the current moment based on the first trajectory information; The motion output module is used to determine the target motion information of the robot at the current moment based on the intermediate motion information and the first motion information, and to determine the navigation instructions to be executed by the robot at the current moment based on the target motion information.

[0017] Thirdly, embodiments of this application provide an electronic device, including a memory, a transceiver, and a processor; A memory for storing computer programs; a transceiver for sending and receiving data under the control of the processor; and a processor for reading the computer programs from the memory and performing the following operations: Obtain the first video frame at the current moment, and determine the visual feature vector corresponding to the first video frame based on the first video frame; The navigation task information for the robot at the current moment is determined based on the first video frame, and the navigation task information is encoded to determine the prompt vector corresponding to the navigation task information. The visual feature vector and cue vector are input into the multimodal model for robot trajectory and motion prediction to determine the first trajectory information and first motion information of the robot at the current moment; the multimodal model mentioned above includes a visual-language-action model. Determine the intermediate action information of the robot at the current moment based on the first trajectory information; Based on the intermediate motion information and the first motion information, the target motion information corresponding to the robot at the current moment is determined, and the navigation instructions to be executed by the robot at the current moment are determined based on the target motion information.

[0018] Fourthly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the steps of the robot navigation method described in the first aspect.

[0019] Fifthly, embodiments of this application provide a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the robot navigation method described in the first aspect.

[0020] In a sixth aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the robot navigation method described in the first aspect.

[0021] The robot navigation method, apparatus, device, and storage medium provided in this application acquire a first video frame at the current moment, determine its corresponding visual feature vector based on the first video frame, determine the robot's navigation task information at the current moment based on the first video frame, encode the navigation task information to determine the corresponding prompt vector, input the visual feature vector and prompt vector into a multimodal model including a visual language action model for robot trajectory and action prediction and analysis, determine the first trajectory information and first action information corresponding to the robot at the current moment, determine the intermediate action information corresponding to the robot at the current moment based on the first trajectory information, determine the target action information corresponding to the robot at the current moment based on the intermediate action information and the first action information, and determine the navigation command executed by the robot at the current moment based on the target action information. In this method, since the robot navigation trajectory and action commands can be directly generated based on the visual language action model and multimodal input, end-to-end robot navigation function can be realized; at the same time, the corresponding action information is determined by the generated trajectory information, and combined with the action information output by the multimodal model at the same moment to determine the robot's final action information to execute the navigation command. In this way, the trajectory information can provide directional guidance for the robot's actions, avoid the robot from deviating, and improve the accuracy of the robot navigation trajectory. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is one of the flowcharts illustrating the robot navigation method provided in the embodiments of this application; Figure 2 This is a second schematic flowchart of the robot navigation method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the architecture of the multimodal model provided in the embodiments of this application; Figure 4 This is a schematic diagram of the data sample sequence provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the robot navigation device provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] With the rapid development of artificial intelligence, multimodal large-scale models, and robotics, robots need autonomous capabilities such as perception, planning, navigation, and movement to replace humans in repetitive and tedious tasks in scenarios such as home services, entertainment, and community governance. Existing technologies mainly employ the following solutions: 1. Traditional Path Planning Schemes: Based on supervised learning (SL) or reinforcement learning (RL) methods, these schemes achieve environmental perception and robot navigation through cascaded and separated modules such as perception, path planning, navigation, and motion control. However, while this technique performs well in specific environments (such as simple or known terrain), its generalization ability is weak, making it difficult to adapt to complex new environments. Supervised learning methods are particularly limited because they rely on large amounts of labeled data, and the models have poor adaptability to new environments. Although reinforcement learning methods have some generalization ability, they are prone to overfitting of perception components (such as limited field of view due to fixed-angle cameras), which affects multi-environment adaptability.

[0026] 2. Hierarchical Task Planning Approach: This approach consists of a high-level task planner (responsible for decomposing tasks into sub-goals) and a low-level control strategy (executing specific actions). It supports long-term logical planning, and its modular design improves scalability. However, this technique is typically expensive to train, especially when dealing with large models with many parameters. Furthermore, language-based approaches require additional alignment processes to adapt to the low-level control strategy, while code-based approaches are limited by the programming capabilities of existing models. For example, some hierarchical planning frameworks proposed in other techniques require additional modules to align the language with the underlying actions, increasing system complexity.

[0027] 3. End-to-end Vision-Language-Action (VLA) Model Solution: This solution combines multimodal pre-trained models (vision, language, and action sequences) to achieve end-to-end robot control. However, currently common end-to-end VLA model solutions do not perform well for tasks in specific scenarios. Some technologies only output action commands or action sequences, which can easily lead to the accumulation of errors in the action sequences, ultimately resulting in trajectory deviations during action execution.

[0028] Based on this, embodiments of this application provide a robot navigation method, apparatus, device, and storage medium, which can solve the above-mentioned technical problems, improve the accuracy of robot recognition of unstructured area accessibility and enhance its generalization and adaptability to different scenarios, thereby improving the robot's environmental adaptability and task execution capabilities.

[0029] It should be noted that the execution subject in the embodiments of this application can be a robot navigation device, an electronic device including a robot navigation device, or other devices, equipment, or systems. The following embodiments will use an electronic device as the execution subject as an example for illustration. The electronic device can be a terminal or a server. The terminal can be, for example, a robot, an electronic device in a robot, or a terminal connected to the robot (such as a mobile phone, personal computer, wearable device, etc.).

[0030] Figure 1 This is one of the flowcharts illustrating the robot navigation method provided in the embodiments of this application, referred to... Figure 1 As shown, the method may include the following steps: Step 102: Obtain the first video frame at the current moment, and determine the visual feature vector corresponding to the first video frame based on the first video frame.

[0031] The first video frame at the current moment includes a depth image (denoted as the first depth map) and an RGB image (denoted as the first RGB image) at the current moment. The first depth map includes the depth information of each point in the current environment, which can reflect the spatial position information of the object. The first RGB image is a two-dimensional perspective view from the robot's perspective, where each pixel represents the RGB three primary colors of the object, which can reflect the semantic information of the current environment.

[0032] Specifically, an image acquisition device can be used to acquire depth maps and RGB images of the current environment, obtaining the first depth map and the first RGB image at the current moment. These two images are then combined to form the first video frame at the current moment. The image acquisition device can be a camera or video camera mounted on a robot, or it can be a camera or video camera installed in the current environment. This image acquisition device can include a depth map acquisition camera and an optical image acquisition camera, or it can include both depth map and optical image acquisition cameras. Figure 1 Cameras for volume capture, etc. When capturing video frames in a scene using image acquisition devices, information such as the capture timestamp and camera pose of each video frame can be recorded simultaneously.

[0033] After obtaining the first video frame at the current moment, the first depth map and the first RGB image in the first video frame can be aligned, fused, and transformed to obtain the visual feature vector corresponding to the first video frame. It can be understood that this visual feature vector incorporates both the depth information and semantic information of the scene, thus possessing visual information. Subsequent predictions of robot trajectory and actions based on this feature vector will be more consistent with reality and more accurate.

[0034] Step 104: Determine the navigation task information for the robot at the current moment based on the first video frame, encode the navigation task information, and determine the prompt vector corresponding to the navigation task information.

[0035] After obtaining the first video frame, relevant descriptive information about the scene at the current moment can be determined from the first video frame. This allows for the construction of the robot's navigation task information at the current moment based on the scene's descriptive information. Alternatively, the navigation task information can be comprehensively determined by combining the relevant descriptive information with user-input navigation prompts. This navigation task information at the current moment includes the robot's specific navigation direction, navigation distance, navigation destination, and gait during navigation.

[0036] After obtaining the navigation task information at the current moment, which is typically in text format, this information can be encoded into a vector to facilitate subsequent processing by a multimodal model. This vector generates a corresponding cue vector. The cue vector clearly defines the navigation task the robot needs to perform in the current scene, allowing the multimodal model to generate accurate navigation trajectories and actions, enabling the robot to execute the navigation task precisely.

[0037] Step 106: Input the visual feature vector and cue vector into the multimodal model for robot trajectory and motion prediction processing to determine the first trajectory information and first motion information of the robot at the current moment; the multimodal model includes a visual language motion model.

[0038] The multimodal model can include a Visual Language Action (VLA) model, and of course, other models, such as the encoding model that converts video frames into visual feature vectors, the encoding model that converts navigation task information into cue vectors, the decoder that decodes trajectory or action information, and attention mechanism modules. Alternatively, the VLA model can include a visual language action encoder, a decoder, or an attention mechanism module. The specific type of VLA model is not specifically limited.

[0039] Specifically, after obtaining the visual feature vector and cue vector at the current moment, this information constitutes multimodal information. This multimodal information can be input into a multimodal model. Within the multimodal model, based on the cue vector, feature extraction, convolution, pooling, and other processing are performed on the visual feature vector, as well as fusion processing of the visual feature vector and cue vector. This enables the prediction of the robot's navigation trajectory and actions at the current moment, ultimately obtaining the first predicted trajectory and first action information for the robot at the current moment. Alternatively, other sensors can continuously collect the robot's motion data during navigation, including its pose, linear velocity, angular velocity, and executed actions. This motion data can also be converted into vectors and, together with the current visual feature vector and cue vector, used as multimodal information, input into the multimodal model for trajectory and action prediction, improving the accuracy of the final trajectory and action prediction.

[0040] The first trajectory information may include multiple discrete trajectory points and the gait of each trajectory point. Each trajectory point has corresponding position information, and the gait may include, for example, climbing or walking. The first action information may be an action sequence, which may include multiple actions. Each action may include the linear velocity, angular velocity, etc., that the robot needs to perform at the current moment. The linear velocity may include, for example, forward, backward, or stopping, and the angular velocity may include, for example, execution, left or right turning, and the specific turning angle.

[0041] Alternatively, the above can also acquire LiDAR radar point cloud data, extract point cloud features, and add a three-dimensional voxel fusion network (VoxelFusion-Net) to process point cloud adjustments to enhance spatial representation.

[0042] Furthermore, for the aforementioned multimodal model to predict trajectories and actions, the prediction can be joint or the actions and trajectories can be separated into independent subsystems for separate prediction.

[0043] Step 108: Determine the intermediate action information of the robot at the current moment based on the first trajectory information.

[0044] In this step, after predicting the robot's first trajectory information at the current moment through a multimodal model, the first trajectory information may include multiple discrete trajectory points and the gait of each trajectory point. Each trajectory point has corresponding position information. By performing interpolation and differentiation on each discrete trajectory point, a continuous sequence of actions can be generated as the robot's intermediate action information at the current moment.

[0045] Step 110: Based on the intermediate motion information and the first motion information, determine the target motion information corresponding to the robot at the current moment, and determine the navigation instructions to be executed by the robot at the current moment based on the target motion information.

[0046] In this step, after determining the corresponding intermediate action information using the first trajectory information at the current moment, the intermediate action information and the first action information predicted by the multimodal model can be weighted and averaged to obtain the robot's final action information at the current moment, which is denoted as the target action information.

[0047] After obtaining the robot's target action information at the current moment, the target action information can be converted into action instructions or navigation instructions and sent to the robot's control module for execution to realize the robot's navigation task.

[0048] In this embodiment, the first video frame at the current moment is acquired, and its corresponding visual feature vector is determined based on the first video frame. The robot's navigation task information at the current moment is then determined based on the first video frame, and the navigation task information is encoded to determine the corresponding cue vector. The visual feature vector and cue vector are input into a multimodal model including a visual language-action model for robot trajectory and action prediction and analysis. This determines the robot's first trajectory information and first action information at the current moment. Based on the first trajectory information, intermediate action information is determined for the robot at the current moment. Based on the intermediate action information and the first action information, the target action information for the robot at the current moment is determined. Finally, based on the target action information, the navigation command executed by the robot at the current moment is determined. In this method, because the robot's navigation trajectory and action commands can be directly generated based on the visual language-action model using multimodal input, end-to-end robot navigation functionality can be achieved. Simultaneously, the generated trajectory information determines the corresponding action information, which is then combined with the action information output by the multimodal model at the same moment to determine the robot's final action information for executing navigation commands. Thus, the trajectory information provides directional guidance for the robot's actions, preventing the robot from deviating from its course and improving the accuracy of the robot's navigation trajectory.

[0049] The above embodiments mention that visual feature vectors can be determined by the first video frame. Regarding the specific method of determining the visual feature vector corresponding to the first video frame based on the first video frame, two possible implementation methods are given below.

[0050] Method 1: The first video frame mentioned above includes the first depth map and the first RGB map at the current moment. Step 102, which determines the visual feature vector corresponding to the first video frame, may include: Obtain the preliminary calibration parameters corresponding to the image acquisition device; the aforementioned image acquisition device is a device for acquiring the first depth map and the first RGB image. The projection matrix of the depth map onto the RGB image is determined based on the preliminary calibration parameters, and the projection matrix is ​​iteratively calculated to determine the visual fusion parameters between the first depth map and the first RGB image. The first depth map and the first RGB map are fused according to the visual fusion parameters to determine the visual feature map corresponding to the first video frame; The visual feature map is encoded to determine the visual feature vector.

[0051] When image acquisition devices that collect depth maps and RGB images are installed (e.g., on a robot), their initial calibration parameters can be determined based on the installation setup. For example, if the image acquisition device includes a depth camera and a regular camera, the initial calibration parameters can include information such as the intrinsic and extrinsic parameter matrices of the depth camera and the regular camera.

[0052] After initial calibration of the image acquisition device and obtaining initial calibration parameters, the projection matrix from the depth map to the RGB image can be determined using the intrinsic and extrinsic parameter matrices of the two cameras from the initial calibration parameters. An optimization algorithm iteratively calculates this projection matrix to determine the optimal projection matrix, i.e., the optimal visual fusion parameters. Then, information from the first depth map is fused into the first RGB image using these visual fusion parameters, achieving visual feature fusion (Vision Fusion). This yields the visual feature map corresponding to the first video frame, which is then encoded using a visual encoder (Vision Encoder) to determine the visual feature vector (Vision Embed) corresponding to the first video frame.

[0053] The aforementioned visual feature map is a visual feature map with depth information, which integrates the object's position information and the semantic information of the scene. Therefore, it has richer information and more accurate prediction results when predicting navigation trajectories and actions in the future.

[0054] Method 2: The first video frame mentioned above includes the first depth map and the first RGB map at the current moment. Step 102 above, which determines the visual feature vector corresponding to the first video frame, may include: The first depth map and the first RGB map are input into a convolutional neural network for fusion processing to determine the visual feature map corresponding to the first video frame; The visual feature map is encoded to determine the visual feature vector.

[0055] Specifically, a convolutional neural network (CNN) can be pre-trained to align and fuse features from both the depth map and the RGB image. This CNN can be trained using multiple sample video frames and the corresponding visual feature vectors for each frame. Each sample video frame can include both a depth map and an RGB image from the same time point. The visual feature vectors for each frame can be obtained manually through feature alignment and fusion. Furthermore, the specific type or architecture of the CNN is not limited here.

[0056] Specifically, after obtaining the first depth map and the first RGB map at the current moment, the first depth map and the first RGB map can be input into the trained CNN for adjustment, alignment and fusion processing to determine the visual feature map corresponding to the first video frame, and the visual encoder is used to encode the visual feature map to determine the visual feature vector corresponding to the first video frame.

[0057] In this embodiment, the projection matrix from the depth map to the RGB image is determined by the initial calibration parameters of the image acquisition device, and visual fusion parameters are iteratively derived. Then, feature fusion is performed on the depth map and the RGB image using the visual fusion parameters, which improves the accuracy of visual feature fusion. Furthermore, feature fusion is performed on the depth map and the RGB image using a pre-trained convolutional neural network, which improves the efficiency of visual feature fusion.

[0058] The above embodiments mentioned that the robot's navigation task information at the current moment can be determined based on the first video frame. The following embodiments provide a possible implementation method.

[0059] In one embodiment, the first video frame includes a first RGB image at the current moment, and step 104, which determines the navigation task information for the robot at the current moment based on the first video frame, may include: The scene description information in the first RGB image is extracted and processed to determine the scene description information at the current moment; Based on the robot's current pose and scene description information, determine the initial navigation task information; Based on the scene description information and the initial navigation task information, determine the navigation task information for the robot at the current moment.

[0060] After obtaining the first video frame at the current moment, the first RGB image within it can be obtained. This first RGB image can then be input into an extraction model, which extracts scene description information from the first RGB image to obtain the scene description information corresponding to the current moment. This extraction model can be, for example, a text feature extraction model, such as BERT and its variants.

[0061] Simultaneously, the robot's current pose can be acquired and output to the user via sensors on the robot (including positioning sensors, attitude sensors, velocity / angular velocity sensors, etc.) or pose measurement devices in the scene. The scene description information corresponding to the first RGB image can also be output to the user, allowing them to publish navigation task information based on the robot's current pose and the scene description information. Specifically, the user can manually sample and select suitable scene description information, combine it with the robot's current pose to determine and publish the navigation task information. After the user publishes the task, the electronic device will be aware of it, and this is recorded as the initial navigation task information. The user can publish this initial navigation task information by inputting it into the electronic device.

[0062] After obtaining the current scene description information and the initial navigation task, the two can be combined to generate the current final navigation task information or prompt information. The navigation task information combines the scene description information and the navigation task information published by the user. It can be given a fixed task format and goal, while loading the scene description information. For example, it can be represented as: {"task": "After turning left 10 meters ahead, switch the walking action to the stair climbing gait"}.

[0063] After obtaining the navigation task information at the current moment, the navigation task information can be used as a language prompt and input into the prompt encoder for encoding to obtain the prompt vector corresponding to the navigation task information.

[0064] In this embodiment, scene description information is extracted from the RGB image, and the initial navigation task information issued by the user is determined by combining the robot's current pose with the current scene description information. Finally, the final navigation task information is determined based on the current scene description information and the initial navigation task information. In this way, the final navigation task information is determined by determining the scene description through the RGB image and combining it with the robot's pose. The determined navigation task information has a higher degree of fit with the actual scene, so the subsequent predicted navigation trajectory is more accurate.

[0065] The above embodiments mentioned that the multimodal model can simultaneously predict the robot's navigation trajectory and actions. The following embodiments will illustrate an implementation method for predicting the robot's navigation trajectory and actions using the multimodal model, which also includes a trajectory decoder and an action decoder.

[0066] Figure 2 This is a second schematic flowchart of the robot navigation method provided in the embodiments of this application. See also... Figure 2 As shown, step 106 above, which inputs the visual feature vector and cue vector into a multimodal model for robot trajectory and motion prediction, to determine the robot's first trajectory information and first motion information at the current moment, may include: Step 202: Input the visual feature vector and the cue vector into the visual language action model to perform robot trajectory and action prediction processing, and determine the prediction vector; the prediction vector includes trajectory information and action information predicted for the robot.

[0067] Step 204: Input the predicted vector into the trajectory decoder for decoding processing to determine the first trajectory information of the robot at the current moment.

[0068] Step 206: Input the predicted vector into the action decoder for decoding processing to determine the first action information of the robot at the current moment.

[0069] Among them, see Figure 3 The schematic diagram of the multimodal model architecture shows that after obtaining the first video frame (including the DepthFrame and RGB Frame, the first depth map and the first RGB image), the first depth map and the first RGB image can be fused using Vision Fusion to obtain a visual feature map, which is then input into the Vision Encoder for encoding processing to determine the visual feature vector (Vision Embd) corresponding to the first video frame. Simultaneously, after obtaining navigation task information or language prompts, these can be input into the Prompt Encoder for encoding processing to obtain the prompt vector (Prompt Embd) corresponding to the navigation task information.

[0070] The multimodal model can include a Visual Language Action Model (VLA) encoder, a trajectory decoder, a trajectory output module, an action decoder, and an action output module. The VLA encoder, based on the Llama2 architecture, receives the current visual feature vector and cue vector through early fusion. After multiple layers of visual and linguistic cross-attention calculations, it outputs a predicted vector that fuses the robot's current trajectory and action information. This predicted vector is then input to the action decoder for action decoding, yielding the first predicted action information or action command for the robot at the current moment. Alternatively, the predicted vector can be directly input to the trajectory decoder for trajectory decoding, obtaining the first predicted trajectory information for the robot at the current moment.

[0071] Alternatively, the multimodal model may further include a memory module, which includes a memory bank and a memory attention module. In step 204, the predicted vector is input to the trajectory decoder for decoding to determine the first trajectory information of the robot at the current moment, including: Retrieve the historical trajectory vector corresponding to the robot's historical trajectory information from the memory bank; The robot's historical trajectory vector and predicted vector are input into the memory cross-attention module for cross-attention processing to determine the trajectory assistance information output by the memory cross-attention module; The trajectory auxiliary information and prediction vector are both input into the trajectory decoder for decoding processing to determine the first trajectory information.

[0072] The memory bank primarily stores the robot's trajectory information at historical times. This trajectory information can be historical trajectories or historical trajectory vectors, etc., and can be continuously collected during the robot's navigation process.

[0073] When predicting the robot's trajectory at the current moment, the historical trajectory information or vectors of the robot at a set historical time can be obtained from the memory bank. The length of the set historical time can be set according to the actual situation, such as 3 to 5 moments before the current moment. After obtaining the historical trajectory vector, it can be input together with the prediction vector output by the encoder of the aforementioned visual language action model into the memory cross-attention module. In the memory cross-attention module, cross-attention calculation is performed on these two parts of information to obtain prior information, which is recorded as trajectory auxiliary information. Then, the trajectory auxiliary information and the prediction vector output by the encoder of the aforementioned visual language action model are input together into the trajectory decoder. In the trajectory decoder, the prediction vector is decoded based on the trajectory auxiliary information to predict the robot's first trajectory information at the current moment.

[0074] By combining the robot's historical trajectory information with current trajectory prediction, the multimodal model can utilize historical trajectory information when processing subsequent frames, thereby improving the accuracy and consistency of trajectory prediction.

[0075] Furthermore, optionally, the memory module further includes a memory encoder, and the method further includes: The first trajectory information is input into the memory encoding module for encoding processing to determine the first trajectory vector corresponding to the first trajectory information; the first trajectory vector is then updated in the memory bank.

[0076] In other words, after the trajectory decoder outputs the predicted first trajectory information, it can be input to the trajectory output module. The trajectory output module then inputs this first trajectory information into the memory encoding module, encoding it into a memory vector, denoted as the first trajectory vector. This first trajectory vector is then stored in the memory bank as historical trajectory information for subsequent trajectory predictions. This continuously updates and enriches the historical trajectory information in the memory bank, aiding in accurate trajectory prediction processing. Simultaneously, the first trajectory information output by the trajectory output module and the first action information output by the action decoder can be input together into the action output module. There, intermediate action information is first calculated based on the first trajectory information, and then combined with the first action information to determine the final target action information.

[0077] In this embodiment, visual feature vectors and cue vectors are jointly input into a visual-language action encoder for trajectory and action prediction processing to obtain predicted vectors. These predicted vectors are then input into an action decoder and a trajectory decoder respectively to obtain the corresponding action and trajectory information. This hierarchical network architecture allows for fast and accurate prediction of the predicted trajectory and action. Furthermore, combining the robot's historical trajectory information to assist in current trajectory prediction helps the multimodal model utilize historical trajectory information when processing subsequent frames, improving the accuracy and consistency of trajectory prediction. Moreover, by continuously updating the predicted trajectory information at the current moment to a memory bank for storage as historical trajectory information for subsequent trajectory predictions, the historical trajectory information in the memory bank is continuously updated and enriched, contributing to accurate trajectory prediction processing in the future.

[0078] The aforementioned multimodal model may include an encoder for a visual-language-action model, which can be pre-trained. The following examples illustrate the training method of the encoder for the visual-language-action model in a multimodal model.

[0079] In one embodiment, the training process of the encoder of the visual language action model includes: Obtain sample visual feature vectors and sample cue vectors of robots in different application scenarios; Based on the pre-trained ResNet-50 and Transformer architecture as the visual and language encoders, the above-mentioned sample visual feature vectors and sample cue vectors are used to fine-tune the encoder of the visual language action model through contrastive learning, so that the visual feature vectors and cue vectors input into the VLA encoder are semantically aligned.

[0080] In this embodiment, the encoder of the visual language action model in the multimodal model is pre-trained, which facilitates the rapid training of the entire multimodal model in the future.

[0081] The aforementioned multimodal model may include an encoder for a visual language action model, as well as a trajectory decoder, an action decoder, and a memory module. After the encoder for the visual language action model is pre-trained, the model parameters of the encoder can be frozen first, and the trajectory decoder, action decoder, and memory module can be trained. The following embodiments illustrate the training methods for the trajectory decoder, action decoder, and memory module in the multimodal model.

[0082] In one embodiment, the training method for the above-mentioned multimodal model includes: Step A1: Acquire the robot's taught action information and predicted action information at each moment during navigation, as well as the robot's actual trajectory information and predicted trajectory information in the world coordinate system during navigation. The taught action information is obtained by manually operating the robot to navigate and recording the robot's actions during navigation. The predicted action information and predicted trajectory information are obtained by using sample video frames collected by the robot during navigation, and by determining the sample visual feature vector and sample cue vector through the sample video frames, and then inputting them into the initial multimodal model to predict the robot's trajectory and actions.

[0083] Step A2: Based on the taught action information and predicted action information, the action decoder in the initial multimodal model is trained using imitation learning. Based on the real trajectory information and predicted trajectory information, the trajectory decoder and memory in the initial multimodal model are trained using supervised learning to obtain the multimodal model.

[0084] In this embodiment, only the trajectory decoder, action decoder, and memory module components of the multimodal model are trained; the weight parameters of other model components remain frozen. The training process employs imitation learning to optimize the action decoder, while supervised learning is used to optimize related modules such as trajectory prediction and the memory module.

[0085] Specifically, we can first acquire the robot's taught action information at various moments during navigation. This taught action information is expert demonstration data, originating from the data acquisition phase. The robot is manually controlled to move from the starting point to the target point along a pre-defined path. The robot's motion data is collected and labeled as motion commands and trajectories, thus obtaining the taught action information and the actual trajectory information. The taught action information includes the linear velocity (forward, stop, backward) and angular velocity (straight ahead, left / right turn θ degrees) controlling the robot's movement. The actual trajectory information is a sequence transformed to the robot's coordinate system relative to its current position. Each element includes not only position coordinates but also corresponding posture and gait, representing the robot's pose and gait at the specified coordinate position.

[0086] Meanwhile, during robot navigation, sample video frames can be continuously acquired to determine sample visual feature vectors and sample cue vectors, which are then input into the VLA encoder for encoding to obtain sample prediction vectors. These vectors are then input into the initial trajectory decoder and initial motion decoder for decoding to obtain predicted trajectory information and predicted motion information.

[0087] Then, imitation learning can be used to train the action decoder. Specifically, this can involve first calculating the first loss between the angular velocity and linear velocity in the action information predicted by the action decoder and the expert demonstration data. The calculation method for this first loss is as follows: ; Where T is the sequence length corresponding to the action information, and D is the action dimension, which is 2 here, representing both linear velocity and angular velocity. It is the d-th dimension action predicted at step t by the multimodal model. It is information about the teaching action.

[0088] Furthermore, the motion information predicted by the motion decoder can also include the predicted robot gait, and the taught motion information also includes the robot gait. A second loss can then be calculated between the robot gait predicted by the motion decoder and the robot gait in the taught motion information. The second loss is calculated as follows: ; Where K represents the number of action categories, such as climbing a hill or walking, which can be preset. It is a one-hot encoding of the teaching action information. It is the robot's gait predicted by the motion decoder.

[0089] After calculating the first loss and the second loss between the action information predicted by the action decoder and the taught action information, the first loss and the second loss can be weighted and summed to obtain the total loss. The action decoder is then trained using the total loss. The calculation formula is as follows: ; Where λ is the weight hyperparameter.

[0090] Of course, the above can also be trained using the PPO (Proximal Policy Optimization) strategy to train the action decoder.

[0091] Alternatively, the robot's actual trajectory information in the world coordinate system during navigation (such as the sequence of actual position coordinates) can be obtained. To ensure the accuracy of the loss function calculation, this actual trajectory information in the world coordinate system needs to be transformed to the robot's coordinate system, so that it is within the same coordinate framework as the robot's predicted trajectory information. Then, supervised learning can be used to train the trajectory decoder and memory module by combining the actual trajectory information and the trajectory information predicted by the trajectory decoder. Specifically, this can involve calculating the third loss between the trajectory information predicted by the trajectory decoder and the actual trajectory information. The calculation method for this third loss is as follows: ; Where M is the trajectory length and D is the coordinate dimension, which is 3 here, i.e., three-dimensional spatial coordinates (x, y, z). These are the predicted coordinates in the trajectory information predicted by the trajectory decoder. These are the actual coordinates in the real trajectory information. The smoothL1 is calculated using the following formula: ; in, It is an adjustable threshold hyperparameter (usually set to 1.0). x For variables.

[0092] After calculating the third loss between the trajectory information predicted by the trajectory decoder and the actual trajectory information, the trajectory decoder and memory module can be trained using the third loss.

[0093] Alternatively, the total loss of the action decoder and the third loss of the trajectory decoder can be summed, and the action decoder, trajectory decoder, and memory module can be further optimized and trained using the summed loss. The method for summing the losses is as follows: ; in, To the loss after summing, For weight hyperparameters.

[0094] Furthermore, to ensure training stability, a phased training strategy can be adopted to train the action decoder, trajectory decoder, and memory module. Optionally, in step A2 above, the action decoder in the initial multimodal model is trained using imitation learning based on the taught action information and predicted action information, and the trajectory decoder and memory in the initial multimodal model are trained using supervised learning based on the real trajectory information and predicted trajectory information, to obtain the multimodal model, including: Obtain the predicted action information and predicted trajectory information of a single sample video frame, freeze the model parameters of the memory module in the initial multimodal model, and train the trajectory decoder and action decoder in the initial multimodal model based on the taught action information, the predicted action information of a single sample video frame, the real trajectory information, and the predicted trajectory information of a single sample video frame to obtain the initially trained trajectory decoder and the initially trained action decoder. Obtain the predicted action information and predicted trajectory information of sample video frames in consecutive frames, unfreeze the model parameters of the memory module in the initial multimodal model, and train the trajectory decoder, action decoder and memory module in the initial multimodal model based on the taught action information, the predicted action information of sample video frames in consecutive frames, the real trajectory information and the predicted trajectory information of sample video frames in consecutive frames, to obtain the multimodal model.

[0095] The phased training strategy includes a two-stage training strategy. The first stage involves collecting multiple single-frame video frames of the robot during navigation. Since the memory module has not yet generated historical trajectory information at the beginning, the model parameters of the memory module are frozen. The trajectory decoder and action decoder are trained in the first stage by combining the predicted trajectory information and predicted action information output by the trajectory decoder and action decoder through each single-frame video frame.

[0096] After the first stage of training is completed, the second stage of model training is carried out. Specifically, multiple sets of continuous frames are collected during the robot's navigation process. Each set of continuous frames includes multiple consecutive video frames. The memory module can save historical trajectory information, unfreeze the model parameters of the memory module, and combine the predicted trajectory information and predicted action information output by the trajectory decoder and action decoder in each set of continuous frames to carry out the second stage of training for the trajectory decoder, memory module, and action decoder.

[0097] Of course, the training processes in steps A1 and A2 mentioned above are included in both the first and second stages of training. The exception is that in the first stage, only the action decoder and trajectory decoder are trained, and the memory module is not trained.

[0098] In this embodiment, the action decoder is trained using imitation learning based on expert demonstration action data and action information predicted by the action decoder. Conversely, supervised learning is used to train the trajectory decoder and memory module based on real trajectory information and trajectory information predicted by the trajectory decoder. By training the action decoder, trajectory decoder, and memory module using two different training methods, the accuracy of the multimodal model training can be improved. Furthermore, by first training the action decoder and trajectory decoder using single-frame images, and then training them using consecutive frames, this staged training approach enhances the stability of the multimodal model training, resulting in a more robust final multimodal model.

[0099] Furthermore, as mentioned above, the multimodal model may include an encoder for the visual language action model, as well as a trajectory decoder, an action decoder, and a memory module. After the encoder, trajectory decoder, action decoder, and memory module of the visual language action model are pre-trained separately, these modules can be jointly trained to achieve end-to-end model training. The following embodiments illustrate the end-to-end training process of the multimodal model.

[0100] In one embodiment, the training method for the above-mentioned multimodal model includes: Obtain a data sample sequence of a preset sequence length; the data sample sequence includes multiple data samples, each of which includes a sample video frame and sample navigation task information at a given moment. Randomly sample the i-th data sample from the data sample sequence, and divide the data sample sequence into a first data sample sequence and a second data sample sequence at the i-th data sample position; the first sequence length of the first data sample sequence and the second sequence length of the second data sample sequence may be the same or different; Based on the first and second data sample sequences, the visual language action model, trajectory decoder, action decoder, and memory module in the initial multimodal model are jointly trained to obtain the multimodal model.

[0101] First, the model parameters of the visual encoder and the cue word encoder can be frozen. Then, multiple consecutive video frames are acquired, along with sample action information and sample trajectory sequences corresponding to each video frame. These multiple consecutive video frames, the sample action information corresponding to each video frame, and the sample trajectory sequences constitute a sample data sequence, the length of which is a preset sequence length, the specific size of which can be set according to actual conditions. The sample data sequence includes multiple sample data. For example, see [link to example]. Figure 4 The diagram shows a sample data sequence. Each sample data includes a video frame, the corresponding sample action information, and a sample trajectory sequence. Each video frame includes a depth map and an RGB map. The video frames, sample action information, and sample trajectory sequences included in the sample data sequence can be represented as Frame_0, Frame_1, ..., Frame_i, ..., Frame_n, Action_0, Action_1, ..., Action_i, ..., Action_n, Traj_0[m0], Traj_1[m1], ..., Traj_i[mi], ..., Traj_n[mn], where n is the preset sequence length, which is greater than 1. The sample trajectory sequence in each sample data includes m (greater than 1) trajectory points.

[0102] Then, by using random sampling, for a data sample with a sequence length of n, the i-th frame is randomly sampled to divide the data sample sequence into two segments, obtaining the first data sample sequence and the second data sample sequence. The first data sample sequence includes sample data from frame 0 to frame i, and the second data sample sequence includes sample data from frame i to frame n. That is, the i-th frame data sample serves as the end frame of the data sample sequence from 0 to i and the initial frame of the data sample sequence from i to n.

[0103] Then, the weights of the VLA encoder and subsequent modules (action decoder, trajectory decoder and memory module) of the multimodal model can be fine-tuned using the first data sample sequence and the second data sample sequence. The training method is the same as that in steps A1 and A2 above, that is, the action prediction based on imitation learning and the trajectory prediction based on supervised learning are combined for training, and finally a trained multimodal model is obtained, realizing end-to-end multimodal model training.

[0104] In this embodiment, by randomly dividing continuous frame data samples into two data sample sequences for end-to-end multimodal model training, the utilization rate of data can be greatly increased, the dependence on expert demonstration data can be avoided, and the generalization ability of the model can be improved.

[0105] Furthermore, the robot navigation method described above in this application can be deployed in two ways based on actual needs: pure edge deployment and edge-cloud combined deployment. Pure edge deployment involves running all algorithm models and code involved in the technical solution of this application on one or more chips mounted on the robot body. Edge-cloud combined deployment involves deploying the VLA encoder module, which has high computing power requirements, on a cloud server, and the robot edge can call the capabilities of this module through an API interface.

[0106] Furthermore, the aforementioned navigation task information is described in natural language, indicating that the robot needs to reach the target point under the current environmental conditions using a specified method. This natural language can be input directly or as speech, which is then converted to text by a speech-to-text module. The navigation task information to be performed is input to the language prompt module (e.g., the module receiving LanguagePrompt), which generates a prompt vector through a prompt word encoder. Simultaneously, this is combined with the visual feature vector encoded from the robot's current visual information and input to the VLA encoder module for trajectory and motion prediction processing.

[0107] Furthermore, the motion output module combines the motion decoder and trajectory information to output motion information or motion commands. After the robot body receives the motion command, it decomposes the linear velocity (forward, stop, backward) and angular velocity (straight ahead, left / right turn θ degrees) in the command into the direction and speed of rotation required by each joint motor, thereby enabling the robot to move stably according to the specified target.

[0108] In summary, this embodiment of the application, based on the Visual Language Action (VLA) model, employs multimodal input (including depth maps with location information, RGB maps with semantic information, and language prompts with navigation task information) to directly generate robot navigation trajectories and action commands, achieving end-to-end autonomous robot planning and navigation functions. The generated navigation trajectory consists of discrete trajectory points and corresponding gaits that the robot needs to traverse. A continuous sequence of actions is generated through interpolation and differentiation. Then, the action commands in the action sequence that are weighted relative to the action decoder output at the same time are used as the final commands executed by the robot.

[0109] At the visual input end, the depth map and RGB image of the video frame are fused and aligned before being input into the encoder of the visual language action model. By combining the spatial location information of the depth map and the semantic information of the RGB image for trajectory and action prediction, the model's spatial perception can be enhanced. At the same time, the cue vector contains cue information about the task target and spatial location, which is conducive to improving the model's spatial understanding and the continuity of output, thereby improving the accuracy of robot navigation.

[0110] Meanwhile, by employing a memory bank to cache historical trajectory information and a cross-attention mechanism to fuse the output encoded with the historical trajectory information with the current output of the multimodal model, historical trajectory tracking can be achieved. The fusion operation of historical trajectory information and the current prediction vector is placed before the trajectory decoder, which increases the stability and continuity of the output trajectory.

[0111] Furthermore, in the imitation learning stage of multimodal models, randomly collecting trajectory points and corresponding action commands from the trajectory data as the start and end points of the data sequence can avoid dependence on demonstration data, improve the model's generalization ability, and significantly enhance the robot's generalization ability in complex new environments. At the same time, through imitation learning and end-to-end fine-tuning, the dependence on large-scale training data can be reduced, thereby lowering training costs.

[0112] The robot navigation device provided in the embodiments of this application is described below. The robot navigation device described below can be referred to in correspondence with the robot navigation method described above.

[0113] Figure 5 This is a schematic diagram of the robot navigation device provided in the embodiments of this application. See also: Figure 5 As shown, the device may include: The visual vector acquisition module 510 is used to acquire the first video frame at the current moment and determine the visual feature vector corresponding to the first video frame based on the first video frame. The cue vector determination module 520 is used to determine the navigation task information for the robot at the current moment based on the first video frame, encode the navigation task information, and determine the cue vector corresponding to the navigation task information. Prediction module 530 is used to input visual feature vectors and cue vectors into a multimodal model for robot trajectory and motion prediction processing, and to determine the first trajectory information and first motion information of the robot at the current moment; the multimodal model includes a visual language action model. Intermediate processing module 540 is used to determine the intermediate action information of the robot at the current moment based on the first trajectory information; The motion output module 550 is used to determine the target motion information of the robot at the current moment based on the intermediate motion information and the first motion information, and to determine the navigation instructions to be executed by the robot at the current moment based on the target motion information.

[0114] The electronic devices involved in the embodiments of this application can be terminals, specifically devices that provide voice and / or data connectivity to users, handheld devices with wireless connectivity, or other processing devices connected to a wireless modem. The names of terminal devices may differ in different systems; for example, in a 5G system, a terminal device can be called User Equipment (UE).

[0115] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application, with reference to... Figure 6 This application also provides an electronic device, which may include: a memory 610, a transceiver 620, a processor 630, and a user interface 640; The memory 610 is used to store computer programs; the transceiver 620 is used to send and receive data under the control of the processor 630; the processor 630 is used to read the computer program in the memory 610 and perform the following operations: Obtain the first video frame at the current moment, and determine the visual feature vector corresponding to the first video frame based on the first video frame; The navigation task information for the robot at the current moment is determined based on the first video frame, and the navigation task information is encoded to determine the prompt vector corresponding to the navigation task information. The visual feature vector and cue vector are input into the multimodal model for robot trajectory and motion prediction to determine the first trajectory information and first motion information of the robot at the current moment; the multimodal model mentioned above includes a visual-language-action model. Determine the intermediate action information of the robot at the current moment based on the first trajectory information; Based on the intermediate motion information and the first motion information, the target motion information corresponding to the robot at the current moment is determined, and the navigation instructions to be executed by the robot at the current moment are determined based on the target motion information.

[0116] Among them, Figure 6In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 630) and memory (memory 610). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 620 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. For different user equipment, the user interface 640 can also be an interface capable of connecting external or internal devices as needed.

[0117] The processor 630 is responsible for managing the bus architecture and general processing, while the memory 610 can store the data used by the processor 630 when performing operations.

[0118] The processor 630 executes any of the methods described in the embodiments of this application according to the obtained executable instructions by calling a computer program stored in the memory 610. The processor and the memory may also be physically separated.

[0119] Optionally, the above multimodal model also includes a trajectory decoder and an action decoder, and the processor 630 is also used to perform the following operations: The visual feature vector and cue vector are input into the visual language action model to perform robot trajectory and action prediction processing to determine the prediction vector; the prediction vector includes trajectory information and action information predicted for the robot. The predicted vector is input into the trajectory decoder for decoding to determine the first trajectory information of the robot at the current moment. The predicted vector is input into the action decoder for decoding to determine the robot's first action information at the current moment.

[0120] Optionally, the multimodal model further includes a memory module, which includes a memory bank and a memory cross-attention module. The processor 630 is also used to perform the following operations: Retrieve the historical trajectory vector corresponding to the robot's historical trajectory information from the memory bank; The robot's historical trajectory vector and predicted vector are input into the memory cross-attention module for cross-attention processing to determine the trajectory assistance information output by the memory cross-attention module; The trajectory auxiliary information and prediction vector are both input into the trajectory decoder for decoding processing to determine the first trajectory information.

[0121] Optionally, the memory module described above also includes a memory encoding module, and the processor 630 is further configured to perform the following operations: The first trajectory information is input into the memory encoding module for encoding processing to determine the first trajectory vector corresponding to the first trajectory information; Update the first trajectory vector to the memory.

[0122] Optionally, the processor 630 is also used to perform the following operations: The system acquires teaching action information and predicted action information of the robot at various moments during navigation, as well as the robot's actual trajectory information and predicted trajectory information in the world coordinate system during navigation. The teaching action information is obtained by manually operating the robot to navigate and recording the robot's actions during navigation. The predicted action information and the predicted trajectory information are obtained by using sample video frames collected by the robot during navigation, and by determining the sample visual feature vector and sample cue vector through the sample video frames, which are then input into the initial multimodal model to predict the robot's trajectory and actions. Based on the taught action information and predicted action information, the action decoder in the initial multimodal model is trained using imitation learning. Based on the real trajectory information and predicted trajectory information, the trajectory decoder and memory in the initial multimodal model are trained using supervised learning to obtain the multimodal model.

[0123] Optionally, the processor 630 is also used to perform the following operations: Obtain the predicted action information and predicted trajectory information of a single sample video frame, freeze the model parameters of the memory module in the initial multimodal model, and train the trajectory decoder and action decoder in the initial multimodal model based on the taught action information, the predicted action information of a single sample video frame, the real trajectory information, and the predicted trajectory information of a single sample video frame to obtain the initially trained trajectory decoder and the initially trained action decoder. Obtain the predicted action information and predicted trajectory information of sample video frames in consecutive frames, unfreeze the model parameters of the memory module in the initial multimodal model, and train the trajectory decoder, action decoder and memory module in the initial multimodal model based on the taught action information, the predicted action information of sample video frames in consecutive frames, the real trajectory information and the predicted trajectory information of sample video frames in consecutive frames, to obtain the multimodal model.

[0124] Optionally, the processor 630 is also used to perform the following operations: Obtain a data sample sequence of a preset sequence length; the data sample sequence includes multiple data samples, each of which includes a sample video frame and sample navigation task information at a given moment. Randomly sample the i-th data sample from the data sample sequence, and divide the data sample sequence into a first data sample sequence and a second data sample sequence at the i-th data sample position; the first sequence length of the first data sample sequence and the second sequence length of the second data sample sequence may be the same or different; Based on the first and second data sample sequences, the visual language action model, trajectory decoder, action decoder, and memory module in the initial multimodal model are jointly trained to obtain the multimodal model.

[0125] Optionally, the aforementioned first video frame includes a first depth map and a first RGB map at the current moment, and the processor 630 is further configured to perform the following operations: Obtain the preliminary calibration parameters corresponding to the image acquisition device; the aforementioned image acquisition device is a device for acquiring the first depth map and the first RGB image. The projection matrix of the depth map onto the RGB image is determined based on the preliminary calibration parameters, and the projection matrix is ​​iteratively calculated to determine the visual fusion parameters between the first depth map and the first RGB image. The first depth map and the first RGB map are fused according to the visual fusion parameters to determine the visual feature map corresponding to the first video frame; The visual feature map is encoded to determine the visual feature vector.

[0126] Optionally, the aforementioned first video frame includes a first depth map and a first RGB map at the current moment, and the processor 630 is further configured to perform the following operations: The first depth map and the first RGB map are input into a convolutional neural network for fusion processing to determine the visual feature map corresponding to the first video frame; The visual feature map is encoded to determine the visual feature vector.

[0127] Optionally, the aforementioned first video frame includes the first RGB image at the current moment, and the processor 630 is further configured to perform the following operations: The scene description information in the first RGB image is extracted and processed to determine the scene description information at the current moment; Based on the robot's current pose and scene description information, determine the initial navigation task information; Based on the scene description information and the initial navigation task information, determine the navigation task information for the robot at the current moment.

[0128] It should be noted that the electronic device provided in this application embodiment can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0129] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call a computer program stored in the memory 730 to execute the steps of a robot navigation method, such as: Obtain the first video frame at the current moment, and determine the visual feature vector corresponding to the first video frame based on the first video frame; The navigation task information for the robot at the current moment is determined based on the first video frame, and the navigation task information is encoded to determine the prompt vector corresponding to the navigation task information. The visual feature vector and cue vector are input into the multimodal model for robot trajectory and motion prediction to determine the first trajectory information and first motion information of the robot at the current moment; the multimodal model mentioned above includes a visual-language-action model. Determine the intermediate action information of the robot at the current moment based on the first trajectory information; Based on the intermediate motion information and the first motion information, the target motion information corresponding to the robot at the current moment is determined, and the navigation instructions to be executed by the robot at the current moment are determined based on the target motion information.

[0130] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0131] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the robot navigation method provided in the above embodiments, such as: Obtain the first video frame at the current moment, and determine the visual feature vector corresponding to the first video frame based on the first video frame; The navigation task information for the robot at the current moment is determined based on the first video frame, and the navigation task information is encoded to determine the prompt vector corresponding to the navigation task information. The visual feature vector and cue vector are input into the multimodal model for robot trajectory and motion prediction to determine the first trajectory information and first motion information of the robot at the current moment; the multimodal model mentioned above includes a visual-language-action model. Determine the intermediate action information of the robot at the current moment based on the first trajectory information; Based on the intermediate motion information and the first motion information, the target motion information corresponding to the robot at the current moment is determined, and the navigation instructions to be executed by the robot at the current moment are determined based on the target motion information.

[0132] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing a processor to perform the steps of the methods provided in the above embodiments, such as including: Obtain the first video frame at the current moment, and determine the visual feature vector corresponding to the first video frame based on the first video frame; The navigation task information for the robot at the current moment is determined based on the first video frame, and the navigation task information is encoded to determine the prompt vector corresponding to the navigation task information. The visual feature vector and cue vector are input into the multimodal model for robot trajectory and motion prediction to determine the first trajectory information and first motion information of the robot at the current moment; the multimodal model mentioned above includes a visual-language-action model. Determine the intermediate action information of the robot at the current moment based on the first trajectory information; Based on the intermediate motion information and the first motion information, the target motion information corresponding to the robot at the current moment is determined, and the navigation instructions to be executed by the robot at the current moment are determined based on the target motion information.

[0133] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0134] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0135] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method of robot navigation, characterized by, include: Obtain the first video frame at the current moment, and determine the visual feature vector corresponding to the first video frame based on the first video frame; Based on the first video frame, determine the navigation task information for the robot at the current moment, encode the navigation task information, and determine the prompt vector corresponding to the navigation task information; The visual feature vector and the cue vector are input into a multimodal model for robot trajectory and motion prediction processing to determine the first trajectory information and first motion information of the robot at the current moment; the multimodal model includes a visual language motion model; Determine the intermediate action information of the robot at the current moment based on the first trajectory information; Based on the intermediate action information and the first action information, the target motion information corresponding to the robot at the current moment is determined, and the navigation command executed by the robot at the current moment is determined based on the target motion information.

2. The method of claim 1, wherein, The multimodal model further includes a trajectory decoder and an action decoder. The step of inputting the visual feature vector and the cue vector into the multimodal model for robot trajectory and action prediction processing, and determining the first trajectory information and first action information corresponding to the robot at the current moment, includes: The visual feature vector and the cue vector are input into the visual language action model to perform robot trajectory and action prediction processing to determine the prediction vector; the prediction vector includes trajectory information and action information predicted for the robot. The predicted vector is input into the trajectory decoder for decoding to determine the first trajectory information of the robot at the current moment. The predicted vector is input into the action decoder for decoding to determine the first action information of the robot at the current moment.

3. The method of robot navigation of claim 2, wherein, The multimodal model further includes a memory module, which comprises a memory bank and a memory cross-attention module. The step of inputting the predicted vector into the trajectory decoder for decoding processing to determine the first trajectory information corresponding to the robot at the current moment includes: Obtain the historical trajectory vector corresponding to the robot's historical trajectory information from the memory bank; The robot's historical trajectory vector and the predicted vector are input into the memory cross-attention module for cross-attention processing to determine the trajectory assistance information output by the memory cross-attention module; The trajectory assistance information and the prediction vector are both input into the trajectory decoder for decoding processing to determine the first trajectory information.

4. The robot navigation method according to claim 3, characterized in that, The memory module further includes a memory encoding module, and the method further includes: The first trajectory information is input into the memory encoding module for encoding processing to determine the first trajectory vector corresponding to the first trajectory information; Update the first trajectory vector in the memory.

5. The robot navigation method according to claim 3, characterized in that, The training methods for the multimodal model include: The system acquires teaching action information and predicted action information of the robot at various moments during navigation, as well as the robot's actual trajectory information and predicted trajectory information in the world coordinate system during navigation. The teaching action information is obtained by manually operating the robot to navigate and recording the robot's actions during navigation. The predicted action information and predicted trajectory information are obtained by using sample video frames collected by the robot during navigation, and by determining sample visual feature vectors and sample cue vectors through the sample video frames, which are then input into an initial multimodal model for robot trajectory and action prediction. Based on the taught action information and the predicted action information, the action decoder in the initial multimodal model is trained using imitation learning, and based on the real trajectory information and the predicted trajectory information, the trajectory decoder and memory in the initial multimodal model are trained using supervised learning to obtain the multimodal model.

6. The robot navigation method according to claim 5, characterized in that, The process of training the action decoder in the initial multimodal model using imitation learning based on the taught action information and the predicted action information, and training the trajectory decoder and memory in the initial multimodal model using supervised learning based on the real trajectory information and the predicted trajectory information, to obtain the multimodal model, includes: Obtain the predicted action information and predicted trajectory information of a single sample video frame, freeze the model parameters of the memory module in the initial multimodal model, and train the trajectory decoder and action decoder in the initial multimodal model based on the taught action information, the predicted action information of the single sample video frame, the real trajectory information, and the predicted trajectory information of the single sample video frame to obtain the pre-trained trajectory decoder and pre-trained action decoder. Obtain the predicted action information and predicted trajectory information of sample video frames in consecutive frames, unfreeze the model parameters of the memory module in the initial multimodal model, and train the trajectory decoder, the action decoder, and the memory module in the initial multimodal model based on the taught action information, the predicted action information of the sample video frames in consecutive frames, the real trajectory information, and the predicted trajectory information of the sample video frames in consecutive frames, to obtain the multimodal model.

7. The robot navigation method according to claim 3, characterized in that, The training methods for the multimodal model include: Obtain a data sample sequence of a preset sequence length; the data sample sequence includes multiple data samples, each of which includes a sample video frame and sample navigation task information at a given moment; The i-th data sample is randomly sampled from the data sample sequence, and the data sample sequence is divided into a first data sample sequence and a second data sample sequence at the i-th data sample. The first sequence length of the first data sample sequence and the second sequence length of the second data sample sequence are the same or different. Based on the first data sample sequence and the second data sample sequence, the visual language action model, trajectory decoder, action decoder and memory module in the initial multimodal model are jointly trained to obtain the multimodal model.

8. The robot navigation method according to claim 1, characterized in that, The first video frame includes a first depth map and a first RGB map at the current time. Determining the visual feature vector corresponding to the first video frame based on the first video frame includes: Obtain the preliminary calibration parameters corresponding to the image acquisition device; the image acquisition device is a device that acquires the first depth map and the first RGB image. The projection matrix of the depth map onto the RGB image is determined based on the preliminary calibration parameters, and the projection matrix is ​​iteratively calculated to determine the visual fusion parameters between the first depth map and the first RGB image. The first depth map and the first RGB map are fused according to the visual fusion parameters to determine the visual feature map corresponding to the first video frame. The visual feature map is encoded to determine the visual feature vector.

9. The robot navigation method according to claim 1, characterized in that, The first video frame includes a first depth map and a first RGB map at the current time. Determining the visual feature vector corresponding to the first video frame based on the first video frame includes: The first depth map and the first RGB map are input into a convolutional neural network for fusion processing to determine the visual feature map corresponding to the first video frame. The visual feature map is encoded to determine the visual feature vector.

10. The robot navigation method according to claim 1, characterized in that, The first video frame includes a first RGB image at the current moment. Determining the navigation task information for the robot at the current moment based on the first video frame includes: The scene description information in the first RGB image is extracted and processed to determine the scene description information at the current moment; Based on the robot's current pose and the scene description information, determine the initial navigation task information; Based on the scene description information and the initial navigation task information, determine the navigation task information for the robot at the current moment.

11. A robot navigation device, characterized in that, include: The visual vector acquisition module is used to acquire the first video frame at the current moment and determine the visual feature vector corresponding to the first video frame based on the first video frame. The cue vector determination module is used to determine the navigation task information for the robot at the current moment based on the first video frame, and to encode the navigation task information to determine the cue vector corresponding to the navigation task information. The prediction module is used to input the visual feature vector and the cue vector into a multimodal model for robot trajectory and motion prediction processing, and to determine the first trajectory information and first motion information of the robot at the current moment; the multimodal model includes a visual language action model; The intermediate processing module is used to determine the intermediate action information of the robot at the current moment based on the first trajectory information; The motion output module is used to determine the target motion information of the robot at the current moment based on the intermediate motion information and the first motion information, and to determine the navigation command executed by the robot at the current moment based on the target motion information.

12. An electronic device, characterized in that, Includes memory, transceiver, and processor; A memory for storing computer programs; a transceiver for sending and receiving data under the control of the processor; and a processor for reading the computer programs from the memory and performing the following operations: Obtain the first video frame at the current moment, and determine the visual feature vector corresponding to the first video frame based on the first video frame; Based on the first video frame, determine the navigation task information for the robot at the current moment, encode the navigation task information, and determine the prompt vector corresponding to the navigation task information; The visual feature vector and the cue vector are input into a multimodal model for robot trajectory and motion prediction processing to determine the first trajectory information and first motion information of the robot at the current moment; the multimodal model includes a visual language motion model; Determine the intermediate action information of the robot at the current moment based on the first trajectory information; Based on the intermediate action information and the first action information, the target motion information corresponding to the robot at the current moment is determined, and the navigation command executed by the robot at the current moment is determined based on the target motion information.

13. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the robot navigation method according to any one of claims 1 to 10.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the robot navigation method as described in any one of claims 1 to 10.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the robot navigation method according to any one of claims 1 to 10.

Images