Quadruped robot end-to-end navigation method, robot and storage medium

Abstract

This application discloses an end-to-end navigation method for a quadruped robot, a robot, and a storage medium. The method acquires multimodal input information from the quadruped robot during navigation tasks, inputs this information into an end-to-end navigation model, utilizes the model to obtain hidden states or multimodal fusion features, inputs these features into a planning adapter to generate planning latent variables, inputs these variables into an action prediction head, and outputs trajectory prediction results. This method achieves end-to-end fusion of multimodal information and low-latency trajectory prediction through a phased training strategy and simulation data construction. It effectively solves the problems of error accumulation and poor real-time performance of traditional cascaded architectures and autoregressive inference by utilizing information distillation in the planning adapter and a non-autoregressive output mechanism.

Description

Technical Field

[0001] This application relates to the field of robotics, and more specifically, to an end-to-end navigation method for a quadruped robot, the robot, and a storage medium. Background Technology

[0002] This section provides only background information relevant to this disclosure and is not necessarily prior art.

[0003] Quadruped robots, with their excellent terrain adaptability and mobility, have shown broad application prospects in fields such as inspection, search and rescue, and logistics delivery. Navigation capability is the core foundation for quadruped robots to perform autonomous tasks. Most existing quadruped robot navigation solutions adopt a cascaded architecture, namely a pipelined processing flow of "perception-recognition-planning-control". In this architecture, the perception module is responsible for environmental understanding, the recognition module is responsible for target detection and semantic extraction, the planning module generates a path based on the perception and recognition results, and the control module converts the path into motion commands. However, this cascaded architecture has significant drawbacks: the links between modules are long, and the output error of the previous module will be passed on and accumulated to subsequent modules, resulting in a decrease in overall navigation accuracy; at the same time, each module needs to be designed and debugged independently, resulting in high deployment costs, high system complexity, and difficulty in achieving end-to-end optimization.

[0004] In recent years, visual language models (VLMs) have made significant progress in multimodal understanding, providing a new technical path for robot navigation. However, directly applying general visual language models to robot navigation tasks faces multiple challenges. First, general visual language models lack task adaptability; a semantic gap exists between their pre-training objectives and the fine-grained motion outputs required for robot navigation, making it difficult for the models to directly generate executable motion plans from general visual-language features. Second, existing visual language models typically rely on autoregressive generation methods for motion output, i.e., generating action sequences label-by-label. This autoregressive mechanism leads to high inference latency, making it difficult to meet the stringent real-time requirements of quadruped robot navigation. Furthermore, although visual-language-action (VLA) models have achieved some success in simple tasks such as robotic arm manipulation, when extended to robot tasks requiring long-range planning and complex environmental understanding, such as navigation and object finding, they still suffer from insufficient generalization ability and output accuracy.

[0005] On the other hand, acquiring training data is also a pressing problem in navigation research for quadruped robots. Data collection for real quadruped robot platforms requires building physical experimental environments, deploying sensors, and manually driving robots. This process is costly, inefficient, and struggles to cover diverse scenarios and task requirements, severely restricting the training and performance improvement of navigation models. Summary of the Invention

[0006] The purpose of this invention is to provide an end-to-end navigation method for quadruped robots to solve the technical problems of error accumulation in existing cascaded navigation architectures, poor real-time performance of action output from general visual language models, and high data acquisition costs on real robot platforms.

[0007] To achieve the above objectives, the first aspect of this application proposes an end-to-end navigation method for a quadruped robot, comprising: The multimodal input information of the quadruped robot in the navigation task is acquired, wherein the multimodal input information includes at least two of the following: visual perception information, navigation target information, task instruction information, and motion history information; The multimodal input information is input into the end-to-end navigation model, and the general multimodal base model or multimodal feature extraction network in the end-to-end navigation model is used to extract fusion features from the multimodal input information to obtain hidden states or multimodal fusion features. The hidden state or multimodal fusion features are input into the planning adapter, which extracts planning features related to future path planning from the hidden state or multimodal fusion features and generates planning latent variables. The planning latent variables are input into the motion prediction head, and the motion prediction head outputs the trajectory prediction result. The trajectory prediction result includes at least one of the following: path points, trajectory points, velocity commands, and attitude commands in the quadruped robot's body coordinate system or local coordinate system.

[0008] According to the end-to-end navigation method for a quadruped robot provided in this application, the step of obtaining multimodal input information of the quadruped robot in the navigation task includes: The current frame image and historical frame image are acquired by the forward-facing camera configured on the quadruped robot, and the current frame image and historical frame image are organized into the visual perception information in chronological order. Extract the motion history information of the quadruped robot and convert the motion history information into a sequence of relative coordinate points; The navigation target information is generated by cropping the target region based on the position of the task object in the historical frame image; The task instruction information is received, and at least two of the visual perception information, the relative coordinate point sequence, the navigation target information, and the task instruction information are combined to form the multimodal input information.

[0009] According to the end-to-end navigation method for a quadruped robot provided in this application, the step of inputting the hidden state or multimodal fusion features into a planning adapter, and having the planning adapter extract planning features related to future path planning from the hidden state or multimodal fusion features and generate planning latent variables includes: Set up K learnable path query vectors, and use the hidden state or the multimodal fusion feature as the key and value of cross attention; Using the path query vector as the query and the hidden state or the multimodal fusion feature as the key and value, construct one or more cross-attention modules, and extract information related to the future path through the cross-attention modules; The multi-query features output by the cross-attention module are flattened, and the flattened features are mapped into a fixed-length vector through a multilayer perceptron projection network to obtain the planning latent variables.

[0010] According to the end-to-end navigation method for a quadruped robot provided in this application, the motion prediction head is a one-time non-autoregressive motion prediction head, which includes a path point prediction head and a two-dimensional relative trajectory prediction head. The planning latent variables are input into the path point prediction head, and a one-time non-autoregressive regression method is used to directly output the path point sequence containing location and orientation information as the main output. The latent variables of the planning are input into the two-dimensional relative trajectory prediction head, and multiple two-dimensional relative trajectory points accumulated and expanded with fixed anchor points as the starting point are output as auxiliary outputs.

[0011] According to the end-to-end navigation method for a quadruped robot provided in this application, the general multimodal base model, the planning adapter, and the one-time non-autoregressive action prediction head are obtained through the following staged training strategy: In the first stage, the visual coding parameters in the general multimodal base model are frozen, and the general multimodal base model is fine-tuned using a two-dimensional image trajectory point dataset to obtain a teacher model, which can output multiple two-dimensional relative trajectory points. In the second stage, the parameters of the general multimodal base model are frozen, the planning adapter and the two-dimensional relative trajectory prediction head are trained, and the two-dimensional image trajectory point dataset is used for supervision so that the planning latent variables learn to inherit the two-dimensional route intention. In the third stage, a warm start is performed using the training checkpoints from the second stage. The existing capabilities of the planning adapter and the two-dimensional relative trajectory prediction head are retained, the path point prediction head is connected, and the two-dimensional image trajectory point dataset and the path point training dataset are used together for training.

[0012] According to the end-to-end navigation method for a quadruped robot provided in this application, the training loss function in the third stage is: L=λrel×Lrel+λwp×Lwp+λori×Lori+λsmooth×Lsmooth Where Lrel is the L1 loss of the two-dimensional relative trajectory point, Lwp is the path point position loss, Lori is the path point orientation loss, Lsmooth is the path point smoothness loss, and λrel, λwp, λori, and λsmooth are the weight coefficients of the corresponding losses.

[0013] According to the end-to-end navigation method for a quadruped robot provided in this application, the pathpoint training dataset is constructed in the following manner: The target building is scanned using a 3D scanning tool to construct a 3D Gaussian sputtering model; Import the three-dimensional Gaussian sputtering model into the simulation platform to construct a simulation scene; Import the quadruped robot model into the simulation scene and complete the sensor configuration and communication environment configuration. In the simulation scenario, commands, current field-of-view image, coordinate points of two-dimensional image trajectory, and path point data in the three-dimensional simulation scenario are collected. The path point training dataset is formed by organizing and annotating the data. The coordinates of the path points are defined in the robot body coordinate system, with the x-axis for forward direction, the y-axis for left direction, and θ for the relative forward orientation angle. Each path point is represented as (x, y, sin(θ), cos(θ)).

[0014] According to the end-to-end navigation method for a quadruped robot provided in this application, the two-dimensional image trajectory point dataset is constructed in the following manner: Scene video data is generated using a video generation model, and image frames are obtained by sampling the scene video data. The sampled image frames are manually annotated to obtain target category, target location, description of the following object, and trajectory supervision information; Scene enhancement processing is performed on the labeled image frames. Preset object textures are added to the images to simulate occlusion scenes, and trajectory information is corrected simultaneously to form training samples suitable for navigation, pedestrian following and target search tasks. Organized according to a unified format, the two-dimensional image trajectory point dataset is formed. Each training sample includes the current frame image, historical frame images, template image, natural language instructions, trajectory markers from the previous time step, and multiple two-dimensional relative trajectory point labels for the current time step. The two-dimensional relative trajectory point labels are accumulated and expanded starting from a fixed anchor point.

[0015] A second aspect of this application proposes a robot comprising: A visual sensor, used to acquire current frame images and historical frame images; The memory stores an end-to-end navigation model, which includes a general multimodal base model or a multimodal feature extraction network, a planning adapter, and an action prediction head. The general multimodal base model or multimodal feature extraction network is used to extract features from multimodal input information and output hidden states or multimodal fusion features. The planning adapter is used to extract planning latent variables from the hidden states or multimodal fusion features through learnable path query vectors and cross-attention mechanisms. The action prediction head is used to map the planning latent variables to trajectory prediction results. A processor, wherein the processor is used to execute the end-to-end navigation method for a quadruped robot according to any of the above-described technical solutions.

[0016] The robot in this embodiment of the invention has the same advantages as the above-described end-to-end navigation method for quadruped robots, which will not be repeated here.

[0017] A third aspect of this application proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the end-to-end navigation method for a quadruped robot according to any of the above-described technical solutions.

[0018] The computer-readable storage medium of this invention has the same advantages as the above-described end-to-end navigation method for quadruped robots, and will not be repeated here.

[0019] Compared with the prior art, the present invention has the following beneficial effects: This invention achieves end-to-end navigation for quadruped robots by combining a general multimodal base model or a multimodal feature extraction network with a learnable planning adapter. Taking the general multimodal base model as an example, navigation capabilities can be adapted simply by freezing its parameters and training a lightweight adapter on top of it, without having to train from scratch, thus significantly reducing training costs and data requirements.

[0020] Based on this, the present invention uses a motion prediction head to output trajectory prediction results. The trajectory prediction results include at least one of path points, trajectory points, velocity commands, and attitude commands in the body coordinate system or local coordinate system. In some embodiments of the present invention, the motion prediction head is a one-time non-autoregressive motion prediction head that directly outputs a sequence of path points, avoiding the error accumulation and inference delay caused by point-by-point generation in the autoregressive method, and achieving efficient and accurate trajectory prediction.

[0021] Meanwhile, this invention unifies multimodal input information into a comprehensive input that includes at least two of the following: visual perception information, navigation target information, task instruction information, and motion history information. This enables the same model framework to respond to various task instructions such as navigation, following, and searching, thereby improving the versatility and flexibility of the navigation system.

[0022] Furthermore, this invention actively extracts information related to future path planning from hidden states or multimodal fusion features through learnable path query vectors and cross-attention mechanisms in the planning adapter, compressing high-dimensional multimodal features into compact planning latent variables, which significantly reduces the computational cost of subsequent action prediction while retaining key planning information.

[0023] In summary, this invention utilizes the perception and understanding capabilities of a general multimodal basic model or a multimodal feature extraction network, combined with a lightweight planning adapter and action prediction head, to achieve end-to-end autonomous navigation of a quadruped robot in complex scenarios with low training cost, balancing navigation accuracy, inference efficiency, and task generalization ability. Attached Figure Description

[0024] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application. In the drawings: Figure 1 This is a flowchart of the end-to-end navigation method for a quadruped robot provided in an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] In this application, the terms "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing this application and its embodiments, and are not intended to limit the indicated device, element, or component to having a specific orientation, or to be constructed and operated in a specific orientation.

[0028] Furthermore, in addition to indicating location or positional relationship, some of the aforementioned terms may also have other meanings. For example, the term "above" may also be used in some cases to indicate a certain dependency or connection relationship. Those skilled in the art can understand the specific meaning of these terms in this application based on the specific circumstances.

[0029] Furthermore, the terms "installation," "setup," "equipped with," "connection," "linked," and "socketing" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral structure; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, components, or parts. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0030] like Figure 1 As shown, this embodiment of the invention provides an end-to-end navigation method for a quadruped robot. The method is executed by the quadruped robot's processor and mainly includes processes such as multimodal input organization, feature extraction, planning latent variable generation, and trajectory prediction.

[0031] In the multimodal input organization phase, the quadruped robot first acquires and combines various perceptual information at the current moment. Specifically, the quadruped robot uses its configured forward-facing camera to acquire current frame images and historical frame images, and organizes these images in chronological order into visual perceptual information, such as a visual input sequence. The resolution and acquisition frequency of the forward-facing camera can be set according to actual needs; for example, the resolution can be set to 640×360 pixels and the acquisition frequency can be set to 10Hz. The reason for including historical frame images in the input sequence is that they contain visual observation information from the robot over a past period. This information helps the model understand the dynamic changes in the scene, such as the direction of movement of moving objects and the direction of passage extension, thus providing temporal context support for path planning.

[0032] The quadruped robot also extracts its own motion history information, converting it into a sequence of relative coordinate points. Representing historical trajectory information in the form of relative coordinates allows the model to perceive the robot's past motion trends and preferences. This prior motion information helps the model make reasonable decisions consistent with historical behavior when multiple paths are available. The sequence of relative coordinate points of historical trajectory information can be represented by a sequence of displacement increments in the robot's body coordinate system.

[0033] Based on the position of the task object in historical image frames, the quadruped robot crops the target region to generate navigation target information, such as a target template image. The target template image provides the model with visual reference information about the task target, enabling the model to focus on specific target objects in complex scenes, such as a specific object to be found or a pedestrian to be followed. By cropping rather than using the entire image as the template input, interference from irrelevant background information is reduced while preserving the key visual features of the target.

[0034] The quadruped robot also receives task instructions, such as natural language instructions. These instructions describe the specific content of the navigation task in text form, such as "go to the red door at the end of the corridor" or "follow the person in the black coat ahead." Natural language instructions provide high-level semantic guidance to the model, enabling the navigation system to understand the user's navigation intentions based on their flexible expressions, rather than being limited to a predefined set of fixed instructions.

[0035] The aforementioned visual perception information, motion history information, navigation target information, and task instruction information are combined to form multimodal input information. This multimodal input organization method uniformly encodes various complementary information such as visual observation, target reference, language semantics, and motion history into the model's input, enabling the model to perform comprehensive reasoning in a unified multimodal feature space. This avoids information loss and error accumulation caused by the independent processing of each module in traditional cascaded architectures. In some embodiments of the present invention, the multimodal input information includes at least two of the above four types of information. For example, it may include only visual perception information and task instruction information; or it may include a combination of visual perception information, navigation target information, and motion history information. Those skilled in the art can select appropriate information combinations according to the specific needs of the navigation task.

[0036] In the feature extraction stage, the aforementioned multimodal input information is input into the end-to-end navigation model. The general multimodal base model or multimodal feature extraction network within the end-to-end navigation model is used to extract fused features from the multimodal input information, resulting in hidden states or multimodal fusion features. The general multimodal base model can be the Qwen series visual language model, which is pre-trained on large-scale image and text data and possesses powerful visual understanding and language reasoning capabilities. Through deep feature extraction from the general multimodal base model, scene structure and dynamic changes in visual perception information, target visual features in navigation target information, semantic intent in task instruction information, and motion trends in motion history information are all encoded into unified hidden states or multimodal fusion features. In embodiments using the general multimodal base model, the last layer of hidden states is extracted because it contains the highest-level semantic abstraction information, which is most relevant and useful for subsequent path planning tasks. In embodiments using other multimodal feature extraction networks, the obtained multimodal fusion features also encode the high-level semantics of the multimodal input information and can be used for subsequent path planning.

[0037] In some embodiments of the present invention, feature extraction employs a pre-trained general multimodal base model to extract the hidden state of the last layer of the model; in other embodiments of the present invention, feature extraction may also employ other multimodal feature extraction networks to output multimodal fusion features.

[0038] During the latent variable generation phase, hidden states or multimodal fusion features are input into the planning adapter. The planning adapter is a key bridge connecting the feature extraction module and the action prediction head. Its core design idea is to distill information related to future path planning from hidden states or multimodal fusion features through a cross-attention mechanism using learnable path query vectors.

[0039] Specifically, the planning adapter first sets up K learnable path query vectors. These vectors learn continuously through gradient optimization during training, with the ultimate goal of learning to "ask" information related to future path planning from hidden states or multimodal fusion features. Using hidden states or multimodal fusion features as keys and values ​​for cross-attention, and path query vectors as queries, one or more cross-attention modules are constructed. In some embodiments of the invention, a single-layer cross-attention module can be used to reduce computational overhead; in other embodiments, a multi-layer cross-attention module can be used to achieve deeper information extraction. Through the cross-attention module, the path query vector can adaptively extract information related to the future path from hidden states or multimodal fusion features, while ignoring features irrelevant to path planning, achieving efficient information distillation from general semantic features to planning-specific features.

[0040] The multi-query features output by the cross-attention module are then flattened and mapped to fixed-length vectors via a multilayer perceptron projection network to obtain the planned latent variables. The dimensionality of the planned latent variables can be set according to actual needs, such as 256-dimensional or 512-dimensional. The purpose of flattening the multi-query features before projection is to integrate the scattered information extracted from multiple path query vectors into a compact, unified representation. The fixed-length design ensures the consistency of the input dimension of the subsequent action prediction head, while also reducing the feature dimensionality, which helps to improve inference efficiency.

[0041] This design of the planning adapter offers several advantages. First, the path query vector is learnable; the model can automatically learn which information to extract from the hidden state or multimodal fusion features based on the supervision signals from the training data, eliminating the need for manual feature selection strategies. Second, the cross-attention mechanism allows the planning adapter to flexibly focus on different positions and semantic levels within the hidden state or multimodal fusion features, extracting the planning information most relevant to the current navigation task. Finally, the planning adapter has far fewer parameters than the general multimodal base model, resulting in lower training and inference overhead. Furthermore, the parameters of the feature extraction module can be frozen during training, allowing training only the planning adapter and effectively preventing the disruption of existing knowledge in the general multimodal base model.

[0042] During the trajectory prediction stage, latent variables are planned and input into the motion prediction head, which then outputs the trajectory prediction results. The trajectory prediction results include at least one of path points, trajectory points, velocity commands, and attitude commands in the quadruped robot's body coordinate system or local coordinate system. In some embodiments of the present invention, the motion prediction head is a one-time non-autoregressive motion prediction head, which directly outputs the trajectory prediction results in a non-autoregressive manner. The trajectory prediction results may include, for example, a sequence of path points in the body coordinate system.

[0043] In some embodiments of the present invention, the one-time non-autoregressive motion prediction head includes two parallel prediction branches: a path point prediction head and a two-dimensional relative trajectory prediction head. The path point prediction head receives planning latent variables and directly outputs a sequence of path points containing position and orientation information as the main output using a one-time non-autoregressive regression method. The two-dimensional relative trajectory prediction head also receives planning latent variables and outputs multiple two-dimensional relative trajectory points accumulated and expanded from fixed anchor points as auxiliary outputs. The path point sequence output by the path point prediction head can be directly used for the motion control of a quadruped robot, while the two-dimensional relative trajectory points output by the two-dimensional relative trajectory prediction head provide a trajectory reference on the image plane. The collaborative design of the two branches enables the model to simultaneously possess three-dimensional spatial navigation and two-dimensional trajectory understanding capabilities.

[0044] The pathpoint prediction head employs a one-time non-autoregressive regression method instead of an autoregressive generation method, which is one of the key design features of this invention. In the autoregressive method, the model needs to generate pathpoints one by one, and the generation of each pathpoint depends on the output of all previous pathpoints, resulting in a serial inference process and a latency that increases linearly with the number of pathpoints. In contrast, the one-time non-autoregressive regression method directly outputs the complete pathpoint sequence in a single forward inference, with all pathpoints generated in parallel. The inference latency is independent of the number of pathpoints, greatly satisfying the real-time requirements of quadruped robot navigation. The length of the pathpoint sequence can be set according to the needs of the navigation task, for example, 4 to 10 pathpoints, each containing position and orientation information.

[0045] In some applications, quadruped robots can send trajectory prediction results to a control module, which then executes navigation motion control based on the predictions. For example, after receiving a sequence of path points in the body coordinate system, the control module converts it into motion commands for each joint of the quadruped robot, driving the robot to move along the predicted path and complete the navigation task.

[0046] The end-to-end navigation method provided in this invention completes the entire process from multimodal input to trajectory output within a unified model framework. It eliminates the need for intermediate interfaces and independent debugging between modules in traditional cascaded architectures, fundamentally eliminating the problem of error accumulation at each level. Simultaneously, the rich semantic understanding capabilities of the general multimodal base model, combined with the task adaptability of the planning adapter, enable the navigation system to flexibly plan paths based on natural language commands and visual perception, exhibiting strong generalization ability and task adaptability.

[0047] The following details the construction methods of each training dataset and the phased training strategy in the embodiments of the present invention.

[0048] The two-dimensional image trajectory point dataset in this embodiment of the invention is constructed in the following manner. First, scene video data is generated using a video generation model. Specifically, a visual language model such as Qwen3-VL can be used to generate a description of the first frame of the scene, and then a video generation model such as LTX2 can be used to generate continuous scene video data based on the first frame. The generated scene video data is sampled at a frequency of 1 to 2 Hz to obtain image frames. By using a video generation model instead of shooting real scenes, a large amount of diverse scene video data can be generated at a very low cost, effectively solving the problem of high cost of real scene data acquisition.

[0049] The sampled image frames are manually labeled to obtain target category, target location, description of the object to be followed, and trajectory supervision information. Manual labeling ensures the quality and accuracy of the training samples. The labeled content includes the category and location of the target in the scene, the description of the object to be followed, and the trajectory information used to supervise the output of the model.

[0050] Scene enhancement processing is performed on labeled image frames by adding preset object textures to simulate occlusion scenarios and simultaneously correcting trajectory information. The purpose of scene enhancement is to improve the model's robustness to complex scenes such as occlusion and interference. By randomly adding preset objects (such as pedestrians and obstacles) to the image, common occlusion situations in real navigation scenarios are simulated, while trajectory information is simultaneously corrected to ensure that the trajectory annotations are consistent with the enhanced scene. The enhanced training samples are suitable for various task types such as navigation, pedestrian following, and target search.

[0051] The data is organized according to a unified format to form a two-dimensional image trajectory point dataset. Each training sample includes the current frame image, historical frame images, template image, natural language instructions, trajectory markers from the previous time step, and multiple two-dimensional relative trajectory point labels for the current time step. In some embodiments of this invention, the two-dimensional relative trajectory point labels for the current time step consist of four two-dimensional relative trajectory points, which are accumulated and expanded starting from a fixed anchor point. For example, the anchor point can be set to the position (500, 979) in the image coordinate system, and the four two-dimensional relative trajectory points are represented as (dx1, dy1), (dx2, dy2), (dx3, dy3), and (dx4, dy4), respectively. The trajectory on the image plane is formed by accumulating and expanding sequentially starting from the anchor point. This representation method of accumulating and expanding starting from a fixed anchor point transforms the trajectory prediction problem into the prediction of a series of relative displacements, reducing the prediction difficulty and improving numerical stability.

[0052] The pathpoint training dataset in this embodiment of the invention is constructed as follows: First, the target building is scanned using a 3D scanning tool to construct a 3D Gaussian sputtering model. 3D Gaussian sputtering technology can reconstruct the 3D structure of a real building scene with high visual fidelity, providing a realistic virtual environment for subsequent simulation training.

[0053] The constructed 3D Gaussian sputtering model is imported into a simulation platform to build a simulation scene. The simulation platform can be a robot simulation platform such as Isaac Sim, which supports both physical and sensor simulations and can simulate a realistic robot operating environment.

[0054] Import the quadruped robot model into the simulation scene and complete the sensor configuration and communication environment configuration. For example, you can import the M20 quadruped robot model into the simulation scene, configure the same forward camera parameters as the real robot, and configure the ROS communication environment to achieve efficient data acquisition.

[0055] The system acquires commands, current field-of-view images, 2D image trajectory coordinates, and path point data in the 3D simulation scene within a simulated environment. In this environment, a quadruped robot can be controlled to move along a preset path, while simultaneously recording commands, camera images, 2D trajectory coordinates, and 3D path point information at various moments. The data acquisition efficiency is significantly higher than that of a real platform.

[0056] A pathpoint training dataset was compiled and annotated. The coordinates of the pathpoints are defined in the robot's body coordinate system, with the x-axis representing the forward direction, the y-axis representing the left direction, and θ representing the relative forward orientation angle (in radians). Each pathpoint is represented as (x, y, sin(θ), cos(θ)). sin(θ) and cos(θ) are used to represent the orientation angle instead of directly using the angle value because these functions avoid the abrupt change in angle between 0° and 360°, ensuring continuity in the orientation representation and facilitating model regression learning. Each training sample contains 10 future pathpoints, providing a longer-term path planning reference. The pathpoint training dataset was constructed using a 3D Gaussian sputtering simulation scenario, obtaining high-quality navigation training data containing 3D spatial information at low cost, effectively compensating for the high cost of data acquisition from real platforms.

[0057] In this embodiment of the invention, the general multimodal base model, planning adapter, and one-time non-autoregressive action prediction head are obtained through a phased training strategy. The phased training strategy is designed according to the principle of gradual accumulation of capabilities from easy to difficult, ensuring that the model can stably learn new capabilities at each stage while retaining the capabilities already learned.

[0058] In the first stage, the visual encoding parameters in the general multimodal base model are frozen. The general multimodal base model is then fine-tuned using a two-dimensional image trajectory point dataset to obtain a teacher model. This teacher model can output multiple two-dimensional relative trajectory points. In some embodiments of this invention, freezing the visual encoding parameters means preserving the visual feature extraction capabilities learned during the pre-training stage, and only fine-tuning the language model portion to adapt to the trajectory prediction task. This freezing strategy ensures the quality of visual features while reducing the computational cost of training and the risk of overfitting. The teacher model serves to provide supervisory signals and a capability foundation for two-dimensional trajectories in subsequent training stages.

[0059] In the second stage, the parameters of the general multimodal base model are frozen, and the planning adapter and the 2D relative trajectory prediction head are trained. Supervision is performed using a 2D image trajectory point dataset, enabling the planning latent variables to learn to incorporate 2D route intent. In some embodiments of this invention, the loss function in this stage is the L1 loss for 2D relative trajectory points. The purpose of freezing the parameters of the general multimodal base model is to protect the feature extraction capabilities obtained from the fine-tuning in the first stage, while allowing the planning adapter to focus on learning how to extract path planning-related information from the hidden states or multimodal fusion features. After this training stage, the planning adapter can transform the rich semantic features of the general multimodal base model into a compact representation with path planning semantics, with 2D route intent information encoded in the planning latent variables.

[0060] In the third stage, a warm start is performed using the training checkpoints from the second stage. The existing capabilities of the planning adapter and the 2D relative trajectory prediction head are retained, while the path point prediction head is integrated. Training is conducted using a hybrid approach, combining the 2D image trajectory point dataset and the path point training dataset. The 2D image trajectory point data constrains the 2D route prediction capabilities to prevent forgetting, while the path point training data supervises position, orientation, and smoothness. This warm start strategy allows the model to expand its 3D path point prediction capabilities beyond its existing 2D trajectory understanding capabilities without learning the path point prediction task from scratch, thus accelerating training convergence and improving final performance. In this hybrid training strategy, the 2D image trajectory point data continuously supervises the 2D relative trajectory prediction head, preventing the model from forgetting its mastered 2D trajectory prediction capabilities when learning new tasks, thus mitigating the catastrophic forgetting problem. The path point training data guides the model to learn the position, orientation, and smoothness of path points in the ontological coordinate system, achieving a transition from 2D image trajectories to 3D spatial paths.

[0061] The training loss function for the third stage is: L = λrel × Lrel + λwp × Lwp + λori × Lori + λsmooth × Lsmooth. Here, Lrel is the L1 loss for two-dimensional relative trajectory points, used to maintain the accuracy of two-dimensional trajectory prediction; Lwp is the path point position loss, used to supervise the accuracy of path point position prediction; Lori is the path point orientation loss, used to supervise the accuracy of path point orientation prediction; and Lsmooth is the path point smoothness loss, used to constrain the smooth transition between adjacent path points, avoiding sharp angles or jitters in the path, ensuring the stability and safety of the robot's motion. λrel, λwp, λori, and λsmooth are the weight coefficients of the corresponding losses, and their specific values ​​can be adjusted according to the training effect. Through the weighted combination of multiple loss terms, the model simultaneously optimizes multiple dimensions of trajectory prediction in the third stage of training, ensuring that the output path point sequence is both accurate and smooth.

[0062] The phased training strategy in this invention has significant technical advantages. First, phased training allows the model's capabilities to accumulate gradually, with each phase having a clear training objective and corresponding training data, reducing training difficulty and instability. Second, by freezing existing module parameters and employing a hybrid training strategy, catastrophic forgetting is effectively prevented, ensuring that the model acquires new capabilities without losing existing ones. Third, the introduction of the teacher model provides a high-quality feature base and training signals for subsequent phases, accelerating the convergence of the planning adapter. Finally, the collaborative use of the 2D image trajectory point dataset and the path point training dataset enables a gradual transfer from large amounts of low-cost 2D data to high-quality 3D data, balancing data scale and data quality.

[0063] After training, the general multimodal base model, planning adapter, 2D relative trajectory prediction head, and waypoint prediction head are combined into a complete end-to-end navigation model. During deployment and runtime, the quadruped robot inputs the multimodal input information collected in real time into the end-to-end navigation model. The model outputs the trajectory prediction result in one forward inference, which can be sent to the robot control module to execute navigation motion control. The entire inference process does not require autoregressive iteration, has low latency, and meets the real-time navigation requirements of quadruped robots.

[0064] The parameter settings and specific implementation methods of each module in this embodiment of the invention can be adjusted according to actual application scenarios. For example, the general multimodal base model can be selected from visual language models of different scales and architectures; the number K of path query vectors can be adjusted according to the accuracy requirements of path planning; the dimension of the planning latent variables can be selected according to the limitations of computing resources; and the length of the path point sequence can be set according to the look-ahead distance of the navigation task. All these adjustments are within the protection scope of this invention.

[0065] This invention also provides a quadruped robot, which includes a vision sensor, a memory, and a processor.

[0066] The vision sensor is used to acquire images of the current frame and historical frames. It can be configured as a forward-facing camera, mounted on the front of the quadruped robot's head or torso to acquire image information of the robot's forward field of view. The resolution and acquisition frequency of the vision sensor can be set according to the requirements of the navigation task.

[0067] The memory stores an end-to-end navigation model, which includes a general multimodal base model or multimodal feature extraction network, a planning adapter, and an action prediction head. The general multimodal base model or multimodal feature extraction network is used to extract features from multimodal input information and output hidden states or multimodal fusion features. The planning adapter is used to extract planning latent variables from the hidden states or multimodal fusion features through learnable path query vectors and cross-attention mechanisms. The action prediction head is used to map the planning latent variables to trajectory prediction results.

[0068] The processor is used to execute the aforementioned end-to-end navigation method for quadruped robots. The processor can be a general-purpose processor, digital signal processor, application-specific integrated circuit, or field-programmable gate array, etc., and can perform inference calculations on the end-to-end navigation model stored in memory, and convert the trajectory prediction results into motion control commands to drive the quadruped robot to perform navigation movements.

[0069] Similar to the above-described method embodiments, the quadruped robot provided in this invention also possesses the beneficial effects of multimodal fusion perception, end-to-end low-latency planning, and efficient task adaptation, which will not be elaborated further here.

[0070] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned end-to-end navigation method for a quadruped robot. The aforementioned computer-readable storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. Computer-readable storage media include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CDROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or apparatus.

[0071] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0072] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0073] Computer program code for performing the operations of embodiments of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0074] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0075] Obviously, those skilled in the art should understand that the various units or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps into a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0076] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An end-to-end navigation method for a quadruped robot, characterized in that, include: The multimodal input information of the quadruped robot in the navigation task is acquired, wherein the multimodal input information includes at least two of the following: visual perception information, navigation target information, task instruction information, and motion history information; The multimodal input information is input into the end-to-end navigation model, and the general multimodal base model or multimodal feature extraction network in the end-to-end navigation model is used to extract fusion features from the multimodal input information to obtain hidden states or multimodal fusion features. The hidden state or multimodal fusion features are input into the planning adapter, which extracts planning features related to future path planning from the hidden state or multimodal fusion features and generates planning latent variables. The planning latent variables are input into the motion prediction head, and the motion prediction head outputs the trajectory prediction result. The trajectory prediction result includes at least one of the following: path points, trajectory points, velocity commands, and attitude commands in the quadruped robot's body coordinate system or local coordinate system.

2. The end-to-end navigation method for a quadruped robot according to claim 1, characterized in that, The acquisition of multimodal input information of the quadruped robot in the navigation task includes: The current frame image and historical frame image are acquired by the forward-facing camera configured on the quadruped robot, and the current frame image and historical frame image are organized into the visual perception information in chronological order. Extract the motion history information of the quadruped robot and convert the motion history information into a sequence of relative coordinate points; The navigation target information is generated by cropping the target region based on the position of the task object in the historical frame image; The task instruction information is received, and at least two of the visual perception information, the relative coordinate point sequence, the navigation target information, and the task instruction information are combined to form the multimodal input information.

3. The end-to-end navigation method for a quadruped robot according to claim 1, characterized in that, The step of inputting the hidden state or multimodal fusion features into the planning adapter, and having the planning adapter extract planning features related to future path planning from the hidden state or multimodal fusion features, and generate planning latent variables, includes: Set up K learnable path query vectors, and use the hidden state or the multimodal fusion feature as the key and value of cross attention; Using the path query vector as the query and the hidden state or the multimodal fusion feature as the key and value, construct one or more cross-attention modules, and extract information related to the future path through the cross-attention modules; The multi-query features output by the cross-attention module are flattened, and the flattened features are mapped into a fixed-length vector through a multilayer perceptron projection network to obtain the planning latent variables.

4. The end-to-end navigation method for a quadruped robot according to claim 1, characterized in that, The action prediction head is a one-time non-autoregressive action prediction head, which includes a path point prediction head and a two-dimensional relative trajectory prediction head. The planning latent variables are input into the path point prediction head, and a one-time non-autoregressive regression method is used to directly output the path point sequence containing location and orientation information as the main output. The latent variables of the planning are input into the two-dimensional relative trajectory prediction head, and multiple two-dimensional relative trajectory points accumulated and expanded with fixed anchor points as the starting point are output as auxiliary outputs.

5. The end-to-end navigation method for a quadruped robot according to claim 1, characterized in that, The general multimodal base model, the planning adapter, and the one-time non-autoregressive action prediction head are obtained through the following phased training strategy: In the first stage, the visual coding parameters in the general multimodal base model are frozen, and the general multimodal base model is fine-tuned using a two-dimensional image trajectory point dataset to obtain a teacher model, which can output multiple two-dimensional relative trajectory points. In the second stage, the parameters of the general multimodal base model are frozen, the planning adapter and the two-dimensional relative trajectory prediction head are trained, and the two-dimensional image trajectory point dataset is used for supervision so that the planning latent variables learn to inherit the two-dimensional route intention. In the third stage, a warm start is performed using the training checkpoints from the second stage. The existing capabilities of the planning adapter and the two-dimensional relative trajectory prediction head are retained, the path point prediction head is connected, and the two-dimensional image trajectory point dataset and the path point training dataset are used together for training.

6. The end-to-end navigation method for a quadruped robot according to claim 5, characterized in that, The training loss function for the third stage is: L=λrel×Lrel+λwp×Lwp+λori×Lori+λsmooth×Lsmooth Where Lrel is the L1 loss of the two-dimensional relative trajectory point, Lwp is the path point position loss, Lori is the path point orientation loss, Lsmooth is the path point smoothness loss, and λrel, λwp, λori, and λsmooth are the weight coefficients of the corresponding losses.

7. The end-to-end navigation method for a quadruped robot according to claim 5, characterized in that, The pathpoint training dataset is constructed in the following manner: The target building is scanned using a 3D scanning tool to construct a 3D Gaussian sputtering model; Import the three-dimensional Gaussian sputtering model into the simulation platform to construct a simulation scene; Import the quadruped robot model into the simulation scene and complete the sensor configuration and communication environment configuration. In the simulation scenario, commands, current field-of-view image, coordinate points of two-dimensional image trajectory, and path point data in the three-dimensional simulation scenario are collected. The path point training dataset is formed by organizing and annotating the data. The coordinates of the path points are defined in the robot body coordinate system, with the x-axis for forward direction, the y-axis for left direction, and θ for the relative forward orientation angle. Each path point is represented as (x, y, sin(θ), cos(θ)).

8. The end-to-end navigation method for a quadruped robot according to claim 5, characterized in that, The two-dimensional image trajectory point dataset is constructed in the following way: Scene video data is generated using a video generation model, and image frames are obtained by sampling the scene video data. The sampled image frames are manually annotated to obtain target category, target location, description of the following object, and trajectory supervision information; Scene enhancement processing is performed on the labeled image frames. Preset object textures are added to the images to simulate occlusion scenes, and trajectory information is corrected simultaneously to form training samples suitable for navigation, pedestrian following and target search tasks. Organized according to a unified format, the two-dimensional image trajectory point dataset is formed. Each training sample includes the current frame image, historical frame images, template image, natural language instructions, trajectory markers from the previous time step, and multiple two-dimensional relative trajectory point labels for the current time step. The two-dimensional relative trajectory point labels are accumulated and expanded starting from a fixed anchor point.

9. A robot, characterized in that, include: A visual sensor, used to acquire current frame images and historical frame images; The memory stores an end-to-end navigation model, which includes a general multimodal base model or a multimodal feature extraction network, a planning adapter, and an action prediction head. The general multimodal base model or multimodal feature extraction network is used to extract features from multimodal input information and output hidden states or multimodal fusion features. The planning adapter is used to extract planning latent variables from the hidden states or multimodal fusion features through learnable path query vectors and cross-attention mechanisms. The action prediction head is used to map the planning latent variables to trajectory prediction results. A processor for performing the quadruped robot end-to-end navigation method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the end-to-end navigation method for a quadruped robot as described in any one of claims 1 to 8.

Images