Robot VLA planning method based on potential space planning

Abstract

The invention discloses a robot VLA planning method based on potential space planning, and belongs to the field of robot technology and artificial intelligence. The method comprises the steps of determining a current state point and a target state point in a preset low-dimensional potential space in response to visual observation and a language instruction; initializing a parameterized semantic guide cable used for representing a high-level semantic path and connecting the current state point and the target state point, and a feasible anchor point chain used for representing a physical feasible path and composed of a plurality of potential anchor points; constructing a composite potential energy field; executing an iterative optimization process; and after the optimization process is converged, decoding the final feasible anchor point chain into a robot action sequence. Through dual characterization and a dynamic optimization framework based on gradient descent, the core contradiction between planning efficiency and physical feasibility is solved, and the planning success rate and semantic consistency are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a robot VLA planning method based on potential spatial planning. Background Technology

[0002] In recent years, with the development of deep learning technology, endowing robots with the ability to understand high-level natural language instructions and combine them with visual perception to complete complex tasks (i.e., vision-language-action planning, VLA) has become a research hotspot in the field of robotics. Traditional robot planning methods usually perform direct searches in high-dimensional state spaces such as pixel space or joint angle space. Such methods face the challenge of the "curse of dimensionality," resulting in huge computational costs, low planning efficiency, and difficulty in effectively associating high-level, abstract semantic instructions with low-level, concrete physical actions.

[0003] To overcome the aforementioned shortcomings, latent space-based planning methods have emerged. These methods map high-dimensional visual observations and verbal commands to a unified, low-dimensional latent space using an encoder. Path planning is then performed within this low-dimensional space, and finally, a decoder converts the planned latent path into a sequence of actions executable by the robot. However, existing technologies still face a deep-seated core contradiction: to ensure high planning efficiency and macroscopic alignment with semantic commands, the representation of the latent path needs to be abstract and continuous; but to ensure the physical feasibility of the final action sequence (such as obstacle avoidance and kinematic constraint satisfaction), each step of the path must correspond to a specific and discrete, validated physical state. Existing methods attempt to simultaneously satisfy these two conflicting needs with a single path representation, often resulting in a planned path that is infeasible after decoding, or sacrificing global optimality and efficiency to ensure feasibility, failing to fundamentally resolve this inherent contradiction. Summary of the Invention

[0004] The first aspect of this application provides a robot VLA planning method based on latent space planning, aiming to resolve the inherent contradiction between abstract planning efficiency and concrete physical feasibility in existing robot VLA planning technologies. The method introduces a dual path representation of semantic guidelines and feasible anchor chains, and reconstructs the planning process into a numerical optimization problem with the goal of minimizing the total potential energy of the system. In an iterative gradient descent process, macroscopic semantic path planning and microscopic physical feasibility verification are carried out simultaneously and mutually correct each other, thereby efficiently generating action sequences highly consistent with high-level instruction semantics while ensuring physical feasibility.

[0005] To achieve the above objectives, the first aspect of this application provides a robot vision-language-action (VLA) planning method based on latent space planning, comprising: determining a current state point and a target state point in a preset low-dimensional latent space based on the robot's visual observations and received language instructions; initializing a semantic guideline and a feasible anchor chain, wherein the semantic guideline is a parameterized curve connecting the current state point and the target state point for representing a high-level semantic path, and the feasible anchor chain is a discrete set of points composed of multiple ordered potential anchors for representing a physically feasible path; performing an iterative optimization process, in which, based on a preset total system potential function, the shape of the semantic guideline and the position of each potential anchor in the feasible anchor chain are updated synchronously through gradient descent until the total system potential function converges to a local minimum; and after the iterative optimization process converges, generating a robot action sequence based on the final feasible anchor chain.

[0006] In one possible implementation of the first aspect, the initialization of the semantic guideline and the feasible anchor chain includes: initializing the semantic guideline as a straight line or a preset smooth curve connecting the current state point and the target state point; and generating the plurality of potential anchors in the feasible anchor chain along the initialized semantic guideline by uniform sampling or non-uniform sampling.

[0007] In one possible implementation of the first aspect, the total potential energy function of the system includes at least one potential energy term defined as follows: a semantic attraction potential, a physical repulsion potential, or an intrachain smoothing potential.

[0008] In one possible implementation of the first aspect, the semantic attraction potential is determined based on the distance between each potential anchor in the feasible anchor chain and the semantic guide line, and the gradient of the semantic attraction potential generates an update amount for pulling the potential anchors toward the direction of the semantic guide line.

[0009] In one possible implementation of the first aspect, the calculation of the semantic attraction potential energy includes: for each potential anchor in the feasible anchor chain, determining its nearest projection point on the current semantic guide; and obtaining the semantic attraction potential energy value of the potential anchor based on the square of the Euclidean distance between the potential anchor and the nearest projection point, combined with a preset attraction coefficient.

[0010] In one possible implementation of the first aspect, the physical repulsion potential is determined based on the degree of conflict between the robot's physical state corresponding to the potential anchor and the constraints of the robot's physical environment, and the gradient of the physical repulsion potential generates an update amount for pushing the potential anchor away from the potential spatial region corresponding to the infeasible physical state.

[0011] In one possible implementation of the first aspect, the calculation of the physical repulsion potential energy includes: decoding the potential anchor point into a corresponding robot physical state; performing a feasibility verification on the robot physical state, the feasibility verification including collision detection or kinematic solvability analysis; and when the result of the feasibility verification is infeasible, obtaining a non-zero physical repulsion potential energy value based on a function used to quantify the severity of the conflict.

[0012] In one possible implementation of the first aspect, the intra-chain smoothing potential is determined based on the relative positions between adjacent potential anchors in the feasible anchor chain, and the gradient of the intra-chain smoothing potential generates an update amount for maintaining the local smoothness of the feasible anchor chain.

[0013] In one possible implementation of the first aspect, during the iterative optimization process, the morphology of the semantic guideline is updated synchronously, including: generating a new smooth curve as the updated semantic guideline by using curve fitting or interpolation methods to generate the set of positions of all potential anchor points in the feasible anchor chain updated based on gradient descent. This implementation reflects the reverse correction effect of the feasible anchor chain on the semantic guideline. When the anchor chain deviates due to physical constraints, the semantic guideline will also adaptively deform accordingly, enabling macroscopic path planning to perceive and adapt to microscopic physical constraints.

[0014] In one possible implementation of the first aspect, generating a robot action sequence based on the final feasible anchor chain includes: sequentially inputting adjacent potential anchor pairs in the final feasible anchor chain into a pre-trained action decoder to generate a series of corresponding robot actions; and combining the series of robot actions to form the robot action sequence.

[0015] A second aspect of this application provides a robot VLA planning device capable of performing the above-described method. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a robot VLA planning method based on potential space planning according to an embodiment of this application.

[0017] Figure 2 This is a schematic diagram illustrating the initialization of semantic guide lines and feasible anchor chains in a latent space according to an embodiment of this application.

[0018] Figure 3 This is a schematic diagram of different potential energy gradient components acting on a potential anchor point in a feasible anchor chain, according to an embodiment of this application.

[0019] Figure 4 This is a schematic diagram illustrating the state evolution of a semantic guideline and a feasible anchor chain during an iterative optimization process, according to an embodiment of this application.

[0020] Figure 5 This is a structural block diagram of a robot VLA planning device according to an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0023] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0024] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0025] Example 1

[0026] This embodiment provides a robot vision-language-action (VLA) planning method based on latent space planning. This method unifies the traditionally separate high-level task planning and low-level motion planning into a dynamically coupled framework based on numerical optimization. By introducing a dual path representation of semantic guidelines and feasible anchor chains, and modeling the planning process as a gradient descent process seeking energy minimization in a composite potential field, this method fundamentally resolves the inherent contradiction in robot planning between the abstract adherence to high-level semantics and the concrete satisfaction of low-level physical constraints. This method ensures that the planned path is not only highly consistent with the user's linguistic intent on a macroscopic level, but also ensures physical feasibility at every microscopic step, including collision-free operation and satisfaction of kinematic constraints.

[0027] like Figure 1 As shown, the method includes:

[0028] S100: In a pre-defined low-dimensional potential space, based on the robot's visual observation and received language instructions, determine a current state point and a target state point.

[0029] This step transforms the high-dimensional, heterogeneous raw information received by the robot from its sensors (such as cameras) and human-computer interaction interfaces (such as text input) into a unified, structured, low-dimensional mathematical representation. The latent space is a vector space trained on massive amounts of data using a deep learning model. This space possesses a key characteristic: the points and vectors within it can effectively encode semantic and physical state information relevant to the robot's task.

[0030] In this low-dimensional space, the complex VLA task is simplified to a well-defined path planning problem: finding a path from the "current state point". Reach the "target state point" The path.

[0031] To achieve this mapping from high-dimensional perception to low-dimensional state definition, this step employs a set of collaborative deep neural network models, specifically including a visual encoder for processing visual information. A language encoder for processing language information. And a multimodal fusion module for integrating multimodal information and ultimately determining the target state point. .

[0032] The visual encoder Used to process high-dimensional image data (For example, a) The RGB image tensor is compressed into an information-dense one. Dimension (for example, ) eigenvectors In this embodiment, the visual encoder It is designed as a Vision Transformer (ViT) based architecture, which includes an image segmentation and linear embedding module, a Transformer encoder layer and a final output module.

[0033] For example, input a size of RGB image tensor .like Pixels Pixels One color channel.

[0034] Image segmentation and linear embedding module on input image tensors Perform a non-overlapping meshing operation. Use a mesh of size [size missing]. The window (for example, (pixels), starting from the top left corner of the image, with Using a step size, sliding cuts are performed in both the horizontal and vertical directions. This process decomposes the original single image into a series of... A grid of image patches. For Input and The block size will generate Image blocks.

[0035] Each (Right now The image patches were all flattened into a single image of length 1. The module then uses a trainable linear projection layer (typically implemented with nn.Linear in PyTorch) to linearly map this 768-dimensional vector to a pre-defined hidden dimension that runs throughout the entire Transformer model. For example, This step transforms each image patch from pixel space to feature space.

[0036] To perform classification or global feature extraction, this module prepends a learnable [CLS] (Classification) embedding vector at the beginning of the 196 image patch feature vector sequence. Simultaneously, to enable the model to understand the original spatial relationships between the image patches (since the subsequent self-attention mechanism is position-independent), a similarly learnable embedding vector with dimensions of […]. The positional embedding is added element-wise to each embedding vector, including the [CLS] symbol.

[0037] The image segmentation and linear embedding module ultimately outputs a shape of (Right now The embedding vector sequence tensor is the input to the subsequent Transformer encoder layer.

[0038] The Transformer encoder layer is the core feature extraction unit of the visual encoder, and it consists of... It consists of stacked identical encoder layers. For example, Each encoder layer contains two key sub-layers.

[0039] The shape is from the previous encoder layer (or, for the first layer, from submodule one). The sequence tensor is fed into a multi-head self-attention (MSA) sublayer. In this sublayer, each vector of the input sequence is transformed into three vectors through three independent linear transformations: Query (Q), Key (K), and Value (V). The core mechanism of MSA is to compute the dot product similarity (attention score) between each vector's Q and the K of all other vectors (including itself), and then use these scores to weight and aggregate the V of all vectors. This process allows the model to compute the importance of "each part" of the image to "all other parts," thereby capturing long-range dependencies. The "multi-head" mechanism (exemplarily, the number of heads...) ) refers to The feature space of dimension is divided into Each subspace performs the self-attention computation independently, and the results are then concatenated. This allows the model to simultaneously focus on information from different representation subspaces. The operations in this sublayer are typically followed by a residual connection and layer normalization.

[0040] The sequence tensor, after processing by the MSA sublayer, then enters the feed-forward network (FFN) sublayer. This is a simple, position-independent, two-layer fully connected neural network. The first layer will... The features of one dimension are extended to a higher dimension (e.g., The second layer then projects it back onto the target layer, using a non-linear activation function (such as GELU). The purpose of this sublayer is to perform a nonlinear transformation at each position in the sequence, enhancing the model's representational power. Similarly, this sublayer is followed by a residual connection and layer normalization.

[0041] go through After sequential processing by each encoder layer, this module outputs a sequence tensor with the exact same shape as the input, but containing deeper and more global feature information. Its shape remains the same. .

[0042] The Final Output Module is the final feature aggregation and output unit of the visual encoder. It receives features from the last Transformer encoder layer, and the feature aggregation module is of shape [missing information]. The sequence tensor.

[0043] This module only focuses on the first position in the sequence, that is, the one corresponding to the [CLS] symbol. dimension( The output vector is a [CLS] symbol that interacts with all image patch features throughout the encoding process, ultimately aggregating the global information of the entire image. To obtain features that ultimately conform to the latent spatial dimension of this application, this module employs a linear projection layer to transform this... A vector of dimension is mapped to the latent space dimension predefined in this application. For example, The final output is a... dimension( The feature vector (in dimensional) is the feature vector of the current visual observation. The final encoding result And it is directly used as the current state point in the latent space. .

[0044] The visual encoder is trained on a large-scale dataset containing robot first-person view images and corresponding proprioceptive states (such as joint angles and end effector poses) using self-supervised learning methods, such as masked image modeling or contrastive learning, to pre-train the model and enable it to learn general visual representations.

[0045] The language encoder Used to process natural language instruction strings Transform into one that is also in that position Target vector in dimensional latent space In this embodiment, It adopts an architecture based on BERT (Bidirectional Encoder Representations from Transformers). Specifically, it includes an input embedding module, Transformer Encoder Layers, and a pooling and output module.

[0046] The input embedding module is the input processing unit of the language encoder, responsible for converting the original text string into a numerical tensor that can be processed by the subsequent neural network. It receives a UTF-8 encoded natural language instruction string. For example, ="Please put the red square into the blue bowl".

[0047] This module first uses a pre-trained WordPiece tokenizer to process the input string. The process involves a tokenizer that breaks down a string into a series of tokens. Common words are mapped to single tokens (e.g., "bowl" -> ["bowl"]), while less common words or roots / suffixes are broken down into sub-word units (e.g., "stacking" -> ["stack", "##ing"]). This process also automatically adds special symbols to the beginning and end of the sequence: [CLS] (for converging the representation of the entire sequence) and [SEP] (for separating sentences). Each token, including the special symbols, is mapped to a unique integer index (ID) in the tokenizer's vocabulary. This module converts the sequence of integer indices into a sequence of real-valued vectors through a trainable embedding layer. The final embedding vector is the sum of the following three parts:

[0048] Token Embeddings: Each token ID is retrieved from an embedding matrix to obtain its corresponding base vector. This matrix has multiple dimensions. ,in It refers to the size of the vocabulary list. It is the hidden dimension of the model (for example, ).

[0049] Position Embeddings: To enable the model to understand the order of words, a learnable positional encoding vector is added to each word embedding. This encoding vector corresponds one-to-one with the position of the word in the sequence.

[0050] Segment embeddings are used to distinguish different sentences. For a single instruction input in this application, all lexical units belong to the same segment, and therefore a common, learnable segment embedding vector A is added.

[0051] The input embedding module ultimately outputs a shape of The embedding vector sequence tensor, where It is the total length of the sequence after word segmentation. It is the hidden dimension of the model (i.e. This tensor is the direct input to subsequent Transformer encoder layers.

[0052] The Transformer encoder layer is the core semantic understanding unit of the language encoder, and it consists of... It consists of stacked identical encoder layers. For example, It includes a multi-head self-attention (MSA) sublayer and a feed-forward network (FFN) sublayer. Similar to the MSA in a visual encoder, this sublayer (exemplarily, with h heads) It calculates the association strength between each lexical unit in the sequence and all other lexical units. This enables the model to resolve ambiguity and long-distance dependencies in language. For example, in the instruction "pick it up," the self-attention mechanism can associate the pronoun "it" with the specific object mentioned earlier (such as "that red square").

[0053] The FFN sublayer performs an independent, non-linear transformation on each word representation in the sequence to enhance the model's representational power. Each encoder layer includes residual connections and layer normalization to ensure training stability and efficiency. After sequential processing by each encoder layer, this module outputs a sequence tensor with the exact same shape as the input, but the vector representation of each word has been deeply integrated with its contextual semantic information. Its shape remains the same. .

[0054] The pooling and output module is the final information aggregation and output unit of the language encoder. The goal of this module is to extract a single vector representing the global instruction intent from the representation of the entire sequence. In this embodiment, the standard [CLS] pooling strategy is used. That is, the first position in the sequence (corresponding to the [CLS] special symbol) is directly selected. dimension( (Dimensional) output vector. Because the [CLS] symbol is in... During the layer encoding process, the layer exchanges information fully with all other tokens in the instruction through a self-attention mechanism, and its final vector representation is considered a high-level generalization of the entire instruction sequence.

[0055] In order to make this The global representation of the dimension is mapped to the latent spatial dimension pre-defined in this application and aligned with visual features. (For example, This module uses a linear projection layer to reduce the dimensionality of the [CLS] vector. The final output is a... dimension( (dimensional) target vector The direction and position of this vector in the latent space encode the core intent of the user's language instructions.

[0056] This language encoder can be trained in a supervised manner on a dataset containing a large number of (language instruction, corresponding target state) pairs. The loss function is typically designed to minimize the encoded language target vector. Latent encoding with the true target state The cosine distance or mean square error between them.

[0057] The multimodal fusion module Used to interpret visual context and language goals To effectively combine these elements and accurately pinpoint the final mission objective state point. This is because the interpretation of language instructions often depends on visual context. In this embodiment, A cross-attention module is employed. This module consists of two stacked cross-attention layers. In each layer, visual features... As a query, and language features These are used as keys and values. This allows the model to "focus" on relevant parts of the visual scene based on language instructions. The output of the cross-attention is then passed through a small multilayer perceptron (MLP) to finally generate... dimensional target state vector .

[0058] This fusion module is trained end-to-end with the language encoder, enabling the entire system to learn how to correctly "ground" language instructions based on visual context.

[0059] For example, suppose the latent space used in this application is a simplified two-dimensional space for ease of visualization (i.e. Its coordinate range is The visual observations received by the robot After visual encoder After processing, it is mapped to the current state point in the latent space. This dimensionless coordinate point represents the robot's current initial configuration within the model. Simultaneously, the language commands received by the robot... "pickup the green apple" after language encoder After processing, the target vector was generated. Subsequently, the multimodal fusion module Will The visual context and The fusion module integrates the target language. For example, there may be multiple apples in a scene, but the fusion module uses an attention mechanism to identify which apple is the specific "green apple" mentioned in the instruction, and calculates the final state that the action of picking up that apple should achieve. After calculation by the fusion module, the final target state point is determined as follows: Thus, a complex, multimodal VLA task has been successfully transformed into a task in a two-dimensional latent space, starting from a point... Move to point The path planning problem.

[0060] S200: Initialize a semantic guideline and a feasible anchor chain.

[0061] The starting point for the planning was determined in S100. and the end point The core task of step S200 is to construct two core data structures for the subsequent iterative optimization process: a "semantic guideline" and a "feasible anchor chain." These two structures directly embody the core contradiction separation concept of this invention. The semantic guideline carries the "abstract-continuous" attribute of the planned path, providing a macroscopic, smooth path guide that conforms to semantic intent. The feasible anchor chain carries the "concrete-discrete" attribute of the planned path, consisting of a series of discrete points that will be used for actual interaction and verification with the physical world. The initialization work in this step aims to provide a reasonable and simple initial form for these two data structures as the starting point for subsequent iterative optimization.

[0062] S210: Initialize the semantic guideline.

[0063] The semantic guide cable is mathematically defined as a parametric curve. , where parameters Every point on the curve They are all in the potential space Dimensional vector. Defined as , Defined as In the initialization phase, the most direct approach is to construct it as a straight line segment. The initial semantic guideline... The mathematical expression is:

[0064]

[0065] This straight line represents the most direct path from the starting point to the ending point without considering any constraints.

[0066] S220: Initialize the feasible anchor chain.

[0067] The feasible anchor chain is defined as a chain consisting of... An ordered set of potential anchor points The purpose of initializing this chain is to scatter a series of discrete "probe points" on the initial semantic guideline. A uniform sampling strategy is employed. Initial position of each anchor point The calculation formula is:

[0068] ,in, .

[0069] Number of anchor points It is a hyperparameter, usually with a value between 0 and 10. The specific value needs to be adjusted according to the complexity of the task and the requirements for path granularity.

[0070] For example, continuing with the two-dimensional latent space example established in S100, where , In step S200, semantic guidance is used. It is initialized as a straight line connecting the two points. Assume the system selects a certain number of anchor points. The system will follow Uniform sampling was performed to generate 21 anchor points. For example, the 10th anchor point located exactly in the middle of the path ( The initial position of ) is:

[0071] .

[0072] like Figure 2 As shown, after initialization, a semantic guide line in the shape of a straight line is obtained, and a series of potential anchor points are uniformly attached to it, which constitute the initial state of the feasible anchor point chain.

[0073] S300: Perform an iterative optimization process.

[0074] This step evolves the initial, idealized dual representation in S200 into a final stable state that simultaneously satisfies high-level semantic constraints and low-level physical constraints through a numerical optimization process. This process is modeled as a gradient descent problem seeking energy minimization in a composite potential field. The entire process is carried out in an iterative loop. In each iteration, the system calculates the gradient of the total potential function with respect to each anchor point position and updates the anchor point positions along the negative gradient direction; subsequently, based on the updated anchor point chain positions, the shape of the semantic guide rope is updated in reverse.

[0075] The entire feasible anchor chain The total potential energy of the system Defined as a weighted sum of three main potential energy terms:

[0076]

[0077] in, It is an anchor chain , It is a semantic guideline. These are the weighting coefficients for each item, used to balance the relative importance of different optimization objectives.

[0078] Semantic attraction potential This is used to ensure that the overall orientation of the feasible anchor chain is consistent with the semantic guideline. It is defined as the sum of the squares of the distances from all anchors to the guideline:

[0079]

[0080] in, Anchor point In the guide cable The nearest projection point on the surface. The gradient of this potential energy. It will produce a Pull towards its projection point The update vector, such as Figure 3 The semantic attraction gradient components are shown in the figure.

[0081] Physical repulsive potential energy This term is used to achieve physical obstacle avoidance and satisfy other physical constraints. It is defined as the sum of the physical conflict levels corresponding to all anchor points:

[0082]

[0083] in, It is a state decoder that decodes potential anchors into physical states. It is a conflict function used to quantize physical states. The degree of infeasibility. For example, if the state Collision with an obstacle, conflict function It can be defined as the square of the penetration depth. If the state If no collision occurs, then The gradient of this potential energy It will produce a The update vector is pushed away from the potential spatial region corresponding to the infeasible physical state.

[0084] Intrachain smoothing potential energy This term is used to maintain the local smoothness of the anchor chain and avoid abrupt inflection points. A common definition is to minimize the approximation of the path length, i.e., the sum of the squares of the distances between adjacent anchor points:

[0085]

[0086] Another alternative definition is an approximation that minimizes curvature, for example:

[0087]

[0088] This embodiment uses the latter because it more effectively smooths the path. The gradient of this potential energy. It will produce a The update vector is pushed to the midpoints of its preceding and following neighbors.

[0089] The iterative optimization process is carried out in a loop until a convergence condition is met (e.g., the change in total potential energy is less than a threshold). Or reach the maximum number of iterations ). In the In the next iteration:

[0090] S310: Update feasible anchor chain.

[0091] For each movable anchor point ( ), calculate the total potential energy function Negative gradient with respect to its position Then, the anchor point position is updated using gradient descent:

[0092]

[0093] in, It is the learning rate (or step size factor), a hyperparameter that controls the step size for each update.

[0094] S320: Update semantic guideline.

[0095] At the position of the anchor chain Following the update, the semantic guideline also needs to be updated accordingly to adapt to the changes in the anchor chain. This reflects the reverse correction of the macroscopic semantic path by specific physical constraints. The update method involves updating the set of anchors... As a set of new control points, a new curve that smoothly passes through or approximates these points is generated using curve fitting methods (e.g., B-spline curve fitting). .

[0096] like Figure 4 As shown, when the feasible anchor chain encounters an obstacle region, the physical repulsive potential energy causes some anchors to deviate from the initial path. After multiple iterations, the anchor chain forms a smooth path that bypasses the obstacle, and the semantic guide rope also deforms accordingly, eventually converging to a physically feasible optimal solution.

[0097] The performance of this optimization process depends on the proper configuration of a series of hyperparameters. The following table shows an example configuration:

[0098] parameter symbol Example value dimension illustrate Attractive potential weight 1.0 Dimensionless The importance of balancing semantic compliance Repulsive potential weight 10.0 Dimensionless Emphasizing the priority of physical obstacle avoidance Smoothing potential weights 0.2 Dimensionless Control the smoothness of the path Number of anchor points 31 indivual Determines the resolution of the path Learning rate 0.05 Dimensionless Controlling the convergence speed and stability of optimization Maximum number of iterations 200 Second-rate One of the optimized termination conditions Convergence threshold 1e-5 (Unit of potential energy) Second optimized termination condition

[0099] State Decoder This is used to decode a single latent anchor point into a robot's physical state (e.g., a 7-DOF joint angle). It is implemented as an MLP with 4 hidden layers, each layer containing 1 / 2 neurons, using ReLU activation function, and the output layer has no activation function. Its training data consists of a large number of randomly sampled or expert-demonstrated robot configurations. The latent encoding is obtained through a pre-trained encoder (which can be part of a visual encoder), and then the decoder is trained to reconstruct the original configuration. The loss function is mean squared error.

[0100] Action Decoder This is used to decode a pair of adjacent potential anchor points into a robot action (e.g., 7-DOF joint velocity). It is implemented as an MLP, taking the concatenated vector of the two anchor points as input. (Dimensional), the output is an action vector. Its network structure is as follows. The training data consists of expert demonstration trajectories, including... For triples, the loss function is also the mean squared error.

[0101] S400: After the iterative optimization process converges, a robot motion sequence is generated based on the final feasible anchor chain.

[0102] When the iterative optimization process of S300 converges, the system obtains a final, stable, and feasible anchor chain. The task of the S400 is to transform this sequence of discrete path points in the abstract latent space into a specific, time-sequential sequence of action instructions that the robot's underlying controller can understand and execute.

[0103] S410: Serialization and decoding.

[0104] The system will traverse the final anchor chain. ,from arrive For each pair of adjacent anchor points splice them together into one A 3D vector is input into the pre-trained action decoder. In the middle, generate an action instruction. .

[0105]

[0106] S420: Combination action sequence.

[0107] The actions generated for each pair of adjacent anchor points in S410 Arranged in order, they form a complete sequence of actions in discrete time steps. This sequence of actions can then be sent to the robot's underlying motion controller, which is responsible for smoothly interpolating and executing these instructions, thereby driving the robot to complete the entire task.

[0108] Example 2

[0109] like Figure 5 As shown, this embodiment provides a robot VLA planning device. The internal structure and working principle of the device are designed to implement the method described in Embodiment 1. The device can be a hardware module integrated into the robot's main controller, or a software system running on a general-purpose computing device (such as a server with a GPU or an embedded computer).

[0110] The device includes:

[0111] The state determination module 510 is responsible for executing step S100. This module integrates the weights of the pre-trained visual encoder, language encoder, and multimodal fusion model, and is configured with an interface for receiving sensor data and user commands.

[0112] The characterization initialization module 520 is responsible for executing step S200. This module receives the output of the state determination module and, based on preset parameters (such as the number of anchor points), executes the initialization. Generate the initial semantic guideline and feasible anchor chain.

[0113] The coupling optimization module 530 is the core computing unit of the device, preferably implemented on a GPU. This module is responsible for executing the iterative optimization process of the S300. It stores the definition of the potential energy function, all hyperparameters, and communicates at high speed with a physical simulation environment for feasibility verification.

[0114] The motion generation module 540 is responsible for executing step S400. This module integrates a pre-trained motion decoder model and transforms the finally converged anchor chain into an executable motion sequence, which is then sent to the robot controller.

[0115] In practice, these four modules work together to achieve an end-to-end, robust, and efficient planning process from high-level multimodal perception to low-level physical actions.

[0116] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to depart from the spirit and scope of the technical solutions of the embodiments of this application.

[0118] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0120] The units described as separate components may or may not be physically separate. A component shown as a unit can be one physical unit or multiple physical units; that is, it can be located in one place or distributed in multiple different places. Depending on actual needs, some or all of the units can be selected to achieve the purpose of this embodiment.

[0121] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware.

[0122] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A robot vision-language-action (VLA) planning method based on latent spatial planning, characterized in that, include: In a pre-defined low-dimensional potential space, a current state point and a target state point are determined based on the robot's visual observations and received language instructions. Initialize a semantic guideline and a feasible anchor chain, wherein the semantic guideline is a parameterized curve that connects the current state point and the target state point to characterize a high-level semantic path, and the feasible anchor chain is a discrete set of points consisting of multiple ordered potential anchors to characterize a physically feasible path. An iterative optimization process is performed. In each iteration, based on a preset total system potential function, gradient descent is used to synchronously update the shape of the semantic guide cable and the position of each potential anchor in the feasible anchor chain until the total system potential function converges to a local minimum. After the iterative optimization process converges, a robot motion sequence is generated based on the final feasible anchor chain.

2. The method according to claim 1, characterized in that, The initialization of the semantic guideline and the feasible anchor chain includes: The semantic guide line is initialized as a straight line or a preset smooth curve connecting the current state point and the target state point; Along the initialized semantic guideline, the plurality of potential anchors in the feasible anchor chain are generated by uniform or non-uniform sampling.

3. The method according to claim 1, characterized in that, The total potential energy function of the system includes at least one potential energy term defined as follows: a semantic attraction potential, a physical repulsion potential, or an intrachain smoothing potential.

4. The method according to claim 3, characterized in that, The semantic attraction potential is determined based on the distance between each potential anchor in the feasible anchor chain and the semantic guide line, and the gradient of the semantic attraction potential generates an update amount for pulling the potential anchors toward the direction of the semantic guide line.

5. The method according to claim 4, characterized in that, The calculation of the semantic attraction potential includes: For each potential anchor in the feasible anchor chain, determine its nearest projection point on the current semantic guideline; The semantic attraction potential energy value of the potential anchor point is obtained by taking the square of the Euclidean distance between the potential anchor point and the nearest projection point, and combining it with a preset attraction coefficient.

6. The method according to claim 3, characterized in that, The physical repulsion potential is determined based on the degree of conflict between the robot's physical state corresponding to the potential anchor point and the constraints of the robot's physical environment. The gradient of the physical repulsion potential generates an update amount to push the potential anchor point away from the potential spatial region corresponding to the infeasible physical state.

7. The method according to claim 6, characterized in that, The calculation of the physical repulsive potential energy includes: Decode the potential anchor point into a corresponding robot physical state; The feasibility of the robot's physical state is verified, including collision detection or kinematic solvability analysis. When the feasibility verification result is infeasible, a non-zero physical repulsion potential energy value is obtained based on a function used to quantify the severity of the conflict.

8. The method according to claim 3, characterized in that, The intra-chain smoothing potential energy is determined based on the relative positions between adjacent potential anchors in the feasible anchor chain, and the gradient of the intra-chain smoothing potential energy generates an update amount to maintain the local smoothness of the feasible anchor chain.

9. The method according to claim 1, characterized in that, During the iterative optimization process, the form of the semantic guide is updated synchronously, including: Based on the set of positions of all potential anchors in the feasible anchor chain updated by gradient descent, a new smooth curve is generated by curve fitting or interpolation methods to serve as the updated semantic guide.

10. The method according to claim 1, characterized in that, The generation of a robot action sequence based on the final feasible anchor chain includes: The adjacent potential anchor pairs in the final feasible anchor chain are sequentially input into a pre-trained action decoder to generate a series of corresponding robot actions. The series of robot actions are combined to form the robot action sequence.

Images