Robot action generation method and system thereof, medium, equipment and program product

Abstract

The embodiment of the invention provides a robot action generation method and system, a medium, equipment and a program product. The method comprises the following steps: acquiring visual input data of a robot and corresponding language instruction data; obtaining an intermediate action representation by using a pre-constructed cross-modal shared semantic space, a corresponding relationship between the shared semantic representation and the intermediate action representation, the visual input data and the language instruction data; wherein the shared semantic representation is obtained on the basis of visual input data and / or language instruction data by using a cross-modal shared semantic space; and action parameters are generated according to the intermediate action representation so as to drive the robot to execute corresponding actions. Due to the fact that the cross-modal shared semantic space can enable semantic information from different modalities to be measured under the unified scale, and the corresponding relation can serve as middle bridging representation from semantics to middle action representation, the ability of a robot to understand and execute common language instructions in diversified real scenes is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of robotics, and in particular to a robot action generation method and system, a medium, equipment and a program product. BACKGROUND

[0002] With the significant progress of visual-language models (VLM) in understanding images and texts, how to effectively connect natural language instructions with the perception and operation capabilities of robots has become one of the key problems for agents to perform tasks in real environments. In scenarios such as home services, warehouse transportation, and human-robot collaboration, users often issue instructions in natural language, expecting robots to perform corresponding actions based on visual perception. In actual interactions, users often issue instructions in daily language, which is often not specific enough or contains ambiguities, making it difficult to reliably map language intentions into executable actions. Therefore, it is necessary to improve the understanding and execution capabilities of robots for ordinary language instructions in diverse real scenarios to ensure the accuracy and robustness of tasks.

[0003] Based on this, embodiments of the present application provide a robot action generation method and system, a medium, equipment and a program product to improve related technologies. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a robot action generation method and system, a medium, equipment and a program product to improve the accuracy and robustness of task execution.

[0005] The purpose of the embodiments of the present application is achieved by using the following technical solutions:

[0006] In a first aspect, the embodiments of the present application provide a robot action generation method, which includes: obtaining visual input data of a robot and corresponding language instruction data thereof; obtaining an intermediate action representation by using a pre-constructed cross-modal shared semantic space, a correspondence between shared semantic representations and intermediate action representations, the visual input data and the language instruction data; wherein the shared semantic representation is obtained based on the visual input data and / or the language instruction data and by using the cross-modal shared semantic space; and generating action parameters according to the intermediate action representation, for driving the robot to perform an action corresponding to the visual input data and / or the language instruction data.

[0007] In some embodiments, the process of obtaining the intermediate action representation includes: converting the shared semantic representation into an action continuous hidden vector and / or an action discrete hidden vector by using the cross-modal shared semantic space and the correspondence; and taking the action continuous hidden vector and / or the action discrete hidden vector as the intermediate action representation.

[0008] In some embodiments, the process of obtaining the intermediate action representation comprises: converting the shared semantic representation into an action continuous latent vector using the cross-modal shared semantic space; and determining a corresponding action discrete latent vector based on the action continuous latent vector and the correspondence, and taking the action discrete latent vector as the intermediate action representation.

[0009] In some embodiments, the process of generating the action parameter according to the intermediate action representation comprises: generating the action parameter using the action discrete latent vector and the shared semantic representation.

[0010] In some embodiments, the correspondence is based on an action latent vector table as an underlying representation space, and the action latent vector table contains a first number of specified discrete latent vectors, wherein the action discrete latent vector is obtained by attention matching a second number of the specified discrete latent vectors, and the second number is less than the first number.

[0011] In some embodiments, the method further comprises: obtaining a task type corresponding to the visual input data and / or the language instruction data; and wherein the process of generating the action parameter according to the intermediate action representation comprises: determining an output dimension of the action according to the task type; and generating the action parameter based on the intermediate action representation and the output dimension.

[0012] In some embodiments, the action parameter is generated by an action decoder; and wherein the action decoder reuses weight parameters of at least part of network layers thereof when the different task types are processed.

[0013] In some embodiments, the cross-modal shared semantic space is obtained by training using specified visual input data, specified language instruction data, and specified human action annotation data; and wherein at least part of the specified visual input data, the specified language instruction data, and the specified human action annotation data are from an Internet data source.

[0014] In a second aspect, the embodiments of the present application provide a robot action generation system, comprising: a data input module configured to obtain visual input data of a robot and corresponding language instruction data thereof; a processing module configured to obtain an intermediate action representation using a pre-constructed cross-modal shared semantic space, a correspondence between a shared semantic representation and the intermediate action representation, the visual input data, and the language instruction data; wherein the shared semantic representation is obtained based on the visual input data and / or the language instruction data and using the cross-modal shared semantic space; and a decoding module configured to generate an action parameter according to the intermediate action representation, so as to drive the robot to perform an action corresponding to the visual input data and / or the language instruction data.

[0015] In some embodiments, the data input module is further configured to obtain a task type corresponding to the visual input data and / or the language instruction data; and the decoding module is further configured to determine an output dimension of the action according to the task type, and generate the action parameter based on the intermediate action representation and the output dimension.

[0016] In a third aspect, embodiments of the present application provide a computer-readable storage medium, the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement any of the above methods.

[0017] In a fourth aspect, embodiments of the present application provide a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements any of the above methods when executing the computer program.

[0018] In a fifth aspect, embodiments of the present application provide a computer program product, the computer program product includes a computer program, and the computer program is executed by a processor to implement any of the above methods.

[0019] Embodiments of the present application provide a robot action generation method and system, medium, device and program product, visual input data of a robot and corresponding language instruction data thereof are obtained; an intermediate action representation is obtained by using a pre-constructed cross-modal shared semantic space, a corresponding relationship between shared semantic representations and intermediate action representations, the visual input data and the language instruction data; wherein the shared semantic representation is obtained based on the visual input data and / or the language instruction data and by using the cross-modal shared semantic space; action parameters are generated according to the intermediate action representation, for driving the robot to perform an action corresponding to the visual input data and / or the language instruction data. Since the cross-modal shared semantic space can enable semantic information from different modalities to be measured in a unified scale, and the above corresponding relationship can serve as an intermediate bridging representation from semantic to intermediate action representation, the understanding and execution ability of the robot for ordinary language instructions in diversified real scenes can be improved, and the accuracy and robustness of task execution are improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] The embodiments of the present application will be further illustrated below in conjunction with the accompanying drawings and specific embodiments.

[0021] Figure 1 is a flowchart of a robot action generation method provided by an embodiment of the present application.

[0022] Figure 2 is a schematic diagram of a robot action generation process provided by an embodiment of the present application.

[0023] Figure 3 This is a schematic diagram of a training process for cross-modal shared semantic space provided in an embodiment of this application.

[0024] Figure 4 This is a schematic diagram of the execution flow of an implicit action coding module and a multi-task action generation module provided in an embodiment of this application.

[0025] Figure 5 This is a schematic diagram of a robot motion generation system provided in an embodiment of this application.

[0026] Figure 6 This is a structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the embodiments of this application.

[0028] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0029] In existing robot motion generation methods, visual and linguistic features are typically regressed end-to-end to low-level robot physical quantities (such as joint angles and absolute pose). However, this approach often struggles to handle ambiguous or polysemous linguistic commands (e.g., "bring it here" should be mapped to different specific actions depending on the object and grasping scenario). Furthermore, it is highly sensitive to robot platforms and sensors, requiring extensive labeled data and rules to cover varying scenarios, which hinders model generalization to new tasks or different platforms. Additionally, the direct regression to low-level robot physical quantities is also susceptible to platform variations.

[0030] Therefore, in order to improve the problem that related methods have difficulty in accurately mapping ambiguous or polysemous language instructions to specific action parameters, this application provides a robot action generation method. Since the cross-modal shared semantic space of this application can make semantic information from different modalities measured at a unified scale, and the correspondence between the shared semantic representation and the intermediate action representation can serve as an intermediate bridge representation from semantics to intermediate action representation, the intermediate action representation obtained based on the cross-modal shared semantic space and the above correspondence obtains the semantic abstraction and composability of the action, thereby improving the robot's ability to understand and execute ordinary language instructions in diverse real-world scenarios, and improving the accuracy and robustness of task execution.

[0031] Figure 1 This is a flowchart illustrating a robot motion generation method provided in an embodiment of this application.

[0032] To address the problem of existing methods failing to accurately map ambiguous or polysemous language instructions to specific action parameters, this application proposes a robot action generation method. See also... Figure 1 The method includes steps S101 to S103.

[0033] Step S101: Obtain the robot's visual input data and its corresponding language command data.

[0034] Step S102: Obtain intermediate action representations using a pre-constructed cross-modal shared semantic space, the correspondence between shared semantic representations and intermediate action representations, the visual input data, and the language instruction data; wherein, the shared semantic representations are obtained based on the visual input data and / or the language instruction data and using the cross-modal shared semantic space;

[0035] Step S103: Generate motion parameters based on the intermediate motion representation to drive the robot to perform actions corresponding to the visual input data and / or the language instruction data.

[0036] In some embodiments, the aforementioned visual input data may refer to image or video sequences related to a scene, object, and / or operation, acquired by the robot through corresponding sensors. The aforementioned visual input data may also refer to derived modal data of the aforementioned image or video sequences. Visual input data can be used to provide at least one of the following: environmental geometry, object attributes, relative position, and dynamic information.

[0037] In some embodiments, language instruction data may refer to natural language text that is semantically associated with the aforementioned visual input data. This natural language text may include at least one of the following: target description, operational intent, step instructions, and task constraints. For example, language instruction data may be "Hand me the red cup" or "Pick up the cup and put it on the tray first." Language instruction data may also be spoken instructions or subtitles accompanying the visual input data.

[0038] Understandably, the "correspondence" in "its corresponding language instruction data" can refer to a semantic connection between visual input data and language instruction data. They can both point to the same object, action, or operational scenario, thus serving as training or inference to generate corresponding actions. For example, language instruction data can describe specific visual input data or a scene within a very short time window. For instance, language instruction data can be paired with visual input data to describe the action or task of that paragraph or scene (such as the narration paired with a scene of pouring water). Or, language instruction data can point to a specific object or target location in the visual input data (such as "put the cup on the table"). Here, visual input data and language instruction data only need to be semantically related or share the same theme; a strict one-to-one mapping is not required. For example, visual input data and language instruction data may not be original paired data, but rather "pseudo-correspondences" formed through retrieval or similarity matching to expand the training samples.

[0039] In some embodiments, feature extraction can be performed on visual input data to obtain visual feature information, and feature extraction can be performed on language instruction data to obtain language feature information. The aforementioned feature extraction can refer to the corresponding encoding of the original modal input (such as visual input data or language instruction data). For example, feature extraction can be a process of extracting low-dimensional or high-dimensional representations from raw pixels, point clouds, or text for downstream cross-modal alignment or action reasoning. This feature extraction can include basic feature extraction (such as convolutional / self-attention encoding), temporal / sequence modeling, pooling, and / or downsampling operations, etc.

[0040] In some embodiments, the visual input data and corresponding language instruction data can be preprocessed before feature extraction. Preprocessing may include unifying the coordinate system, normalization, denoising, and time normalization (such as fixed length or segmentation). Those skilled in the art can determine the specific preprocessing method according to actual needs, and no specific preprocessing method is limited here.

[0041] In some embodiments, visual feature information can refer to vectorized data used to characterize visual input data. For example, visual feature information can be an embedded vector. Visual feature information can include at least one of the following: object attributes (such as category, color, material), geometric information (such as position, depth, pose), scene layout, motion information (optical flow / trajectory), and task-related contextual information. Visual feature information can also be a convolutional feature map used to characterize visual input data. Visual feature information can also be a three-dimensional geometric representation such as a depth map / point cloud. Those skilled in the art can determine the specific form of visual feature information according to actual needs, and no specific limitations are imposed here.

[0042] Understandably, visual feature information can provide visual semantic signals for cross-modal shared semantic spaces, helping to determine the object, region, or constraint to which an action should be directed. Visual feature information can also provide physical and geometric constraints (such as object size, accessibility, and obstacles) for action generation.

[0043] In some embodiments, language feature information may refer to semantic vectors or vector sequences obtained by mapping language instruction data. Language feature information may include a sentence-level overall intent vector. Language feature information may also include fine-grained representations at the token or clause level. Language feature information may include task type embedding information. Those skilled in the art can determine the specific form of language feature information according to actual needs; no specific limitations are imposed here.

[0044] Understandably, linguistic feature information can be used to express at least one of the following information in linguistic instruction data: target, action intent, constraints, and contextual information. Linguistic feature information can also be used to align with visual feature information in a cross-modal shared semantic space to jointly constrain the generation of intermediate action representations.

[0045] In some embodiments, the aforementioned cross-modal shared semantic space can be used to uniformly represent the latent vector space of different modal information (such as visual modal information and linguistic modal information, and may also include action modal information). In the cross-modal shared semantic space, semantically corresponding visual features, linguistic features, and action representations are mapped to vector representations that are close to each other, thus providing a unified metric basis for inter-modal matching, retrieval, and conversion. The role of the cross-modal shared semantic space is to eliminate direct differences between representations of different modalities, aligning images, video frames, or scene descriptions with natural language instructions in the same semantic coordinate system, enabling the subsequent derivation of intermediate action representations based on vector similarity or learned mapping relationships. The cross-modal shared semantic space can be mapped from the original inputs using encoders of each modality, and can be trained on large-scale data using cross-modal contrastive learning to enhance semantic alignment capabilities. For example, when the visual input data is an image of a cup on a table and the language instruction data is "bring the cup over", the relevant visual encoder and language encoder map the visual input data and language instruction data to similar vectors in the shared semantic space. The distance between these vectors and the pre-learned intermediate action vectors representing specific grasping actions is relatively small, so the corresponding intermediate action representations can be retrieved or generated accordingly, thereby achieving reliable conversion and generalization from multimodal semantics to specific actions.

[0046] In some embodiments, a shared semantic representation can refer to combining visual modalities (such as visual input data) and linguistic modalities (such as linguistic instruction data) extracted by their respective encoders into a unified fused semantic vector through methods such as splicing, projection, or fusion modules. This fused semantic vector can serve as an anchor point for cross-modal alignment and retrieval. A shared semantic representation can also refer to preserving visual feature information and linguistic feature information as parallel or aligned representations, respectively. That is, a shared semantic representation can include visual feature information and / or linguistic feature information, or it can be the aforementioned fused semantic vector obtained by fusing visual feature information and linguistic feature information. The former (visual feature information) preserves spatial / low-level visual information to assist in action generation, while the latter (linguistic feature information) carries high-level instruction semantics for decision-making. The two can achieve comparability and joint use by placing them in the same embedding space (such as a cross-modal shared semantic space) or by applying alignment constraints (such as mutual similarity constraints, cross-attention, or projection alignment). In practical implementation, depending on the task requirements, fused vectors or parallel aligned vector groups can be used as anchor points for contrastive learning, or the weighted similarity between the two can be used as a matching metric. This balances semantic consistency between modalities with the preservation of unique information for each modality, thus flexibly supporting different training strategies and downstream decoding modules or models. The role of shared semantic representation is to characterize the scene and / or intent with a unified abstract semantic. For example, when a cup is visually detected and the verbal instruction is "bring the cup over," the corresponding shared semantic representation is a vector representing "the target object is the cup, and the target action is to retrieve it."

[0047] In some embodiments, shared semantic representations can be obtained by mapping visual input data and / or language instruction data to a pre-constructed cross-modal shared semantic space. For example, the cross-modal shared semantic space can be pre-constructed offline during the training phase. For instance, visual feature information, language feature information, and action feature information can be obtained based on visual input data, language instruction data, and action annotation data, using corresponding visual encoders, language encoders, and action encoders, respectively. Then, the mid-to-high-level features of the three modalities (such as visual feature information, language feature information, and action feature information) are projected onto the same dimension and normalized using their respective projection heads. The projection head can be a small neural network (such as a single-layer linear transform or a one- or two-layer multilayer perceptron) connecting the respective encoders to a cross-modal shared semantic space. The projection head maps the mid- to high-level features output by each encoder to a representation of the same dimension and normalizes them. This reduces the scale and distribution differences of features generated by different encoders, removes modality-specific irrelevant information, and improves cross-modal alignment. Furthermore, the projection head decouples the representation used for contrastive learning from the encoder's internal representation, improving training stability and generalization. Simultaneously, a vector quantization module can be used to quantize the latent representation generated by the action encoder into a set of discrete tokens. Training objectives can include contrastive loss, codebook reconstruction loss, commitment loss, and downstream action reconstruction / prediction loss. The contrastive loss is used to bring positive samples closer and push negative samples further away in the cross-modal shared semantic space; the reconstruction loss and commitment loss constrain the quantized representation. During the inference phase, visual input data and / or language instruction data can be input into the corresponding encoder and mapped to the cross-modal shared semantic space through a trained projection head to obtain the aforementioned shared semantic representation. That is, during the training phase, the corresponding encoder, projection head, action latent vector table, and decoder can be updated / trained end-to-end, and the cross-modal shared semantic space can be shaped based on loss terms such as contrastive loss. During the inference phase, the corresponding modules that have been frozen (or fine-tuned) after training can be used to directly execute the mapping, retrieval, and decoding processes.

[0048] It is understandable that shared semantic representation can refer to a representation form that enables semantic comparability and joint utilization across visual, linguistic (and action) modalities. Shared semantic representation can be obtained in several equivalent ways. One implementation is to project or fuse the encoded features of each modality into a common embedding space (such as a cross-modal shared semantic space), thereby obtaining a unified fused vector or a parallel aligned vector group. Alternatively, "sharing" between modalities can be achieved logically or functionally by imposing alignment constraints (such as similarity constraints, cross-attention, projection alignment, or weighted similarity measures) on the features of each modality, without having to physically merge all modalities into a single vector. In other words, a cross-modal shared semantic space is a typical implementation scheme for shared semantic representation, but it is not the only necessary means to obtain shared semantic representation. Any equivalent representation method that achieves semantic consistency, comparability, and joint utilization between modalities should be considered a feasible way to achieve "shared semantic representation." In some embodiments, intermediate action representation can refer to an abstract and computable potential action representation. The essence of intermediate action representation lies in serving as a bridge layer between semantic intent (such as visual and / or linguistic input) and specific executable action parameters (such as 6D grasping pose, 3D trajectory, and joint angle sequence). Intermediate action representation can be a set of latent vectors or discrete tokens that can be learned, retrieved, and combined. It can be used to encode action semantics and structural information in a denser, reusable, and composable manner. Intermediate action representation can be continuous real-valued vectors or a sequence of latent vectors composed of discrete codebook indices. It can possess decodeability (e.g., it can be mapped by a decoder to specific parameters required for the task) and composability (e.g., complex actions can be combined from several basic tokens). It can also possess platform-independent abstraction to facilitate transfer between different robots / tasks. Intermediate action representation can be obtained or matched from shared semantic representations through learned mapping functions or attention retrieval mechanisms. For example, a combination of intermediate vectors can represent the action primitive set of "grasp, lift," which can then be decoded into specific grasping postures and lift trajectories to drive robot execution.

[0049] In some embodiments, the "correspondence between shared semantic representations and intermediate action representations" can refer to a pairing relationship established between shared semantic representations and intermediate action representations through learnable mapping or retrieval mechanisms. This correspondence can be learned through projection networks, cross-attention, or nearest neighbor retrieval. This correspondence can match or map shared semantic representations to one or a set of intermediate action vectors to ensure that semantic intent can be accurately and generalizedly translated into decodeable action combinations. Furthermore, it can support precise mapping of explicit instructions (e.g., mapping "grab the cup" to specific grasping or lifting actions), and also support completion of ambiguous or vague instructions (e.g., retrieving the most suitable grasping and moving combination from semantic and environmental information for "take it here").

[0050] In some embodiments, the above correspondence can be used to generate intermediate action representations using a predefined or trained action latent vector table (codebook) or atomic action library. The action latent vector table may include several latent vectors and / or index entries, which represent high-level, composable action primitives or latent vector bases. The aforementioned action latent vector table can provide the necessary implicit action representations for generating intermediate action representations, replacing the method of directly regressing to obtain action parameters.

[0051] Understandably, the "correspondence between shared semantic representations and intermediate action representations" can be a mapping and similarity constraint obtained through learning or training. It describes how visual-language anchors (such as shared semantic representations) map to intermediate action representations that can be decoded into specific actions. The action latent vector table (codebook) is the basic element or "vocabulary" that constitutes these intermediate action representations. That is, the action latent vector table itself is a static or learnable set of vectors (similar to a vocabulary), where each entry represents a latent vector of an atomic or primitive action. The correspondence is a dynamic mapping mechanism or function determined by model parameters. The correspondence can map shared semantic representations to combinations or weighted sums of one or more entries in the action latent vector table through attention mechanisms, similarity matching, or combined weights to obtain intermediate action representations. The aforementioned correspondence uses the action latent vector table (codebook) as the underlying representation space and is trained using contrastive learning and reconstruction / quantization constraints to ensure that intermediate action representations can be reliably decoded, thereby achieving interpretable, composable, and generalizable conversion from semantics to actions. In short, the action latent vector table provides representation atoms, while the correspondence defines how to use these atoms to characterize intermediate action representations.

[0052] In some embodiments, the correspondence between the shared semantic representation and the intermediate action representation can be obtained through joint training. For example, the high-dimensional features of the relevant visual encoder, language encoder, and action encoder can be extracted first, and the action features can be discretized into several tokens in the action latent vector table through vector quantization, serving as the underlying space of the intermediate action representation. Then, using the shared semantic representation as the anchor, the latent vector obtained from the action encoder after quantization as the positive sample, and the erroneous action latent vectors obtained in the same batch or through online mining as the negative sample, a triplet loss or similar method is used to force the anchor to be closer to the correct action latent vector in the cross-modal shared semantic space, while keeping it away from the negative sample. During the training phase, the reconstruction loss of the action latent vector table can be optimized in parallel to ensure that the latent vector can effectively represent the action details. Through this joint contrastive learning and quantization reconstruction multi-task loss, the model can learn to map visual / language intent to composable intermediate action tokens in the same semantic space (such as the cross-modal shared semantic space), providing a reliable mapping relationship for the subsequent attention retrieval and decoding module based on the action query vector to generate specific actions.

[0053] In some embodiments, the aforementioned correspondence can be used to abstract specific action values ​​(such as joint angles and trajectory coordinates) and other action feature information into semantic action primitives or latent vector bases. These action primitives or latent vector bases can serve as intermediate representations for combination and / or retrieval. This correspondence can reduce direct dependence on specific robot sensors / physical quantities, transforming action feature information into semantic representations to facilitate cross-platform transfer and few-sample fine-tuning. Several action primitives or latent vector bases can be combined with different weights or orders to generate composite actions, improving the completion and generalization capabilities of fuzzy instructions. This correspondence can be used to force the learning of repeatable and interpretable action "atoms," reducing redundancy and improving learning efficiency. In other words, the aforementioned action latent vector table can be a composable and searchable semantic "action vocabulary" compressed or abstracted from specific actions, which can be used to bridge vision and language understanding with the generation of specific robot actions, improving the completion and cross-domain generalization capabilities of fuzzy instructions. This correspondence can serve as an intermediate layer between the visual / language modality and the relevant action decoding module, making it easier to align and exchange the representations at both ends.

[0054] In some embodiments, the aforementioned action latent vector table may include a pre-set or learned set of continuously differentiable vectors. That is, the aforementioned action latent vector table can be viewed as a set of differentiable, continuous embedding vectors E = {e...} kHowever, no hard quantization is performed. For a given visual / language query vector q or action query vector q, weights are obtained by calculating similarity, etc., and then a weighted sum or linear combination is used as an intermediate action representation. The intermediate action representation is then input into the relevant decoding module or model for training.

[0055] In some embodiments, several primitive combinations can be retrieved using the aforementioned correspondence to obtain a coarse action representation. Then, a refined residual is predicted by continuous branches and added to the retrieval vector, thereby compensating for quantization errors and improving accuracy. During the training of the action latent vector table, reconstruction loss, vector quantization commitment loss, and residual regression loss can be used. In the inference phase, a fast retrieval can be performed first, followed by the addition of the residual to meet fine-grained control requirements.

[0056] In some embodiments, the aforementioned action latent vector table can be divided according to semantic or temporal hierarchy. The high-level part of the aforementioned action latent vector table can represent abstract semantic units (such as "pick up," "move to," and "put down"), while the low-level part of the aforementioned action latent vector table can represent fine-grained control blocks or short-term trajectory fragments (such as local grasping postures and fine-tuning trajectories). The relevant decoder or module can first provide the high-level content in a hierarchical manner, and then gradually expand to the low-level action sequences. During the training process of the action latent vector table, a staged training method can be adopted, such as learning the high-level first and then learning the low-level.

[0057] In some embodiments, the aforementioned action latent vector table can be designed as a queryable key-value store. A cross-attention mechanism allows the visual / language query vector to dynamically retrieve the most relevant action primitives or latent vector bases for the current context, thereby achieving context-based primitive selection. For example, the same instruction may retrieve different primitive combinations in different scenarios. In other words, the aforementioned action latent vector table can be embedded as a set of key-value pairs into structures such as encoder and decoder networks based on attention mechanisms. The visual / language query vector directly performs attention matching on the aforementioned action latent vector table, outputting multiple dynamically weighted action primitives or latent vector bases for use by downstream decoding modules or models. The aforementioned action latent vector table can serve as an intermediate representation source before the action query vector, or it can be inserted into different layers of the network model as an additional key-value source within the existing cross-attention mechanism. During the training of the aforementioned action latent vector table, Top-k sparse retrieval can be used to select the top k most relevant dictionary items, and learnable key-value mappings can be introduced to improve matching accuracy.

[0058] In some embodiments, the aforementioned action latent vector table can be obtained through the following construction or training methods. First, starting with multi-source data acquisition and preprocessing, the visual language action dataset (such as videos containing human operations and their time-aligned text descriptions and human / hand keypoint trajectories) and a small amount of robot action data undergo temporal segmentation and normalization preprocessing. Then, a temporally-aware action encoder maps each action feature information into a continuous latent vector representation. Based on this continuous latent vector representation, the aforementioned action latent vector table can be constructed by combining quantization mechanisms such as vector quantization and variational autoencoder (VAE). The initial action latent vector table is a random or cluster-based set of prototype vectors. During the training phase, a reconstruction loss can be used to enable the relevant decoding modules or models to reconstruct the original action sequence from the indices of the aforementioned action latent vector table. Furthermore, a commitment loss can be introduced to constrain the action encoder output to approximate the action primitives or latent vector basis in the aforementioned action latent vector table. The action primitives or latent vector basis in the aforementioned action latent vector table can also be updated using methods such as exponential moving average (EMA) to maintain stable training. To achieve cross-modal alignment, a contrastive learning loss can be incorporated into the visual and linguistic feature learning during the joint training phase. This reduces the distance between correct visual and linguistic feature information and their corresponding action codebook embeddings, while increasing the distance between negative sample pairs. This allows the entries in the action latent vector table to carry high-level semantic action primitives. The contrastive learning loss can be either InfoNCE (Information-Noise Contrast Estimation) or triplet loss. For higher execution accuracy, a hybrid structure can be used, such as providing high-level semantic primitives with a discrete codebook followed by continuous residual branches, or refining the codebook using a hierarchical / productive quantization structure, to achieve a trade-off between generalization ability and continuous control accuracy. During the training phase, the aforementioned action latent vector table can be pre-trained on large-scale internet data to learn general action primitives. Then, it can be fine-tuned with a small amount of robot action data, and the contrastive loss, reconstruction loss, and codebook constraints can be jointly optimized to obtain an action latent vector table that supports cross-modal alignment and can be retrieved and combined by the corresponding decoder. Those skilled in the art can flexibly adjust the specific construction or training method of the above-mentioned action latent vector table according to the actual needs of the scenario, and no specific restrictions are imposed here.

[0059] In some embodiments, visual input data and language instruction data can first be input into corresponding feature extractors to obtain a set of visual feature information and a set of language feature information, respectively. Then, these visual and language feature information are mapped to a pre-constructed cross-modal shared semantic space and fused within that space to generate a shared semantic representation for subsequent action generation. The specific generation of action parameters can be accomplished through a unified multi-task decoder.

[0060] The following are the different ways of using the above correspondence in the reasoning stage and the corresponding explanations in the training stage.

[0061] For example, the aforementioned action latent vector table can be used for retrieval / combination before generating the action query vector. Specifically, a shared semantic representation can be obtained first, and the action encoder generates an initial action query intent based on at least one of the following: task type, prior information, and language instruction semantics (such as language feature information). This initial action query intent can be regarded as an "intent vector" or a set of intent vectors to describe the type of action or high-level goal to be performed, but it has not yet been directly mapped to a specific physical control quantity. Before generating the final action query to be sent to the relevant decoding module or model, the aforementioned initial intent vector can be matched with the aforementioned action latent vector table to retrieve or combine one or more latent vector bases as intermediate representations. Specifically, the matching can be performed in any of the following ways. For example, one way is to calculate the similarity between the intent vector and each entry in the aforementioned action latent vector table, and then weight and sum the codebook entries according to the similarity to obtain a continuous combined vector. Another way is to select several entries in the aforementioned action latent vector table that are most similar to the intent, and directly concatenate or fuse the vectors of these entries in a certain order or through a small combination network into a fixed-dimensional combined vector. Another approach is to first perform nearest neighbor retrieval to obtain a set of discrete code indices, and then feed the vector sequences corresponding to these indices into the subsequent synthesis module, which outputs the final combined vector. Regardless of the specific combination mechanism used, the goal is to map the original initial intent to a high-level combination of action atoms based on the aforementioned latent action vector table, ensuring that the action query carries composable action atom semantics, rather than direct joint angles or absolute position values. After retrieval / combination, the combined vector can either completely replace the original initial intent vector, or the combined vector can be residually / concatenated with the original intent vector to obtain the final action query vector. Subsequently, the relevant decoding module or model receives the action query vector generated or enhanced by the aforementioned correspondence and combines it with visual / linguistic semantic representations to generate a specific action output, such as generating a 6D (6-dimensional) grasping posture or a 3D (3D) trajectory. During the inference phase, this process can proceed in the order of generating intent, retrieving / combining the latent action vector table, and then decoding using intermediate action representations as queries. During the training phase, the aforementioned correspondence needs to learn semantically represented action atoms. For example, the aforementioned latent action vector table can be pre-trained using a large amount of human operation data or synthetic action data (e.g., learning the entries of the latent action vector table through encoding, discretization, and reconstruction). Then, joint training steps such as cross-modal contrastive learning and behavior cloning can be used to correctly map the initial intent vector to the appropriate codebook entries, enabling the relevant decoding modules or models to generate the correct physical actions from these entries. This method of using latent action vector tables can abstract action representations into composable atomic units, thereby better completing ambiguous instructions.

[0062] For example, the aforementioned action latent vector table can be used as an additional information source during the cross-attention process of the relevant decoding module or model. Specifically, when performing cross-attention-based fusion of visual and linguistic representations in a cross-modal shared semantic space, the relevant decoding module or model does not only use visual / linguistic tokens as the key-value information source for attention, but can simultaneously introduce the entries of the aforementioned action latent vector table as another information source. This allows the relevant decoding module or model to dynamically reference or read the entries of the aforementioned action latent vector table during the attention fusion process of each or several layers. For instance, when the decoding module or model performs attention calculations in a specified network layer, it can treat the shared semantic representation and the entries of the aforementioned action latent vector table together as readable memory items. The relevant decoding module or model can simultaneously initiate attention to both the visual / linguistic memory and the memory of the aforementioned action latent vector table based on the intermediate representation of the current layer, extracting useful information from both sides and merging them into the output representation of that layer. In this way, while fusing external semantic information, the relevant decoding module or model can dynamically extract the action atoms most relevant to the current context from the action latent vector table, or combine multiple codebook entries to form more complex intermediate representations. The following explains how entries from the aforementioned action latent vector table are combined into intermediate action representations. One approach is to have the relevant decoding module or model calculate attention weights for the entries in the action latent vector table and use these weights to perform a weighted summation of the entry vectors, thus obtaining a dynamic combined vector at each layer and time step. A two-step attention strategy can also be used: first, candidate focal points are obtained from the visual / linguistic representation, and then the focal point is used to retrieve and combine entries from the codebook; or conversely, the codebook is retrieved first, and then the visual / linguistic representation is finely normalized. During the inference phase, the relevant decoding module or model can rely on the aforementioned action latent vector table as a memory source at multiple layers, gradually injecting action atoms into the output during semantic fusion when generating actions. During the training phase, the model needs to learn when, at which layer, and with what intensity to refer to the entries in the aforementioned action latent vector table. Therefore, supervision or regularization terms are often added to encourage the reasonable use of the aforementioned action latent vector table, such as explicitly guiding attention to focus more on specific entries for the correct actions corresponding to the training samples, or using contrastive loss to make the visual / linguistic representation closer to the correct entries in the latent space.

[0063] For example, the combined vectors from the aforementioned action latent vector table can be used as conditional inputs before the relevant decoding module or model begins decoding. Specifically, a retrieval and combination can be performed on the aforementioned action latent vector table based on the task input and shared semantic representation to obtain one or more representative combined vectors. This combined vector is then provided as conditional input to the relevant decoding module or model before it formally begins generating actions. For instance, shared semantic representations can be extracted and extracted from vision and language. Then, the action encoder or a dedicated retrieval module uses this shared semantic representation as a query to search the aforementioned action latent vector table for the most relevant entries to the task. The found entries are fused into a fixed-dimensional conditional vector according to a predetermined combination strategy. The fusion method can be to sum several entries by weight to obtain a continuous vector, or to concatenate several entry vectors and map them into a fixed-dimensional conditional vector through a small network, or to generate a conditional vector from discrete code index sequences through embedding and aggregation layers. After the conditional vector is formed, it is input to the relevant decoding module or model along with the shared semantic representation. For example, one approach is to concatenate the conditional vector with the initial state or input tokens of the relevant decoding module or model, so that the relevant decoding module or model uses the condition as static background information throughout the decoding process. Another approach is to inject the conditional vector into each layer of the relevant decoding module or model, continuously influencing the output of the relevant decoding module or model through modulation layers (such as element-wise scaling and translation), gating, or as additional input tokens. During the inference phase, the general order can be: semantics, codebook retrieval, obtaining the combined vector, passing the combined vector as a condition to the decoder, and the decoder generating actions based on the condition and semantics. During the training phase, the generation process of the combined vector obtained based on the aforementioned action latent vector table can be pre-trained as an independent training module, or it can be jointly trained with the decoder to better adapt the combined vector to the downstream action generation task.

[0064] It is understood that the different uses of the aforementioned implicit action vector tables are not mutually exclusive; that is, they can be used in parallel or in series. The different uses of the aforementioned implicit action vector tables are merely illustrative examples, and those skilled in the art can flexibly adjust the use of the aforementioned implicit action vector tables according to actual needs.

[0065] It is understandable that the combined vector obtained based on the entries in the aforementioned action latent vector table is essentially a vector for decoding or as a query, mapped from several entries in the action latent vector table through a specified learnable or predefined combination mechanism. This combined vector can be generated at least at the following three different time points. The first is after the action encoder generates the initial intent but before it is fed into the relevant decoding module or model. The second is dynamically calculated based on the intermediate representation of each layer or several layers during multi-layer cross-attention matching in the relevant decoding module or model. The third is that it can be generated as a global condition before decoding begins, or dynamically retrieved and updated multiple times during the decoding process. The specific combination mechanism can be implemented in the following ways: One is soft attention weighted summation, which calculates similarity scores for the entries in the aforementioned action latent vector table using the query vector, and uses these scores to weight the entry vectors to obtain a continuous combined vector. Another is hard retrieval plus embedding, which finds the most similar entries and directly concatenates their vector sequences or feeds them into a small synthetic network to output the final fixed-length vector. Another approach is the discrete index sequence form, where a series of retrieved discrete indices are input into an embedding layer and a pooling layer for processing, resulting in a combined vector. The specific method chosen depends on the trade-offs between interpretability, compression, real-time performance, and training difficulty. Regardless of the method, the combined vector can be fused with the original semantic / intent vector through direct replacement, residual fusion, or parallel concatenation before being passed to the relevant decoding module or model.

[0066] In some embodiments, the aforementioned action latent vector table can be used in the fusion process of visual and linguistic feature information. Specifically, the action latent vector table can be used to intervene in the semantic representation process of vision and language at an earlier fusion stage. The goal is to ensure that cross-modal representations are based on or referenced by action atoms from the generation stage, thereby enhancing semantic consistency and interpretability. Specific implementation methods can include at least the following: One implementation method is to first project onto the basis of the aforementioned action latent vector table before fusion. For example, visual features and linguistic features first perform matching or attention queries on the entries of the aforementioned action latent vector table, obtaining responses (such as a set of weights or entry indexes) for each modality to the entries of the aforementioned action latent vector table. Then, the responses of each modality to the aforementioned action latent vector table are aggregated, thereby directly obtaining a shared semantic representation on the atomic set of the aforementioned action latent vector table. This implementation method allows different modalities to be aligned on the same atomic set, facilitating subsequent action mapping. Another implementation method is to fuse first and then quantize to the aforementioned action latent vector table. For example, vision and language are first fused interactively in a conventional manner to obtain continuous shared semantic representations. Subsequently, the shared semantic representation is quantized onto the entries or distribution of the aforementioned action latent vector table to obtain discrete or semi-discrete shared semantic representations. This implementation preserves fine-grained interaction information between the two modalities while grounding the final semantics in composable action atoms, thus improving interpretability and composability during the generation stage. One implementation involves using entries from the aforementioned action latent vector table as additional tokens in the fusion process. For example, several tokens can be directly added to the fusion network, allowing visual and linguistic tokens to perform attention calculations together with tokens from the aforementioned action latent vector table within the same fusion layer. This embeds the information from the aforementioned action latent vector table into the cross-modal representation generation process at multiple levels.

[0067] It is understandable that using the aforementioned action latent vector table in the process of fusion of visual and linguistic feature information can align cross-modal semantics with action atoms from the beginning of generation, improve the quality of retrieval alignment, enhance the completion capability of fuzzy instructions, and reduce inter-domain differences during subsequent decoding.

[0068] The following are descriptions of the training phase adapted to the inference phase described above. A unified pre-training and fine-tuning approach can be adopted, such as a two- or three-stage training process. In the first stage, cross-modal pre-training can be performed on large-scale internet video and text data to update the cross-modal shared semantic space and preliminary action semantics. In the second stage, the aforementioned correspondences can be pre-trained separately or fine-tuned jointly with the main network, making the entries in the aforementioned action latent vector table composable action atoms. In the third stage, domain-adaptive fine-tuning can be performed on small-scale robot data, calibrating the output to real robot physical quantities (such as joint angles and end-effector poses) and further optimizing based on execution feedback.

[0069] As can be seen, the action primitives or latent vectors in the aforementioned action latent vector table in this application embodiment can serve as intermediate representations for combination and / or retrieval. The aforementioned action latent vector table can be used to remove direct dependence on specific robot sensors / physical quantities and transform them into semantic-level representations, facilitating cross-platform transfer and few-sample fine-tuning. Several action primitives or latent vectors can be combined with different weights or orders to generate composite actions, improving the completion and generalization capabilities for fuzzy language instructions. It can be used to force the learning of repeatable and interpretable action "atoms," reducing redundancy and improving learning efficiency. It facilitates the retrieval or matching of the most relevant latent vectors using visual feature information and linguistic feature information, or a fusion of both, in a cross-modal shared semantic space, achieving cross-modal alignment. Since different modalities can be mapped to a cross-modal shared semantic space, which allows semantically similar visual and linguistic combinations to be close to corresponding actions (such as the most relevant action latent vectors), it facilitates obtaining intermediate action representations through similarity retrieval or regression. The aforementioned implicit action vector table can provide discrete or continuous, composable action primitives, allowing high-level fuzzy instructions to be mapped to combinations of action primitives. Intermediate action representations can be reduced to specific action parameters, thereby achieving a robust bridge and transfer generalization from semantics to physical actions. In other words, the embodiments of this application can improve the interpretation capability of fuzzy language instructions through implicit action representation, and also enhance cross-task generalization and the consistency of parameter output.

[0070] To address the alignment and generalization difficulties arising from the diversity of fuzzy language instructions and action expressions in related methods, in some embodiments, the process of obtaining the intermediate action representation may include: using the cross-modal shared semantic space and the correspondence to convert the shared semantic representation into a continuous latent vector and / or a discrete latent vector of actions; and using the continuous latent vector and / or the discrete latent vector of actions as the intermediate action representation.

[0071] In some embodiments, continuous latent vectors (LCVs) can refer to a low-dimensional, continuous vector representation. LCVs can be used to encode specific action elements in a compact and smooth manner at the semantic level. LCVs can be viewed as the result of mapping visual and linguistic contextual information into a continuous latent space. LCVs can refer to the continuous latent vectors output by the corresponding action encoder, i.e., unquantized real-valued vectors.

[0072] Understandably, continuous latent vectors (LCVs) can compress complex visual and linguistic features into fixed-dimensional vectors, facilitating storage, retrieval, and subsequent processing. Actions with similar semantics are close in the latent space, making it easier to interpolate / complete fuzzy instructions and achieve smooth transitions. LCVs can be mapped to the physical control quantities of different robot platforms using different decoding modules or models. LCVs can be composed of linear or nonlinear combinations of the aforementioned correspondences of action primitives or latent vector bases. In other words, LCVs do not have to be a single, indivisible vector; they can be viewed as a combination of several latent vector bases according to specified weights or combination rules, thereby achieving the completion and generalization of fuzzy instructions. The specified weights or combination rules can be learned during the training process of the aforementioned correspondences.

[0073] It is understandable that continuous latent vectors of actions can exist as continuous real-valued vectors or weighted combinations of several continuous primitives. Discrete latent vectors of actions can exist as several action primitives or latent vector bases in the above correspondence. Continuous latent vectors of actions can synthesize intermediate states through interpolation / recombination of primitive weights, which can perform smooth completion or interpolation-based generalization for fuzzy instructions. Discrete latent vectors of actions can form new actions through codeword combinations, favoring combinatorial generalization, in order to concatenate / serialize atomic codewords to express complex instructions.

[0074] In some embodiments, the fusion result of visual feature information and linguistic feature information can refer to a joint representation obtained by fusing visual feature information and linguistic feature information in a cross-modal shared semantic space. The joint representation can be a vector or a vector sequence. The above fusion can employ cross-attention mechanisms, pooling fusion, concatenation, etc. Those skilled in the art can determine the specific fusion method according to the actual scenario requirements, and no specific limitations are imposed here. The above fusion result preserves the joint information of visual context and linguistic intent, and can serve as a direct basis for generating intermediate action representations.

[0075] For example, visual feature information v can be mapped to a cross-modal shared semantic space and interacted with an action latent vector table to obtain a visually induced latent vector zv, such as retrieving a weighted vector from the codebook through attention matching. Similarly, linguistic feature information l can be mapped to the cross-modal shared semantic space and interacted with the aforementioned action latent vector table to obtain a linguistically induced latent vector zl. Then, latent vectors zv and zl are synthesized into a continuous action latent vector z1 according to specified rules (such as concatenation, weighted averaging, or cross-attention). The continuous action latent vector z1 can be directly used as an intermediate action representation. Alternatively, visual and linguistic feature information can be fused in the cross-modal shared semantic space (e.g., using cross-attention or weighted combination) into a joint vector f. The joint vector f can then interact with the aforementioned action latent vector table to obtain a continuous action latent vector z2. For example, codebook vectors can be retrieved / weighted from a dictionary based on an attention mechanism, or a pre-trained mapping network can be used to map f into a continuous action latent vector. Alternatively, visual feature information v and linguistic feature information l can be fused outside the cross-modal shared semantic space to obtain a fusion result. Subsequently, the fusion result is interacted with the aforementioned action latent vector table within the cross-modal shared semantic space to obtain continuous action latent vectors.

[0076] Understandably, when considering obtaining continuous latent vectors of actions using cross-modal shared semantic space and correspondences as the first approach, and obtaining discrete latent vectors of actions using the same approach as the second approach, the continuous latent vectors in the first approach can be directly generated by a dedicated mapping network (e.g., the network output is a continuous intermediate action representation), or obtained by differentiable weighted combination of several latent vectors in the action latent vector table (e.g., soft assignment, weighted summation). In contrast, the discrete latent vectors in the second approach primarily rely on discrete tokens from the action latent vector table or quantized latent vector indices; that is, they use combinations of latent vectors obtained through hard quantization / retrieval as intermediate action representations. The essential meaning of the action latent vector tables in both approaches is essentially the same: both serve as lexical or primitive representation dictionaries of action semantics, but their functional emphasis differs. For example, the first approach emphasizes using continuous mapping or soft combination to obtain a more refined, interpolable latent vector space, while the second approach emphasizes using combinations of discrete tokens to achieve dense and reusable action primitive construction. There are also some differences during the training phase. For example, if soft combination or continuous mapping is used, the mapping network is typically trained with end-to-end regression / contrast loss and attention weights can be optimized in parallel. If an action latent vector table is used, the relevant quantization module needs to be trained simultaneously, and methods such as pass-through estimation are used to achieve differentiable training of discrete choices. There are also some differences in the inference stage. For example, continuous mapping / soft combination can directly output continuous action latent vectors and pass them to the relevant decoding module, while using an action latent vector table, the corresponding token can be retrieved through nearest neighbor lookup or discrete index and then decoded into an action, or, when necessary, discrete choices can be replaced with soft approximation to obtain a smooth output. Overall, these two methods can complement each other and coexist in the same system. The above correspondence can be obtained by joint learning of cross-modal contrastive learning and reconstruction / quantization loss. The training stage determines which representation and quantization mechanism to use, and the inference stage generates the final intermediate action representation according to the mapping or retrieval method determined in the training stage and provides it to the downstream decoding module.

[0077] As can be seen, by utilizing the aforementioned cross-modal shared semantic space, this application embodiment can ensure that visual feature information and linguistic feature information are comparable and / or fused in the same coordinate system. Furthermore, it provides composable action primitives or latent vector bases, enabling complex actions to be reconstructed from these primitives. Continuous latent vectors of actions can achieve semantic continuity and interpolation through weighted combination or mapping, thereby improving generalization and the ability to complete fuzzy instructions. In other words, since the obtained continuous latent vectors of actions can be smooth, low-dimensional, and semantically meaningful action representations, it is convenient to subsequently generate actions of different dimensions / types, improving the generalization ability for unseen scenes or fuzzy instructions. Through the prior constraints of the cross-modal shared semantic space and the aforementioned correspondence, the dependence on a large amount of robot physical data can be reduced, enhancing cross-domain transfer and combination capabilities, thereby improving the robustness and reusability of action generation. Semantically consistent and compact action embeddings (such as continuous latent vectors of actions) can be obtained, improving the alignment accuracy from vision and language to action and cross-domain generalization ability, while reducing the dependence on a large amount of robot annotation data. For ambiguous or multi-solution instructions, reasonable action candidates can be generated through the combination / interpolation of latent vectors, which can facilitate the further generation of action parameters for relevant decoding modules or models for various task types.

[0078] To improve the compressibility, reusability, and noise resistance of action representations in related methods, in some embodiments, the process of obtaining the intermediate action representation may include: using the cross-modal shared semantic space to convert the shared semantic representation into a continuous latent vector of actions; determining the corresponding discrete latent vector of actions based on the continuous latent vector of actions and the correspondence, and using the discrete latent vector of actions as the intermediate action representation.

[0079] Understandably, utilizing a cross-modal shared semantic space allows shared semantic representations (such as visual and linguistic feature information, or a fusion of both) to be closer to correct action representations (such as continuous latent vectors) in the same space, and further away from incorrect action representations. This enables the establishment of comparability between visual perception, linguistic commands, and actions at the semantic level, fundamentally improving the accuracy and discriminative power of semantic matching. Action representations (such as continuous latent vectors) can carry complex and subtle action information (such as speed, curve shape, and minute posture adjustments). Therefore, when mapping from visual and linguistic feature information to actions, they can retain necessary continuous details, facilitating subsequent accurate reconstruction of action parameters. Here, the role of action representations (such as continuous latent vectors) is to carry high-level semantics of visual and linguistic information, expressing the fuzzy semantic intent of "what should be done," facilitating subsequent mapping to more discrete or executable action primitives or latent vector bases.

[0080] In some embodiments, the discrete latent vector of an action can refer to a code vector or a vector corresponding to a code index selected or combined from the aforementioned action latent vector table. The function of "determining the corresponding discrete latent vector of an action based on the continuous latent vector of the action and the corresponding relationship" is to map the continuous semantic representation onto a discrete, composable set of "action atoms" as an intermediate action representation. The determined discrete latent vector of an action can be used for more stable retrieval, decoding, combination, and physical mapping, thereby improving generalization and interpretability.

[0081] In some embodiments, the discrete latent vector of an action may include one or more action primitives or latent vector bases in the action latent vector table described above. Each action primitive or latent vector base represents an atomic action or action primitive or a subcomponent of an action primitive.

[0082] Understandably, based on discrete latent vectors of actions, action primitives or latent vector bases can be combined as "vocabularies" (similar to word sequences in language models) to generate action combinations not present in the training set, thereby improving generalization ability. Specific actions can be generated through a limited combination of action primitives or latent vector bases, which is easier to train and more robust than directly regressing high-dimensional sensor values ​​(such as joint angles). When the semantics of language instructions are ambiguous, the most suitable action primitives or latent vector bases can be retrieved or combined from the aforementioned action latent vector table based on continuous latent vectors of actions to serve as the corresponding discrete latent vectors of actions. Continuous latent vectors of actions can serve as a unified intermediate representation, facilitating inter-modal similarity comparison and retrieval within a shared semantic space across modalities.

[0083] In some embodiments, when the intermediate action is represented as a continuous latent vector, action parameters can be directly generated from the continuous latent vector. Specifically, during the training phase, a corresponding related decoding module or model can be used with paired target action data (such as pose, joint angle sequences, or trajectory points) as the supervision target, and the mapping from the continuous latent vector to specific action parameters can be learned by minimizing the reconstruction loss. For example, the related decoding module or model can use a multilayer perceptron for 6D pose regression. During the inference phase, the related decoding module or model uses the continuous latent vector as input to generate action parameters. If the related decoding module or model is a deterministic model, the action parameters can be directly output. If the decoding module is a generative model, multiple candidate parameters can be sampled and the optimal solution can be selected through physical constraints. Afterwards, the output action parameters are post-processed to ensure executability and safety, and finally the verified action parameters are sent to the corresponding modules of the robot.

[0084] In some embodiments, when the intermediate action is represented as a discrete latent vector, the action parameters can be directly generated using the discrete latent vector. Specifically, during the training phase, training samples with action annotations can be input into the action encoder to obtain a continuous latent vector. Then, this continuous latent vector is vector-quantized and mapped to the index of the nearest discrete latent vector in the initial action latent vector table, thus obtaining a discrete latent vector, which serves as the intermediate action representation. The discrete latent vector can be used as input to the relevant decoding module or model for reconstructing or generating the corresponding action and calculating the reconstruction loss. It can also be used as a target for cross-modal alignment, used for comparative learning with the visual and linguistic modalities in a shared semantic space across modalities, to constrain the correct visual and linguistic combination to be close to the corresponding discrete action representation.

[0085] As can be seen, the embodiments of this application ensure the semantic consistency of the determined continuous latent vectors of actions by sharing a semantic space across modalities. The aforementioned correspondence, through discretization, can cluster the continuous space into several action primitives or latent vector bases. The quantization process suppresses small perturbations and overfitting, and forces the learning of composable action primitives or latent vector bases. Combined with training mechanisms such as commitment loss, the aforementioned correspondence can efficiently capture common action patterns, thus achieving robust and generalizable discrete action representations. In other words, a set of discrete, semantically well-clustered, and reusable action atoms can be obtained, thereby improving the robustness and compression efficiency of intermediate action representations. It can also enhance the generalization ability to fuzzy instructions and cross-domain data, while facilitating the combination of complex actions and reducing dependence on large amounts of robot labeled data. It can improve the reusability and noise resistance of action representations, which is beneficial for generalization and few-sample adaptation during subsequent decoding.

[0086] To improve the problem of poor consistency between the generated results and linguistic feature information or multimodal fusion context in related methods, in some embodiments, generating action parameters based on the intermediate action representation may include: generating the action parameters using the action discrete latent vector and the shared semantic representation.

[0087] In some embodiments, discrete latent vectors of action can be combined with linguistic feature information. For example, the discrete latent vectors can be used as query vectors and linguistic features as keys for cross-attention matching, or the two can be concatenated and processed through several fully connected layers or corresponding decoding modules or models, with the corresponding decoding modules or models outputting action parameters in the corresponding format. Alternatively, the discrete latent vectors of action can be combined with the fusion results of the aforementioned visual and linguistic feature information through similar attention interaction or concatenation processing, and then decoded into action parameters by the corresponding decoding modules or models. When receiving the combined representation, the corresponding decoding modules or models can also dynamically select or activate the corresponding output heads based on the task type to generate action outputs of different dimensions / formats (such as 6D grasping poses, 3D trajectory point sequences, or discrete action sequences). When necessary, post-processing steps such as coordinate transformation or inverse kinematics can be used to map the data into joint commands executable by the robot. During training, common constraints such as reconstruction loss, behavior cloning loss, contrastive learning loss, and commitment loss of the aforementioned action latent vector table can be used to ensure that the action parameters jointly generated by the discrete latent vectors of action and linguistic feature information or fusion results accurately reflect the semantics of the instructions and possess executability and generalization ability.

[0088] As can be seen, this embodiment directly generates target action parameters by using discrete latent vectors of actions as the primary semantic input and combining linguistic feature information or the aforementioned fusion results as conditional context (such as through concatenation, attention, or cross-attention mechanisms). When generating action parameters, task conditions or different output heads can be used to adapt to different parameter formats. Since discrete latent vectors of actions can provide robust semantic prototypes, and linguistic feature information or the aforementioned fusion results can provide contextual constraints for the decoding process, the two can achieve a conditional mapping from semantics to action parameters through the combined use of attention or conditional generation mechanisms. Furthermore, the quantized latent vectors can suppress noise and strengthen composable action primitives or latent vector bases, thereby learning more robust and generalizable parameter generation mappings. In other words, this embodiment can map high-level semantic action atoms into precise and instruction-related physical parameters, reducing the uncertainty of ambiguous language instructions, improving the accuracy and task adaptability of downstream execution, and maintaining the model's generalization ability for multi-task and cross-domain scenarios.

[0089] In some embodiments, the correspondence is based on the aforementioned action latent vector table as the underlying representation space. The action latent vector table may contain a first number of specified discrete latent vectors, wherein the action discrete latent vectors are obtained by attention matching on a second number of the specified discrete latent vectors, the second number being less than the first number. Those skilled in the art can adjust the specific content contained in the action latent vector table according to the actual needs of the scenario, and no specific limitations are imposed here.

[0090] It can be understood that "the correspondence is based on the aforementioned latent action vector table as the underlying representation space" means mapping and aligning visual and linguistic representations to the vector space defined by the latent action vector table. The matching relationships between visual / linguistic modalities and action modalities are all established and measured on the underlying representation determined by the latent action vector table. In other words, the semantic primitives of intermediate action representations can be provided by the latent action vector table. The correspondence and combination of each modality can use the latent action vector table as a common semantic basis, thus facilitating the generation or reconstruction of specific actions by retrieving or weighting latent vectors.

[0091] In some embodiments, the specified discrete latent vector can be obtained by quantizing the continuous latent vector of the specified action. This quantization can be nearest neighbor quantization, hard assignment, or soft assignment (such as attention weighting or probability assignment). The specified discrete latent vector can also be obtained through other discretization methods, such as hard quantization after weighted combination based on soft attention, product quantization, residual quantization, etc. Those skilled in the art can determine the first quantity, the second quantity, and the specific method of specifying the discrete latent vector according to the actual scenario requirements; no specific limitations are imposed here.

[0092] In some embodiments, the pre-acquired continuous latent vectors of actions can be first quantized to obtain a first number of specified discrete latent vectors. Then, when generating specific actions, a single specified discrete latent vector can be selected, or attention matching can be used to focus only on a smaller subset of entries (such as a second number of specified discrete latent vectors) from the aforementioned action latent vector table. These attention-weighted or combined entries are then used to construct the final action discrete latent vector. In other words, the action discrete latent vector can be generated using a method of "quantization and subset attention retrieval / combination".

[0093] In some embodiments, continuous latent vectors of actions can refer to representations in the form of real-valued vectors or vector sequences. Continuous latent vectors of actions can be directly regressed to continuous control variables (such as joint angles, end-effector poses, trajectory points, etc.). Discrete latent vectors of actions can refer to representations that map actions to discrete codebook indices or combinations of several codebook entries. That is, actions are viewed as composable sequences of semantic primitives. Continuous latent vectors of actions can include one or a second number of continuous latent vectors. Continuous latent vectors of actions can be obtained by combining a second number of continuous latent vectors. Accordingly, the aforementioned action latent vector table can contain a first number of continuous latent vectors, with the second number being less than the first number. Discrete latent vectors of actions can include one or a second number of discrete latent vectors. Discrete latent vectors of actions can be obtained by combining a second number of discrete latent vectors. Accordingly, the aforementioned action latent vector table can contain a first number of discrete latent vectors, with the second number being less than the first number. For example, the first number can be 256, and the second number can be 4. For example, the first number can be 1024, and the second number can be 8.

[0094] In some embodiments, a continuous latent vector of motion can be a continuous vector representation mapped from the original motion signal (such as human or robot motion trajectory, joint angles, end-effector pose sequence, or gesture label). This continuous latent vector of motion can serve as an implicit representation of the corresponding motion, preserving the semantic and temporal information of the motion.

[0095] In some embodiments, the continuous latent vector of the action, which represents the intermediate action, can be directly used as an input to the correlation decoding module or model. The correlation decoding module or model is used to generate action parameters.

[0096] For example, when receiving visual input data (such as displaying a small cup) and verbal instruction data (such as bringing the cup over), after feature extraction and fusion of the visual input data and verbal instruction data, a query vector q can be generated in the cross-modal shared semantic space. The original action sequence (such as hand trajectories and grasping postures in human motion videos) is encoded into corresponding continuous latent vectors of motion by an action encoder. These continuous latent vectors of motion are then compared with a pre-obtained table of the aforementioned continuous latent vectors of motion E = {e1, ..., e...}. i , ..., e N Establish a mapping. For example, based on the query vector q and each e iCalculate similarity scores (e.g., dot product or negative Euclidean distance), and select a second number k (e.g., k=3) of specified discrete latent vectors from the aforementioned action latent vector table (e.g., pinching and lifting, holding and lifting, pushing). Normalize these k terms according to their similarity using a normalized exponential function (softmax) to obtain attention weights, such as [0.6, 0.3, 0.1]. Then, these attention weights can be used to weightedly combine the specified discrete latent vectors to obtain a discrete action latent vector. Alternatively, when discretization is needed, nearest neighbor or other discretization strategies can be used to map the combination to some codebook indices. The latent vector combination corresponding to this codebook index serves as an intermediate semantic representation for fuzzy instruction completion and is fed into a multi-task decoder with shared underlying parameters. The output format is determined by combining the task type input and the dynamic dimension controller, thereby mapping the fuzzy "take" instruction to a specific action output.

[0097] In some embodiments, the above correspondence can be trained through the following general process. First, a large amount of human operation videos and their accompanying language instruction data (such as narration, subtitles, or text descriptions) from the Internet are collected and preprocessed, and temporal motion signals with human action annotations (such as human or hand keypoint sequences, motion capture trajectories, or trajectory parameters obtained by visual tracking / inverse kinematics) are extracted from the videos. Then, the above human action data is used as the training target of the corresponding action coding model; wherein, the action coding model can be an action encoder, for example, the action encoder can be a vector quantization variational autoencoder (VQ-VAE). During the training phase, the action encoder can map the above temporal motion signals into continuous latent representations. Subsequently, vector quantization (VQ) is used to map the nearest neighbors of the continuous latent representations to a learnable discrete vector table. The corresponding decoding module then uses the quantized vectors to reconstruct the original action signal. During the training phase, the commitment loss of vector quantization and the embedding vector update loss can be applied simultaneously to minimize the reconstruction error. For example, a pass-through estimator using Vector Quantization Variational Autoencoder (VQ-VAE) or a codebook update based on Exponential Moving Average (EMA) can be employed to obtain the aforementioned correspondence with composability. For instance, the human action data corresponding to "put the cup down on the table" might be encoded into multiple most relevant codebook entries or discrete indices; these most relevant codebook entries or discrete indices might include composable atomic units such as "approach, grasp, lift, move, place." The relevant decoding module or model can reconstruct continuous grasping and carrying trajectories based on the most relevant codebook entries or discrete indices. Secondly, the aforementioned correspondence, action encoder, decoder, visual feature extractor, and language encoder can be jointly trained or fine-tuned. Furthermore, cross-modal contrastive learning, such as triplet loss or Information Noise Contrast Estimation (InfoNCE), can be used to make the correct visual and linguistic representations closer to the corresponding action primitives or latent vector bases in the cross-modal shared semantic space, while negative samples push away mismatched combinations. This joint training can simultaneously optimize the motion encoder, the aforementioned correspondence, and the decoder during the pre-training phase to enhance intermodal alignment. Alternatively, a vector quantization variational autoencoder (VQ-VAE) can be pre-trained independently first, and then fine-tuned jointly with the visual feature extractor and the language encoder. Finally, domain adaptation can be performed using a small amount of real robot motion data to calibrate the mapping from latent vectors to specific robot motion parameters. Those skilled in the art can flexibly adjust the specific process of obtaining the aforementioned correspondence according to the needs of the actual scenario; no specific limitations are imposed here.

[0098] In some embodiments, the aforementioned action latent vector table may contain a first number of specified continuous latent vectors, wherein the action continuous latent vectors are obtained by attention matching on a second number of specified continuous latent vectors, the second number being less than the first number. The specific process for obtaining the action latent vector table in this case can refer to the aforementioned process for obtaining an action latent vector table containing several specified discrete latent vectors, and will not be repeated here.

[0099] In some embodiments, the aforementioned action latent vector table may contain several specified continuous latent vectors and / or specified discrete latent vectors. The specific process for obtaining this action latent vector table can refer to the aforementioned process for obtaining an action latent vector table containing several specified discrete latent vectors, and will not be repeated here.

[0100] As can be seen, the embodiments of this application can maintain the semantic coverage and composability provided by the aforementioned action latent vector table, and significantly reduce computational / storage overhead and improve retrieval efficiency by performing attention matching only on a smaller subset during the inference stage. Furthermore, by performing soft combination of multiple codebook entries instead of hard nearest neighbor replacement, quantization errors can be reduced, the smoothness of the representation can be improved, and the generalization ability to fuzzy instructions can be enhanced. The aforementioned quantization provides stable discrete semantic primitives, and attention matching provides a weighted combination and selection mechanism. The combination of the two expands the effective representation space and improves the robustness and generalization problems caused by directly hard-selecting a single entry.

[0101] In some embodiments, where the task type corresponding to the visual input data and / or language instruction data is always a single type, action parameters can be generated directly from the intermediate action representation without the task type participating in the generation of action parameters.

[0102] To address the issues of overly complex decoding processes and insufficient generalization capabilities caused by inconsistent output formats across various operational tasks (such as 6D grasping poses, 3D trajectories, and discrete motion sequences) in related methods, in some embodiments, the robot motion generation method may further include: acquiring a task type corresponding to the visual input data and / or the language instruction data; wherein, generating motion parameters based on the intermediate motion representation includes: determining the output dimension of the motion based on the task type; and generating the motion parameters based on the intermediate motion representation and the output dimension.

[0103] Understandably, task type serves as a conditional signal, guiding the relevant decoding modules or models to activate or configure the output structure corresponding to that task type when generating action parameters, thereby mitigating the mismatch caused by a uniform, fixed output. Shared underlying parameters of the relevant decoding modules or models preserve general semantics across tasks, allowing task-specific heads to learn only the details mapped to the target dimension, reducing learning difficulty and parameter duplication. Dimensional control combining dynamic / conditional design can simultaneously optimize general representations and task-specific mappings during training, ensuring that the decoding results are both semantically consistent and satisfy physical / format constraints.

[0104] As can be seen, this application's embodiments introduce task types during the decoding stage, determine the dimension and structure of the action output based on the task type, and then generate corresponding action parameters according to that dimension based on the shared intermediate action representation. Furthermore, through conditional decoding or dynamic selection / configuration of the output header, a single decoder can generate action outputs in multiple formats. In other words, it enables the same related decoding module or model to support multiple tasks and multiple action formats, reducing parameter redundancy caused by designing independent decoding modules or models for different tasks, improving the generalization ability of related decoding modules or models on new or few-sample tasks, and reducing deployment complexity. It also reduces decoding errors or additional post-processing caused by output dimension mismatch.

[0105] To address the issues of redundant decoding modules or models, large parameter count, high training and deployment costs, and poor cross-task generalization in multi-task action generation scenarios, some embodiments generate the action parameters by an action decoder. Specifically, for different task types, the action decoder reuses the weight parameters of at least some of its network layers.

[0106] In some embodiments, intermediate action representations and linguistic feature information can be input into a unified action decoder, which may include several layers of transformers, convolutional layers, or multilayer perceptrons (MLPs). Most of the preceding layers serve as a shared backbone to learn general action semantic representations, while the last few layers or the output branch into different mapping heads based on the task type. For example, a 6D mapping head that projects the shared representation into poses and quaternions can be used for 6D grasping, while a different 3D mapping head that unfolds the shared representation into a time series can be used for 3D trajectories. Alternatively, a small number of parameters of the shared network layers can be modulated by a task-conditional modulator, or lightweight task-adaptive layers can be inserted to allow for differentiation in output dimension and format by allowing a small number of task-specific parameters while maintaining the reuse of most network weights. During the training phase, the shared network layers, each task head, and / or task-adaptive layers are jointly or fine-tuned in stages to achieve "action parameters generated by the action decoder and reuse of at least some network layer weights across different tasks." Those skilled in the art can flexibly adjust the specific training process of the relevant decoding modules or models according to the actual needs of the scenario.

[0107] Understandably, shared network layers enable decoding modules or models to learn general action semantics and mapping rules from vision and language to actions. The features and / or weights learned by shared network layers can be used by different task types, thereby achieving knowledge transfer and efficient parameter reuse. At the top level, task conditionalization or lightweight branches are used to adapt to task differences, preserving the generality of shared representations while meeting task-specific output requirements.

[0108] In one embodiment, the relevant action encoding module and the relevant action decoding module can maintain a one-to-one correspondence. The relevant action encoding module can map the original action or trajectory to the latent vectors or discrete index sequences in the aforementioned action latent vector table, and the relevant action decoding module can use the aforementioned implicit representation as an intermediate semantic basis to generate the final action parameters. Since the relevant action encoding module and the action decoding module share the representation space for action semantics, they can reuse the same set or partially overlapping underlying networks (such as primary feature extraction layers, temporal / attention modules, or embedding layers), thereby reducing the frequency of repeatedly training independent low-level network structures for each task or each type of action. This sharing mechanism not only significantly reduces model parameters and storage overhead, but also improves the consistency of representations across different tasks, which is beneficial to the model's generalization ability in scenarios with sparse data or cross-task transfer. In addition, the intermediate action representation based on the same action latent vector table allows the relevant action decoding module to dynamically adjust the output format through a small amount of upper-layer control information (such as task type embedding or dimension controller), further reducing redundancy and improving reusability.

[0109] As can be seen, the embodiments of this application achieve multi-task output by allowing the action decoder to reuse the weights of at least some network layers for different task types when generating action parameters. For example, this is achieved by sharing low-level parameters and making conditional adjustments at the top level or in task-specific headers. An architecture using a shared encoder / decoder backbone plus task-conditional inputs or lightweight task heads can be adopted, allowing the same decoding module or model to reuse common representations across different tasks, branching only where necessary to meet task-specific output dimensions or format requirements. In other words, this significantly reduces the total number of parameters and computational overhead of the decoding module or model, reduces training and deployment resources, and simultaneously improves sample utilization and cross-task transfer learning capabilities. Sharing representations also suppresses overfitting on various tasks, improves generalization performance on tasks with few samples or new tasks, and facilitates the maintenance and expansion of multi-task systems.

[0110] In some embodiments, a task-specific decoder can be used to generate action parameters based on intermediate action representations. That is, the task-specific decoder can generate action parameters only for a preset task type.

[0111] To address the issue of insufficient alignment and generalization of visual, linguistic, and action modalities due to the scarcity and heterogeneity of robot-specific data in related methods, in some embodiments, the cross-modal shared semantic space can be obtained by training with specified visual input data, specified linguistic instruction data, and specified human action annotation data. At least a portion of the specified visual input data, specified linguistic instruction data, and specified human action annotation data originates from internet data sources.

[0112] In some embodiments, multimodal training data and a cross-modal shared semantic space can be constructed and trained according to the following general process. First, a large number of human operation videos (such as assembly or handling clips) are collected from the Internet. Video frames or short time-series clips are used as visual input data, and corresponding automatic speech recognition subtitles or manually annotated instruction texts are used as language instruction data. At the same time, human keypoint trajectories, hand pose sequences, or manually annotated action labels (such as "grab, lift, move to the table") are used as human action labels. These triples (such as visual clips, instruction texts, and action labels) are mixed with a small amount of real robot operation data and input into the relevant model or module. Through triple loss and other methods, the correct visual and language vectors are brought closer to the corresponding action vectors in the cross-modal shared semantic space, while incorrect pairings are pushed away, thereby learning a cross-modal shared semantic space that can align visual, language, and action data from multiple sources (at least some of which are from the Internet).

[0113] In some embodiments, the multimodal training data may include at least human-operated videos and / or their corresponding natural language descriptions from the Internet.

[0114] Understandably, internet data sources offer scale and semantic diversity, and methods such as triplet loss can solidify the above correspondences in a high-dimensional semantic space. Human action annotations, as behavioral examples, provide rich supervisory information for the mapping from vision and instructions to actions, thereby achieving more robust cross-modal transfer and generalization.

[0115] As can be seen, this embodiment of the application trains a cross-modal shared semantic space by using specified visual input data, specified language instruction data, and specified human action annotation data together, and allows at least some of the training samples to come from internet data sources. This leverages large-scale, diverse triples containing visual, linguistic, and action information for comparative learning to narrow the semantic distance of correct modal pairings. This cross-modal shared semantic space can serve as an intermediate representation for subsequent action generation or retrieval, enabling data from different sources to be utilized on the same semantic scale. In other words, it can significantly enhance the semantic alignment capability of visual, linguistic, and action information, improve generalization performance in new scenarios or on low-sample robot platforms, and reduce dependence on large amounts of robot annotation data.

[0116] Figure 2 This is a schematic diagram of a robot motion generation process provided in an embodiment of this application.

[0117] See Figure 2 The diagram illustrates a data input layer, a cross-modal feature extraction layer, a core technology layer, and an output layer. The technical approach of this flowchart involves starting with multi-source data, using a specially designed cross-modal feature extraction layer, aligning and representing visual, linguistic, and action modalities within a shared cross-modal semantic space, and then using implicit action encoding mechanisms and dynamic dimension decoders to transform high-level semantics into executable actions in different formats. Finally, the output layer provides diverse action parameters. Furthermore, it leverages massive amounts of internet data and human action annotations to enhance semantic alignment capabilities, and utilizes implicit action encoding and dynamic decoding mechanisms to achieve better generalization and adaptation when facing fuzzy instructions or multiple task requirements.

[0118] See Figure 2The data input layer can receive large amounts of internet video data, structured or semi-structured language instruction sets, and corresponding human action annotations. Internet video data provides rich visual scenes and behavioral examples. Language instruction sets provide user intent and task descriptions. Human action annotations help to obtain the corresponding relationships. To ensure consistency in subsequent processing, these three types of raw data can be preprocessed in a multimodal data preprocessing module. For example, frame extraction, keyframe or temporal slicing, visual enhancement, and normalization can be performed on internet video data; word segmentation, embedding, and context aggregation can be performed on language instruction sets; and format unification and time alignment can be performed on human action annotations to generate a multimodal input representation that is easy for relevant modules or models to use.

[0119] See Figure 2 The cross-modal feature extraction layer can include a visual feature extractor, a language encoder, and a motion encoder. The visual feature extractor can extract visual feature information from preprocessed video frames or images, such as objects, gestures, scene layout, and temporal changes. The language encoder can convert preprocessed instruction text into linguistic feature information to capture intent, action words, and object relationships. The motion encoder can encode preprocessed human action annotations or trajectory data into representations that express the temporal and dynamic characteristics of the actions. The outputs of the three encoders can be input into a feature fusion module, which aligns and integrates features from different modalities through attention mechanisms or cross-modal interaction strategies to form a multimodal feature representation for downstream semantic alignment and decoding.

[0120] See Figure 2The core technology layer can implement at least three execution paths for cross-modal alignment, implicit action encoding, and dynamic dimension decoding, respectively. These three execution paths can all be associated with a cross-modal shared semantic space. The first execution path embodies the cross-modal alignment mechanism. For example, cross-modal contrastive learning can be performed using a combination of visual and linguistic elements and action representations. Using triplet loss, based on the relative relationship between positive and negative samples, an alignment method can be learned that makes correct visual and linguistic combinations closer to their corresponding actions and further away from non-corresponding actions. Through this contrastive learning strategy, relevant modules or models can bring semantically identical information from different modalities closer together in the cross-modal shared semantic space, thereby establishing a stable cross-modal semantic mapping. The second execution path embodies implicit action encoding. For example, the output of the corresponding action encoder can be further mapped to the implicit expression of the above correspondence, i.e., the codebook implicit expression. The aforementioned implicit action vector table can include a set of discrete or continuous implicit vector bases. Each implicit vector base can be regarded as a vectorized representation of a basic action unit or action intention. After mapping the original action sequence to the aforementioned latent vector table, complex actions can be represented as combinations or sequences of several latent vector bases, thereby achieving semantic compression and abstraction of actions. Implicit action encoding enables not only the representation of explicit, atomic actions, but also the description of ambiguous or multifaceted operational intentions using higher-level semantic units. The third execution path is used to generate action parameters. This path can be used to decode high-level semantics in the cross-modal shared semantic space into specific action parameters according to task requirements using a multi-task action decoder. The multi-task action decoder can include a dynamic dimension decoder. A dynamic dimension decoder has a mechanism that dynamically adjusts the output dimension according to the task type, such as outputting corresponding pose parameters when six-dimensional grasping posture is required, and outputting trajectory sequences when three-dimensional motion trajectory is required. By using the task type as a conditional input, the corresponding decoder can flexibly configure different output structures and use a small number of task-specific parameters to complete the format conversion while reusing the weight parameters of at least some of its network layers. These three execution paths can work collaboratively. The cross-modal alignment mechanism execution path establishes semantic consistency between vision, language, and action through contrastive learning. The implicit action encoding execution path provides structured action semantic units through the aforementioned implicit action vector table. The dynamic dimension decoder execution path can extract relevant information from the cross-modal shared semantic space and generate executable action parameters according to specific task requirements. The cross-modal shared semantic space can contain intent information extracted from visual input data and language instruction data, as well as action intention representations abstracted from action data.

[0121] See Figure 2The output layer can be used to input the corresponding decoding results into the task adaptation output module based on the conditional representation provided by the cross-modal shared semantic space and combined with the task type (such as 6D grasping posture, 3D motion trajectory, or other action types). The task adaptation output module can output the corresponding action parameters. For example, for a six-dimensional grasping posture, it can generate the end-effector pose and grasping parameters; for a three-dimensional motion trajectory, it can output a spatial point sequence or a time-parameterized trajectory; for other action types, it can be formatted and constrained according to the corresponding action format. The task adaptation output module can perform necessary physical constraint verification and format conversion to ensure that the generated action not only semantically satisfies the instruction intent but is also physically acceptable to the target platform.

[0122] Figure 3 This is a schematic diagram of a training process for a cross-modal shared semantic space provided in an embodiment of this application. The general technical idea of ​​this training process diagram is to form a closed-loop learning mechanism by starting with data input, feature extraction, triple construction, calculation of similarity matrix, calculation of contrast loss, parameter optimization, and then updating the cross-modal shared semantic space (i.e., the training process of the cross-modal shared semantic space), which can gradually build reliable cross-modal semantic alignment on large-scale heterogeneous data.

[0123] See Figure 3 Starting with data input, the system can receive three types of modal inputs: visual frames, language commands, and action sequences. Visual frames can be single-frame images or multi-frame video streams, reflecting the appearance and temporal changes of scenes and objects. Language commands can be text descriptions of tasks or intentions, or speech-to-text instructions. Action sequences can be human operations or robot execution trajectories corresponding to visual and language descriptions; these can take the form of keypoint sequences, hand trajectories, Cartesian space trajectories, or robot joint state sequences, etc. Subsequently, feature extraction is performed on these three types of modal inputs, for example, by their respective feature extractors, to obtain high-level embedding representations for each modality. The visual feature extractor can convert visual frames into multi-scale visual embeddings, the language encoder can convert language commands into semantic language embeddings, and the action encoder can encode action sequences into action embedding vectors. Feature extraction for each modality can either employ independent network structures or share some low-level parameters.

[0124] See Figure 3After obtaining the corresponding modal embeddings, triples can be constructed to facilitate cross-modal contrastive learning. The core idea of ​​triple construction is to treat related visual and linguistic embeddings as one set of positive sample pairs, and the corresponding action embeddings as another set of positive sample pairs, representing the correct cross-modal pairing relationship. Simultaneously, to provide discriminative signals for training the model to distinguish between correct and incorrect matches, negative sample pairs can also be constructed. Negative sample pairs can be random modal combinations. Random modal combinations can be obtained by randomly combining embeddings from different modalities within a batch, such as pairing a visual and linguistic embedding with a non-corresponding action embedding, or pairing a visual embedding with a mismatched linguistic embedding, etc. The construction of negative samples can employ random sampling or more complex sampling strategies, such as hard negative sample mining, to enhance the discriminative ability of the training. During the training phase, visual and linguistic embeddings can serve as anchors in contrastive learning and, together with the action embeddings in the positive / negative sample pairs, are used for cross-modal alignment.

[0125] See Figure 3After constructing the positive and negative sample pairs, a similarity matrix can be calculated for the generated embedding set. The similarity matrix records the pairwise similarity between embeddings of different modalities in a batch, measuring the correlation between visual, linguistic, and action embeddings. Similarity can be measured using vector similarity methods, such as cosine similarity or dot product. Semantic intuition similarity can also be used as a metric. The similarity matrix efficiently obtains the relative tightness of all positive and negative pairs in a batch, serving as input for subsequent loss calculations. Based on the similarity matrix, a triplet loss can be calculated, and the corresponding network model or module can be trained. The goal of this loss is to bring correctly paired modal embeddings closer together in the cross-modal shared semantic space, while pushing incorrectly paired embeddings further apart, thereby gradually forming a consistent mapping between visual, linguistic, and action embeddings. Specifically, during training, the joint embeddings of visual and linguistic data are encouraged to be closer to the corresponding action embeddings, while simultaneously reducing the similarity between visual or linguistic data and mismatched actions. In this way, the corresponding modules or models can learn how to aggregate different modal representations describing the same semantics in a cross-modal shared semantic space, so that the correct action representation can be accurately indexed using visual and linguistic information during subsequent retrieval or decoding. After the loss function is calculated, the parameters of the relevant trainable modules can be updated through parameter optimization. Parameter optimization can be performed by using a deep learning optimizer to jointly train or stage-wise train modules such as feature extractors, modality projection layers, and action encoders. The optimization objective can be to minimize the weighted combination of contrastive loss and other parallel supervision terms (such as reconstruction loss or behavior cloning loss). As training iterates, the parameters of each part of the relevant modules or models are continuously adjusted, and the embedding space is gradually reshaped and tends to stabilize, thus forming a unified cross-modal shared semantic space. The update of the cross-modal shared semantic space can be reflected in the change of the relative positions between visual, linguistic, and action embeddings. Correct trimodal combinations exhibit a clustered structure in this space, while incorrect combinations are dispersed.

[0126] See Figure 3To improve training effectiveness and engineering feasibility, several strategies can be adopted. First, since the construction of triples and the sampling strategy for negative samples significantly affect training efficiency, a hybrid negative sample strategy can be used, such as random negative samples, hard negative samples, and cross-batch negative samples, to balance stability and discriminativity. Second, the calculation of the similarity matrix is ​​usually completed within mini-batches to control computational and memory overhead; approximate retrieval or feature compression methods can be introduced if necessary. Third, pre-training individual modal encoders followed by joint contrastive training can accelerate convergence and improve the final alignment quality. Furthermore, for large-scale data from the internet, temporal segmentation, action normalization, and data augmentation can be performed during preprocessing to reduce the impact of source domain noise on alignment results. Finally, during training, the distribution changes of positive and negative samples in the cross-modal shared semantic space and the retrieval accuracy can be periodically evaluated, and hyperparameters such as the learning rate and sampling ratio can be adjusted accordingly. Figure 3 Through iterative iterations and parameter updates at each step, a model or module can ultimately be obtained that maps visual, linguistic, and action representations to the same cross-modal shared semantic space. This cross-modal shared semantic space not only facilitates the retrieval of appropriate action embeddings through joint visual and linguistic embeddings during the inference phase, but also provides stable semantic conditions for subsequent action decoding and multi-task generation modules, thereby improving the completion capability of fuzzy instructions and the generalization of action generation.

[0127] Figure 4 This is a schematic diagram of the execution flow of an implicit action coding module and a multi-task action generation module provided in an embodiment of this application.

[0128] See Figure 4The implicit action coding module can take raw action sequences as input. These raw action sequences can come from keypoints in human operation videos, hand trajectory records, object pose change sequences, or joint angle / end-effector pose sequences of the robot body. The implicit action coding module can map these raw action sequences to an intermediate representation space through corresponding action coding modules or models. A corresponding codebook mapping can map continuous raw action sequences to indices or corresponding combinations of latent vectors in the aforementioned action latent vector table, thereby compressing the raw action sequences into a composable set of discrete latent vectors, thus decomposing complex continuous actions into several reusable and composable high-level semantic units. Furthermore, the semantic representation of language instruction data can be extracted simultaneously, i.e., language instruction semantics. Language instruction semantics can reflect information such as intent, object description, and action expectation in the language instruction data. To achieve semantic alignment between actions and language, relevant modules or models can use an attention matching mechanism to interact the language instruction semantics with the discrete latent vector set in the aforementioned action latent vector table. The essence of attention matching is to retrieve the most relevant combination of latent vectors from the aforementioned action latent vector table, ensuring that the high-level semantics of the language can find corresponding or similar latent vector representations within the table. This matching can be learned during the training phase from labeled action-language pairs, or it can be achieved by combining a pre-trained action latent vector table with online retrieval. During this process, if the language command data is ambiguous or unclear, such as expressions like "bring it here" or "move it to the side" that do not specify a particular grasping posture or path, attention matching can be used. Based on the semantics of the language command, the contextual information obtained through semantic alignment, and the set of discrete latent vectors already learned in the action latent vector table, the most suitable implicit action representation can be inferred and the missing information filled in—that is, fuzzy command completion. The completion result of fuzzy command completion is the implicit action completion output of the fuzzy command. The completion result can use a set or series of latent vectors to represent the expected action intention rather than direct physical control quantities.

[0129] See Figure 4The implicit action encoding module outputs completed latent vectors and / or combinations of latent vectors, which can then be input to the shared underlying parameters of the multi-task action generation module. These shared underlying parameters can refer to a set of network parameters or feature representation layers common across different tasks. These shared underlying parameters can be used to transform implicit semantic conditions (such as contextual information obtained through semantic alignment) into a general representation suitable for further use by relevant decoding modules or models. Connecting the completion results of fuzzy instruction completion to these shared underlying parameters aims to ensure that related subsequent tasks benefit from the same high-level semantic abstraction, thereby improving the consistency of interpretation of ambiguous instructions and enhancing cross-task transferability. Through this design, implicit completion can not only serve a single-task-type decoder but also become a shared semantic prior in a multi-task system, reducing the need for repeated learning of fuzzy intent mappings across different tasks. The multi-task action generation module can receive the above completion results along with additional task type input as a conditional signal. The task type input can indicate the type of action to be generated. Based on this task type signal, a dynamic dimension adjustment mechanism can be used to dynamically configure the output dimensions and data format of subsequent related decoding modules or models to match the specific task's requirements for action representation. Dynamic dimensionality adjustment can be achieved by a dynamic dimensionality controller. This controller can determine the type of decoder, the number of output channels, and whether adjustments to the timing length or output precision are needed based on the task type input and shared semantic input. The dynamic dimensionality controller can switch the output structure in real-time during inference and can also learn the feature mapping methods for different tasks through conditional learning during training.

[0130] See Figure 4 The multi-task action generation module can employ a parameter-sharing mechanism to reduce redundancy and enhance knowledge transfer between tasks. Specifically, the relevant module can divide the relevant network layers into a low-level shared layer and several task-specific decoders. The low-level shared layer can be used to extract and integrate high-level semantics, linguistic and visual conditions from the implicit action encoding module, as well as task type input. The parameters of the low-level shared layer can be universal across all tasks, ensuring consistency in understanding fuzzy instructions and basic action semantics across different tasks. Each task has its own task-specific decoder above the low-level shared layer. These dedicated decoders can further transform the shared representation according to task requirements to generate action representations in specific formats and dimensions. The task-specific decoders can be different network heads or branches of the same structure but with independent parameters, mapping the universal semantic representation to the task-related action space.

[0131] See Figure 4After processing by the task-specific decoder, motion space mapping can be performed to output executable motion commands using the decoding module or model. This step may include converting high-level motion representations into low-level control quantities that the robot or actuator can understand, such as mapping the decoding results to six-dimensional poses and corresponding gripping parameters, or mapping trajectory vectors to end-effector position and velocity commands at each time step, or further mapping them to joint angle sequences of drive motors. The motion space mapping module may include physical constraint correction, path planning, or inverse kinematics solving to ensure that the generated motion is both semantically consistent with the instruction intent and physically safe for execution by the target robot. Finally, the motion output layer outputs these mapping results in the target format for use by the robot's relevant controllers or actuators. The implicit motion encoding module and the multi-task motion generation module can work closely together. The implicit motion encoding module provides semantic completion for ambiguous or incomplete instructions through the aforementioned implicit motion vector table and attention matching. Shared low-level parameters pass this completed, high-level semantic condition to the corresponding task-specific decoder. The dynamic dimension controller can adjust the structure and output dimensions of the decoding module or model according to the task type, while the task-specific decoder and action space mapping can complete the final conversion from semantics to executable control variables. This design enables the sharing of semantic understanding capabilities at the underlying level when facing multiple tasks, while maintaining targeted decoding capabilities at the upper level, thus balancing generalization and task refinement requirements. At the same time, by incorporating the results of fuzzy instruction completion into the shared part, it is possible to quickly transfer and concretize action intentions learned from large-scale Internet data into executable actions with fine-tuning of a small number of specific robot examples, improving zero-shot or few-shot generalization ability in new scenarios, new objects, or new platforms.

[0132] The above describes the method for generating robot motions. This application also provides a robot motion generation system. Figure 5 This is a schematic diagram of a robot motion generation system provided in an embodiment of this application. See also... Figure 5 The system may include: a data input module 101, used to acquire the robot's visual input data and its corresponding language instruction data; a processing module 102, used to obtain intermediate action representations by utilizing a pre-constructed cross-modal shared semantic space, the correspondence between shared semantic representations and intermediate action representations, the visual input data and the language instruction data; and a decoding module 103, used to generate action parameters based on the intermediate action representations, so as to drive the robot to perform actions corresponding to the visual input data and / or the language instruction data.

[0133] In some embodiments, the data input module is further configured to acquire a task type corresponding to the visual input data and / or the language instruction data. The decoding module is also configured to determine the output dimension of the action based on the task type; and to generate the action parameters based on the intermediate action representation and the output dimension.

[0134] In this embodiment, the specific functions and effects of the robot motion generation system can be explained by referring to other embodiments of this application, and will not be repeated here.

[0135] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the methods described in the above embodiments.

[0136] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements any of the methods described above.

[0137] The computer program product may be a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the computer program product of the embodiments of this application is not limited thereto, and the computer program product may be any combination of one or more computer-readable media.

[0138] See Figure 6 , Figure 6 This is a structural block diagram of a computer device provided in an embodiment of this application.

[0139] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement any of the methods described in the above embodiments.

[0140] The embodiments of this application do not limit the computer device, which may be, for example, a local computer device, a cloud computer device, a distributed computer device, etc.

[0141] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected through internal connection paths.

[0142] The memory 110 is used to store computer programs, which in some implementations may include code for implementing the methods of the embodiments of this application.

[0143] The processor 120 executes the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information, and output operation results and other data. In some implementations, when the solutions of the embodiments of this application are implemented by software or firmware, the computer program used to implement the solutions of the embodiments of this application can be stored in the processor 120 and executed by the processor 120.

[0144] The memory 110 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any memory of these and other suitable types. As an example, the memory 110 includes random access memory (RAM), cache memory, and read-only memory (ROM). The memory 110 stores a computer program that can be executed by processor 120, causing processor 120 to implement the steps of any of the methods described above.

[0145] The processor 120 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 120 can be any conventional processor.

[0146] In implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware in the processor 120 or by instructions in software form. The method disclosed in the embodiments of this application can be directly implemented by the hardware processor, or by a combination of hardware and software modules in the processor 120. The software modules can be located in mature storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in the memory 110, and the processor 120 reads the information in the memory 110 and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are not provided here.

[0147] In some implementations, in addition to the hardware units described above, computer devices may also include software modules, such as operating systems, basic input / output systems (BIOS), and application software.

[0148] An operating system is used to manage the hardware and / or software resources of a computer device; it is the kernel and foundation of the computer. The operating system handles fundamental tasks such as managing and configuring memory, determining the priority of system resource allocation, controlling input and output devices, operating the network, and managing the file system. To facilitate user operation, most operating systems provide a user interface for interaction with the system.

[0149] The BIOS is used to perform hardware initialization during the power-on boot phase and to provide runtime services for the operating system and applications. In some implementations, the BIOS can also monitor and display processor temperature and execute temperature protection strategies.

[0150] Application software, also known as an application program, can be understood as software written for a specific user application purpose, and is one of the main categories of computer software. For example, application software can be a program used to achieve purposes such as power control and temperature management.

[0151] It is understood that the specific examples in this application are only intended to help those skilled in the art better understand the implementation of this application, and are not intended to limit the scope of protection of this application.

[0152] It is understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0153] It is understood that the various implementation methods described in this application can be implemented individually or in combination, and this application does not limit them.

[0154] Unless otherwise stated, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0155] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0156] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the embodiments described above can be referred to the corresponding processes in other embodiments, and will not be repeated here.

[0157] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0158] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the technical solution in this application, depending on actual needs.

[0159] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0160] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to related technologies, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0161] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating robot actions, characterized in that, The method includes: Acquire the robot's visual input data and its corresponding language command data; The intermediate action representation is obtained by utilizing a pre-constructed cross-modal shared semantic space, the correspondence between shared semantic representation and intermediate action representation, the visual input data, and the language instruction data; wherein, the shared semantic representation is obtained based on the visual input data and / or the language instruction data and by utilizing the cross-modal shared semantic space; Action parameters are generated based on the intermediate action representation to drive the robot to perform actions corresponding to the visual input data and / or the language instruction data.

2. The method according to claim 1, characterized in that, The process of obtaining the intermediate action representation includes: Using the cross-modal shared semantic space and the correspondence, the shared semantic representation is converted into continuous latent vectors of actions and / or discrete latent vectors of actions; The continuous latent vector and / or discrete latent vector of the action are used as the intermediate action representation.

3. The method according to claim 1, characterized in that, The process of obtaining the intermediate action representation includes: The shared semantic space across modalities is used to convert the shared semantic representation into continuous latent vectors of actions. Based on the continuous latent vector of the action and the correspondence, the corresponding discrete latent vector of the action is determined, and the discrete latent vector of the action is used as the intermediate action representation.

4. The method according to claim 2 or 3, characterized in that, The step of generating action parameters based on the intermediate action representation includes: The action parameters are generated using the discrete latent vector of the action and the shared semantic representation.

5. The method according to claim 3, characterized in that, The correspondence is based on an action latent vector table as the underlying representation space. The action latent vector table contains a first number of specified discrete latent vectors. The action discrete latent vectors are obtained by attention matching on a second number of specified discrete latent vectors. The second number is less than the first number.

6. The method according to claim 1, characterized in that, The method further includes: obtaining the task type corresponding to the visual input data and / or the language instruction data; The step of generating action parameters based on the intermediate action representation includes: Based on the task type, determine the output dimension of the action; The action parameters are generated based on the intermediate action representation and the output dimension.

7. The method according to claim 6, characterized in that, The action parameters are generated by an action decoder; wherein, for different task types, the action decoder reuses the weight parameters of at least some of its network layers.

8. The method according to claim 1, characterized in that, The cross-modal shared semantic space is obtained by training with specified visual input data, specified language instruction data, and specified human action annotation data; wherein at least a portion of the specified visual input data, specified language instruction data, and specified human action annotation data comes from Internet data sources.

9. A robot motion generation system, characterized in that, include: The data input module is used to acquire the robot's visual input data and its corresponding language command data; The processing module is used to obtain intermediate action representations by utilizing a pre-constructed cross-modal shared semantic space, the correspondence between shared semantic representations and intermediate action representations, the visual input data, and the language instruction data; wherein, the shared semantic representations are obtained based on the visual input data and / or the language instruction data and by utilizing the cross-modal shared semantic space; The decoding module is used to generate motion parameters based on the intermediate motion representation, so as to drive the robot to perform actions corresponding to the visual input data and / or the language instruction data.

10. The system according to claim 9, characterized in that, The data input module is further configured to acquire the task type corresponding to the visual input data and / or the language instruction data; wherein, the decoding module is further configured to determine the output dimension of the action according to the task type, and generate the action parameters based on the intermediate action representation and the output dimension.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

12. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method of any one of claims 1 to 8.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images