Robot action generation method and device, robot, medium and product

Abstract

Embodiments of the present application provide a robot action generation method and device, a robot, a medium and a product. The method comprises: receiving visual observation information corresponding to a target operation task and task instruction information, and performing feature extraction based on the visual observation information and the task instruction information to obtain visual representation information and instruction representation information; determining environment representation information for representing an environment context in the visual observation information based on the visual representation information; determining object intermediary representation information based on a cross-modal association between the visual representation information and the instruction representation information; determining action intermediary representation information based on the instruction representation information; and generating robot action data based on the visual representation information, the environment representation information, the instruction representation information, the object intermediary representation information and the action intermediary representation information. The embodiments of the present application reduce the influence of visual confusion factors and language expression deviation on the robot action generation result.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to robot motion generation methods and apparatus, robots, media, and products. Background Technology

[0002] With the development of artificial intelligence, robot perception and control, and embodied intelligence technologies, robots are increasingly being applied in various scenarios such as home service, industrial assembly, warehouse sorting, laboratory automation, medical assistance, and educational companionship. In these applications, robots need to generate corresponding motion data based on their perception of the surrounding environment and task instructions given by the user or upper-level task planning module. This data is then used to control robotic arms, mobile chassis, end effectors, or other actuators to perform tasks such as grasping, placing, moving, pushing, pulling, opening, closing, inserting, and organizing. Currently, visual confusion and language expression deviations significantly impact the robot's motion generation results, making it difficult to meet the needs of practical applications.

[0003] Based on this, embodiments of this application provide a method and apparatus for generating robot motions, a robot, a medium, and a product to improve related technologies. Summary of the Invention

[0004] The purpose of this application is to provide a method and apparatus for generating robot motions, as well as a robot, medium, and product, to reduce the impact of visual confusion factors and language expression deviations on the robot motion generation results.

[0005] In a first aspect, embodiments of this application provide a robot motion generation method, comprising: receiving visual observation information and task instruction information corresponding to a target operation task, and performing feature extraction based on the visual observation information and the task instruction information respectively to obtain visual representation information and instruction representation information; determining environmental representation information for representing the environmental context in the visual observation information based on the visual representation information; determining object mediation representation information based on the cross-modal association between the visual representation information and the instruction representation information, wherein the object mediation representation information is used to represent the task object associated with the target operation task; determining action mediation representation information based on the instruction representation information, wherein the action mediation representation information is used to represent the action semantics or action distribution features associated with the target operation task; and generating robot motion data based on the visual representation information, the environmental representation information, the instruction representation information, the object mediation representation information, and the action mediation representation information.

[0006] Secondly, embodiments of this application provide a robot motion generation device, comprising: a feature extraction module, configured to receive visual observation information and task instruction information corresponding to a target operation task, and perform feature extraction based on the visual observation information and the task instruction information respectively to obtain visual representation information and instruction representation information; an environment representation module, configured to determine environment representation information for representing the environmental context in the visual observation information based on the visual representation information; an object mediation representation module, configured to determine object mediation representation information based on the cross-modal association between the visual representation information and the instruction representation information, wherein the object mediation representation information is used to represent the task object associated with the target operation task; a motion mediation representation module, configured to determine motion mediation representation information based on the instruction representation information, wherein the motion mediation representation information is used to represent the motion semantics or motion distribution features associated with the target operation task; and a motion generation module, configured to generate robot motion data based on the visual representation information, the environment representation information, the instruction representation information, the object mediation representation information, and the motion mediation representation information.

[0007] Thirdly, embodiments of this application provide a robot, which includes a control module for performing the steps of any of the methods described above.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0009] Fifthly, embodiments of this application provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the steps of any of the above methods.

[0010] This application provides a robot motion generation method and apparatus, robot, medium, and product. It adopts the concept of co-participating environmental context representation, object mediation representation, and motion mediation representation in motion generation, thereby reducing the impact of visual confusion factors and language expression deviation on the robot motion generation results. Attached Figure Description

[0011] The embodiments of this application are further described below with reference to the accompanying drawings and specific implementation details.

[0012] Figure 1 This is a flowchart illustrating a robot motion generation method provided in an embodiment of this application.

[0013] Figure 2 This is a schematic diagram of the structure of an action generation model provided in an embodiment of this application.

[0014] Figure 3 This is a structural block diagram of a robot motion generation device provided in an embodiment of this application.

[0015] Figure 4 This is a structural block diagram of a robot provided in an embodiment of this application.

[0016] Figure 5 This is a structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the embodiments of this application.

[0018] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0019] With the development of artificial intelligence, robot perception and control, and embodied intelligence technologies, robots are increasingly being applied in various scenarios such as home services, industrial assembly, warehouse sorting, laboratory automation, medical assistance, and educational companionship. In these applications, robots need to generate corresponding motion data based on the perception results of the surrounding environment and the task instructions given by the user or the upper-level task planning module. This data is then used to control the robotic arm, mobile chassis, end effector, or other actuators to complete tasks such as grasping, placing, moving, pushing, pulling, opening, closing, inserting, and organizing.

[0020] In robotic operations, visual observation information and task command information are two crucial inputs. However, in open environments, the visual scenes and verbal expressions faced by robots exhibit significant uncertainty. Changes in the visual scene and verbal expression increase the difficulty of generating robot actions. Relevant robot action generation methods often rely on the statistical correlation between visual input, verbal input, and action data in the training samples. When certain backgrounds, lighting, table textures, camera perspectives, or verbal phrases frequently co-occur with specific action results in the training samples, the model may mistakenly use these factors as important bases for action generation. When the environmental background or verbal expression changes, the robot action data may become biased. Simply relying on increasing the data scale or adding data augmentation methods is insufficient to fundamentally reduce the impact of changes in the visual environment and verbal expression on the action generation results.

[0021] See Figure 1 , Figure 1 This is a flowchart illustrating a robot motion generation method provided in an embodiment of this application.

[0022] To reduce the impact of visual confusion and language expression deviation on the robot motion generation results, this application provides a robot motion generation method, including steps S101 to S105.

[0023] Step S101: Receive visual observation information and task instruction information corresponding to the target operation task, and perform feature extraction based on the visual observation information and the task instruction information to obtain visual representation information and instruction representation information.

[0024] Step S102: Based on the visual representation information, determine the environmental representation information used to represent the environmental context in the visual observation information.

[0025] Step S103: Based on the cross-modal association between the visual representation information and the instruction representation information, determine object mediation representation information, which is used to represent the task object associated with the target operation task.

[0026] Step S104: Determine action mediation information based on the instruction representation information, wherein the action mediation representation information is used to represent the action semantics or action distribution features associated with the target operation task.

[0027] Step S105: Generate robot motion data based on the visual representation information, the environmental representation information, the instruction representation information, the object mediation representation information, and the action mediation representation information.

[0028] A target operation task refers to an operation task that a robot needs to perform based on visual observation information and task instruction information. Target operation tasks can include grasping tasks, placement tasks, movement tasks, opening and closing tasks, pushing and pulling tasks, assembly tasks, and tidying tasks. For example, a target operation task could be "placing the blue cup on the table into the yellow tray," or "grabbing the red block and placing it in the designated area." Target operation tasks are used to define the task objective served by the robot's motion generation process, enabling the robot's motion data to be generated around the target object, target position, and target action intent.

[0029] Visual observation information refers to the visual perception data obtained by a robot when performing a target operation task. Visual observation information can include image data from one or more perspectives, such as a global view image captured by the robot's head camera, a local view image captured by the end effector camera of the robotic arm, and a top-down view image captured by a camera above the worktable. Visual observation information can be used to characterize target objects, containers, obstacles, background areas, lighting conditions, texture information, and spatial relationships between objects in the scene where the target operation task is performed.

[0030] Task instruction information refers to instructions used to describe the target operation task. Task instruction information can be natural language text, text obtained from speech recognition, structured task descriptions, or semantic instructions output by the upper-level task planning module. For example, task instruction information could be "put the blue cup into the yellow tray" or "grab the left cube and put it into the right box." Task instruction information is used to provide the robot with semantic information such as the operation object, operation action, target position, and operation constraints.

[0031] Visual representation information refers to visual feature data obtained by extracting features from visual observation information. Visual representation information can include multiple visual sub-representations, each of which can correspond to an image patch, object region, local texture region, spatial location region, or multi-view fusion region. Through visual representation information, pixel-level data in visual observation information can be converted into feature-level data that is easy for action generation models to process, supporting subsequent environmental representation extraction, object mediation representation determination, and action data generation.

[0032] Instruction representation information refers to semantic feature data obtained by feature extraction from task instruction information. Instruction representation information can include multiple instruction sub-representations, each corresponding to a word, phrase, semantic fragment, or structured instruction field in the task instruction information. Through instruction representation information, the textual expression in the task instruction information can be converted into semantic features that are easy for action generation models to process, supporting subsequent cross-modal association calculations and the determination of action mediation representations.

[0033] Contextual information refers to environmental factors in visual observation that are weakly related to the target task but may affect the action generation result. Contextual information can include lighting intensity, shadows, table texture, background objects, background color, camera viewpoint, reflective areas, and the distribution of cluttered objects. Contextual information does not necessarily directly determine which task object the robot should grasp or which type of action it should perform, but if the action generation model relies too heavily on contextual information, it may produce action generation biases when the environment changes.

[0034] Environmental representation information refers to feature information determined based on visual representation information, used to characterize the environmental context in visual observation information. Environmental representation information can be obtained through cross-attention processing between query information and visual representation information, or by extracting background-related regions, non-task-object regions, or environment-related features from the visual representation information. Environmental representation information can be used to explicitly provide environmental context-related features to the action generation model, enabling the model to distinguish between task-object-related features and environmental context-related features during the fusion process, thereby reducing the interference of environmental context on robot motion data.

[0035] Cross-modal association refers to the matching, correspondence, or semantic association between visual representation information and instruction representation information. Cross-modal association can be determined through methods such as similarity calculation, attention calculation, projection matching, and relevance ranking. For example, visual representation information and instruction representation information can be projected onto the same feature space, and the degree of matching between visual sub-representations and instruction sub-representations can be calculated to determine which visual regions are more relevant to the target object, target location, or operational semantics in the task instruction information.

[0036] Object-mediated representation information refers to intermediate feature information used to represent task objects associated with a target operation task, determined based on cross-modal associations between visual representation information and instruction representation information. Object-mediated representation information can be used to represent the relative positional relationships between target objects, target containers, operation areas, and task objects. For example, in the target operation task of "placing the blue cup in the yellow tray," object-mediated representation information can be used to represent the spatial relationships between the blue cup, the yellow tray, and the blue cup and the yellow tray. By using object-mediated representation information, the reliance of action generation models solely on the representation of task instructions for action prediction can be reduced.

[0037] Intermediate action representation information refers to intermediate feature information determined based on instruction representation information, used to represent the semantics of actions or the distribution characteristics of actions associated with a target operation task. Intermediate action representation information can be used to represent action types such as grasping, moving, placing, pushing away, pulling back, rotating, and inserting, as well as the distribution characteristics of action stages, action trends, action trajectory distribution, or future action sequences. For example, in the target operation task of "placing the blue cup on the yellow tray," intermediate action representation information can be used to represent the semantics of actions such as "approaching the cup, picking up the cup, moving to the tray, and releasing the cup."

[0038] Action semantics refers to the semantic content in task instruction information related to the robot's action execution method. Action semantics can include action category, action sequence, action direction, action target, action constraints, and action stage. For example, "grasp" corresponds to the end effector approaching the target object and closing the gripper, while "place" corresponds to the end effector carrying the target object, moving it above the target container, and releasing the target object. Action semantics is used to guide the action generation model in determining the execution direction and action mode of the robot's action data.

[0039] Motion distribution features refer to characteristic information used to characterize the possible distribution range or trend of robot motion data. Motion distribution features can include the distribution characteristics of future motion trajectories, the distribution of latent motion variables, the mean characteristics of motion, the variance characteristics of motion, and the probability characteristics of motion stages. Motion distribution features can be used to express the different motion implementation methods that the same task instruction information may correspond to in different visual scenarios, thereby improving the flexibility and adaptability of robot motion data generation.

[0040] Robot motion data refers to data used to control a robot to perform target operations. Robot motion data can include position increments, attitude increments, joint angles, joint velocities, joint torques, gripper opening and closing commands, or motion sequences over multiple target control cycles. For example, robot motion data can include changes in end-effector pose and gripper opening and closing states over eight target control cycles, used to control the robot to gradually approach, grasp, move, and release the target object.

[0041] In this embodiment, after receiving visual observation information and task instruction information corresponding to the target operation task, the robot motion generation method extracts visual representation information and instruction representation information respectively. On the one hand, the robot motion generation method determines environmental representation information based on visual representation information, enabling the environmental context in the visual observation information to participate in subsequent motion generation as an independent feature factor. On the other hand, the robot motion generation method determines object mediation representation information based on the cross-modal association between visual representation information and instruction representation information, so as to anchor the object semantics in the task instruction information to the task object in the visual scene. At the same time, the robot motion generation method determines motion mediation representation information based on instruction representation information to represent the motion semantics or motion distribution features associated with the target operation task. Subsequently, the robot motion generation method uses visual representation information, environmental representation information, instruction representation information, object mediation representation information, and motion mediation representation information together for motion generation, thereby generating robot motion data that can control the robot to perform the target operation task.

[0042] This embodiment determines environmental representation information based on visual representation information, explicitly introducing environmental context factors such as lighting, texture, background objects, and camera viewpoint into the motion generation process as features. This reduces the over-reliance of robot motion generation methods on a single visual background or specific environmental conditions. By determining object-mediated representation information based on cross-modal association between visual and instruction representation information, the accuracy of the correspondence between task instruction information and task objects in the visual scene can be improved, thereby reducing motion deviations caused by unstable task object positioning. By determining motion-mediated representation information based on instruction representation information, the action semantics or action distribution features in the task instruction information can be introduced into the motion generation process, thereby improving the matching degree between robot motion data and the target operation task. By using visual, environmental, instruction, object-mediated, and motion-mediated representation information together to generate robot motion data, the robot motion generation method can simultaneously consider the visual scene, environmental context, language semantics, task objects, and action intentions, thereby improving the robustness and generalization ability of robot motion generation in open scenes.

[0043] This embodiment addresses the problem that related visual language action models easily learn the superficial correlation between visual background, language expression and action, leading to insufficient stability in action generation when the environment or language expression changes. It adopts the concept of co-participating environmental context representation, object mediation representation and action mediation representation in action generation, thereby reducing the impact of visual confusion factors and language expression deviation on the robot action generation results.

[0044] In one example, the robot motion generation method receives visual observation information and task instruction information corresponding to the target operation task. The visual observation information includes a global image captured by the head camera and a local image captured by the wrist camera, both with a size of 224×224. The task instruction information is "Place the blue cup in the yellow tray." The robot motion generation method performs visual encoding on the visual observation information, obtaining 196 visual sub-representations, each with a dimension of 768; the method also performs text encoding on the task instruction information, obtaining 8 instruction sub-representations, each with a dimension of 768. Based on the visual representation information, the robot motion generation method determines environmental representation information, which represents the environmental context such as a white tabletop, side lighting, and background clutter. The method calculates the cross-modal association between the visual and instruction representation information, identifying 12 visual sub-representations with high relevance to "blue cup" and "yellow tray" from the 196 visual sub-representations, and generates object mediation representation information. The robot motion generation method determines motion mediation information based on instruction representation information. This mediation information represents the semantics of the "grasp-move-place" action. Subsequently, the method integrates visual representation information, environmental representation information, instruction representation information, object mediation representation information, and motion mediation representation information to generate robot motion data within the target prediction time domain. The target prediction time domain can include at least one target control cycle, the number and duration of which can be determined based on the target robot's control frequency. For example, when the target prediction time domain is 1000 milliseconds, for a type I robot, the method can generate robot motion data for the next 10 target control cycles, each cycle being 100 milliseconds; for a type II robot, the method can generate robot motion data for the next 30 target control cycles, each cycle being approximately 33.3 milliseconds. The robot motion data includes the end effector's position increment in three dimensions, end effector attitude increment, and gripper opening / closing commands. Based on the robot motion data, the robot controls its robotic arm to approach the blue cup, close the gripper, move above the yellow tray, and open the gripper, thereby completing the target operation task.

[0045] In some embodiments, the step of extracting features based on the visual observation information and the task instruction information to obtain visual representation information and instruction representation information may include: visually encoding visual data from at least one perspective in the visual observation information to obtain the visual representation information; and text encoding the task instruction information to obtain the instruction representation information.

[0046] Feature extraction refers to the process of converting visual observation information and task instruction information from raw data form into feature data form. Visual observation information can be represented as images, video frames, or multi-view image sequences, while task instruction information can be represented as natural language text, speech recognition text, or structured task descriptions. Through feature extraction, pixel information, spatial structure information, and scene semantic information in visual observation information can be converted into visual representation information, and word information, semantic relationships, and task intent in task instruction information can be converted into instruction representation information. This provides a unified feature foundation for the subsequent generation of environmental representation information, object mediation representation information, action mediation representation information, and robot motion data.

[0047] Visual data from at least one perspective can refer to visual data from one or more image acquisition perspectives. At least one perspective can include the robot's head view, robot wrist view, robot chest view, workbench top-down view, external camera view, etc. Visual data from one perspective can provide local or overall visual information about the scene where the target operation task is located. Visual data from multiple perspectives can provide information about the task object, background area, occlusion relationships, and spatial relationships from different directions, thereby improving the completeness of the visual observation information in representing the target operation task scene. For example, the robot's head view can be used to observe the overall layout of the tabletop, while the robot's wrist view can be used to observe the details of the task object near the gripper.

[0048] Visual encoding refers to a process of feature transformation of visual data from at least one perspective in visual observation information. Visual encoding can be implemented using methods such as image block encoding, region feature encoding, multi-view feature encoding, or temporal image feature encoding. Image block encoding divides visual data into multiple image blocks and extracts visual sub-representations for each block; region feature encoding extracts visual sub-representations for task object regions, background regions, or candidate operation regions; multi-view feature encoding encodes visual data from different perspectives and combines or aligns the visual sub-representations corresponding to different perspectives. Through visual encoding, visual data can be transformed into visual representation information that is convenient for subsequent computation.

[0049] Text encoding refers to a processing method that transforms the semantic features of task instruction information. Text encoding can include word segmentation, word embedding, semantic encoding, contextual semantic modeling, or structured field encoding. When the task instruction information is natural language text, text encoding can convert words, phrases, or semantic fragments into instruction sub-representations. When the task instruction information is a structured task description, text encoding can convert operation object fields, action fields, target location fields, and constraint fields into corresponding instruction sub-representations. Through text encoding, task instruction information can be converted into instruction representation information that is easy to correlate and calculate with visual representation information.

[0050] This embodiment visually encodes visual data from at least one perspective in the visual observation information, converting scene information, task object information, and environmental context information from different perspectives into visual representation information, thereby improving the understanding of the visual scene in the subsequent action generation process. By text-encoding the task instruction information, the object semantics, action semantics, and target position semantics in the task instruction information are converted into instruction representation information, thereby improving the semantic understanding of the target operation task in the subsequent action generation process. By obtaining visual representation information and instruction representation information separately, a feature basis can be provided for cross-modal association between visual representation information and instruction representation information, thereby improving the matching accuracy between task instruction information and task objects in the visual scene. By converting visual observation information and task instruction information into computable representation information respectively, the impact of original image noise, language expression differences, and multi-view differences on robot motion data generation can be reduced. This embodiment addresses the problem that original visual data and original instruction data are difficult to directly and collaboratively participate in action generation in related robot motion generation methods. It adopts the concept of visual encoding and text encoding to extract visual representation information and instruction representation information respectively, thereby providing a stable feature basis for the generation of environmental representation information, object-mediated representation information, action-mediated representation information, and robot motion data.

[0051] In one example, the robot motion generation method receives visual observation information and task instruction information corresponding to the target operation task. The visual observation information includes global view images captured by the robot's head camera and local view images captured by the robot's wrist camera, each image being 224×224 pixels. The task instruction information is "Place the blue cup into the yellow tray." The robot motion generation method visually encodes the global view images captured by the head camera, obtaining 196 global view visual sub-representations; it also visually encodes the local view images captured by the wrist camera, obtaining 196 local view visual sub-representations; each visual sub-representation has a dimension of 768. The robot motion generation method can use 392 visual sub-representations as visual representation information. The robot motion generation method textually encodes the task instruction information, dividing it into semantic units such as "will," "blue," "cup," "place," "yellow," and "tray," obtaining 6 instruction sub-representations, each with a dimension of 768. Thus, the robot motion generation method obtains visual representation information and instruction representation information. The visual representation information can be used to represent the spatial relationships between the blue cup, the yellow tray, the table background, and the camera viewpoint, while the instruction representation information can be used to represent the task object, action semantics, and target position.

[0052] In some embodiments, determining environmental representation information for representing the environmental context in the visual observation information based on the visual representation information may include: performing cross-attention processing on the visual representation information based on query information to obtain initial environmental representation information; performing environmental reconstruction processing on the initial environmental representation information based on candidate context representation information to obtain reconstructed environmental representation information; and determining the environmental representation information based on the reconstructed environmental representation information.

[0053] Query information refers to query feature information used to extract context-relevant features from visual representation information. Query information can be pre-defined query vectors, learnable query vectors, query representations corresponding to background regions, query representations corresponding to lighting or texture, query representations corresponding to non-task object regions, etc. In cross-attention processing, query information can serve as an index for actively retrieving context-relevant features, enabling robot motion generation methods to filter context-relevant features from visual representation information.

[0054] Cross-attention processing refers to a method of extracting feature information related to the query information from the visual representation information based on the matching relationship between query information and visual representation information. In cross-attention processing, the query information can be used as the query term, and the visual representation information as the key and value terms. Attention weights are calculated between the query information and multiple visual sub-representations in the visual representation information, and then the visual sub-representations are weighted according to these attention weights. Thus, cross-attention processing can obtain initial environmental representation information related to the environmental context, such as background, lighting, texture, and camera viewpoint, from the visual representation information.

[0055] Initial environmental representation information refers to the preliminary features of the environmental context obtained after cross-attention processing of visual representation information through query information. This initial environmental representation information can include features related to the environmental context from visual observation information, such as texture features of the background region, desktop color features, lighting variation features, and appearance changes caused by camera viewpoint. As input for subsequent environmental reconstruction processing, this initial environmental representation information provides a foundation for determining more stable environmental representation information.

[0056] Candidate context representation information refers to candidate feature information used to represent different environmental context types or environmental components. Candidate context representation information can include multiple candidate context representations, each corresponding to an environmental context mode, such as a strong light environment representation, a weak light environment representation, a wood grain tabletop representation, a white tabletop representation, a cluttered background representation, a clean background representation, a close-up view representation, or a top-down view representation. Candidate context representation information can provide a selectable and combinable foundation of environmental features for environmental reconstruction processing, enabling the initial environmental representation information to be mapped to a relatively stable environmental context expression space.

[0057] Environment reconstruction processing refers to the process of reconstructing initial environment representation information based on candidate context representation information. Environment reconstruction processing may include steps such as similarity calculation, matching weight determination, and weighted fusion of candidate context representations. For example, the degree of matching between the initial environment representation information and multiple candidate context representations can be calculated, and the multiple candidate context representations can be weighted and combined according to the degree of matching to obtain the reconstructed environment representation information. Through environment reconstruction processing, complex and noisy environment context features can be transformed into reconstructed features obtained by combining candidate context representations, thereby improving the stability of environment context representation.

[0058] Reconstructed environmental representation information refers to the reconstructed environmental context features obtained after reconstructing the initial environmental representation information. Reconstructed environmental representation information can be used to represent the combined relationship between the environmental context and multiple candidate context representations in visual observation information. For example, reconstructed environmental representation information can represent that the current visual observation information simultaneously possesses environmental context features corresponding to "white desktop," "side lighting," and "background clutter." Compared to the initial environmental representation information, reconstructed environmental representation information can reduce the impact of local noise, accidental textures, and image detail variations on the representation of the environmental context.

[0059] Environmental characterization information can be obtained directly from reconstructed environmental characterization information, or by normalizing, projecting, adjusting dimensions, or converting the format of the reconstructed environmental characterization information.

[0060] This embodiment improves the accuracy of environmental context feature extraction by performing cross-attention processing on visual representation information based on query information, selectively extracting initial environmental representation information relevant to the environmental context. By reconstructing the initial environmental representation information based on candidate context representation information, environmental factors such as background, lighting, texture, and camera viewpoint in complex visual scenes can be transformed into more stable reconstructed environmental representation information, thereby reducing the impact of local noise and accidental background details on the environmental representation. By determining the environmental representation information based on the reconstructed environmental representation information, the environmental context can participate in subsequent action generation as an independent feature factor, thus reducing the over-reliance of robot action generation methods on a single background, fixed lighting, or specific viewpoint. This embodiment addresses the problem that some robot action generation methods tend to establish a superficial correlation between environmental context and task actions, leading to insufficient stability in action generation when the environment changes. It adopts the concept of extracting environmental context from query information and reconstructing the environment based on candidate context representation information, thereby improving the stability of environmental representation information and reducing the impact of visual confusion factors on the robot action generation results.

[0061] In one example, the robot motion generation method obtains visual representation information comprising 196 visual sub-representations, each with a dimension of 768. The method sets eight query parameters, each with a dimension of 768, and performs cross-attention processing on the 196 visual sub-representations based on these eight parameters, resulting in eight initial environment sub-representations. These eight initial environment sub-representations collectively constitute the initial environment representation information. The candidate context representation information includes 16 candidate context representations, each with a dimension of 768. These 16 candidate context representations are used to represent environmental contexts such as a white desktop, a wood-grain desktop, a strong light environment, a weak light environment, a cluttered background, and a clean background, respectively. The method calculates the matching degree between the initial environment representation information and the 16 candidate context representations, and performs weighted fusion on the candidate context representations with high matching degrees to obtain the reconstructed environment representation information. Furthermore, the robot motion generation method determines environmental representation information based on reconstructed environmental representation information. This environmental representation information is used to represent the white desktop, side lighting, and background clutter in the current visual observation information, and participates in the subsequent generation of robot motion data together with visual representation information, instruction representation information, object mediation representation information, and motion mediation representation information.

[0062] In some embodiments, determining object mediation information based on the cross-modal association between the visual representation information and the instruction representation information may include: performing projection processing on the visual representation information and the instruction representation information respectively to obtain visual projection information and instruction projection information; determining cross-modal matching information based on the visual projection information and the instruction projection information; and determining the object mediation information based on the cross-modal matching information.

[0063] Projection processing refers to the process of transforming visual representation information and instruction representation information into a comparable common feature space. Since visual representation information originates from visual observation information, and instruction representation information originates from task instruction information, their data sources, semantic granularity, and numerical distributions in the original feature space may differ. Therefore, projection processing can be performed on visual representation information and instruction representation information respectively through methods such as linear mapping, multilayer perceptual mapping, attention mapping, and normalization mapping. Through projection processing, visual content such as object regions, background regions, and spatial locations in visual representation information can have a computable matching basis with linguistic content such as object words, action words, and target location words in instruction representation information.

[0064] Visual projection information refers to visual feature information obtained after projecting visual representation information. Visual projection information can include multiple visual projection sub-information, each corresponding to an image patch, object region, candidate operation region, or spatial location region in the visual observation information. Visual projection information is used for cross-modal matching with instruction projection information to determine which regions in the visual observation information are related to the task object, target location, or object attribute in the task instruction information. For example, visual projection information can be used to represent visual content such as the blue cup region, the yellow tray region, the tabletop region, and the region near the end effector of the robotic arm.

[0065] Instruction projection information refers to linguistic feature information obtained after projecting instruction representation information. Instruction projection information can include multiple instruction projection sub-information, each corresponding to words, phrases, object semantics, attribute semantics, action semantics, or target location semantics in the task instruction information. For example, in the task instruction information "Place the blue cup on the yellow tray," the instruction projection information can include instruction projection sub-information corresponding to semantic content such as "blue," "cup," "place," "yellow," and "tray." Instruction projection information is used to match with visual projection information to associate the object semantics in the task instruction information with the specific task object in the visual observation information.

[0066] Cross-modal matching information refers to the matching relationship between visual projection information and instruction projection information. Cross-modal matching information can be determined through methods such as similarity calculation, dot product calculation, cosine distance calculation, attention weight calculation, and relevance matrix calculation. Cross-modal matching information can represent the degree of matching between each visual projection sub-information and each instruction projection sub-information. For example, the visual projection sub-information corresponding to the blue cup area in the visual observation information may have a high degree of matching with the instruction projection sub-information corresponding to "blue" and "cup" in the task instruction information; conversely, the visual projection sub-information corresponding to the desktop background area in the visual observation information may have a low degree of matching with the instruction projection sub-information corresponding to "cup" in the task instruction information.

[0067] This embodiment improves the computability of cross-modal matching by projecting visual and command representation information separately, transforming visual and linguistic features into a comparable common feature space. By determining cross-modal matching information based on visual and command projection information, the correspondence between visual regions and command semantics can be obtained, improving the alignment accuracy between task command information and visual observation information. By determining object mediation representation information based on cross-modal matching information, task objects in task command information can be anchored to specific objects or regions in visual observation information, reducing motion generation deviations caused by unstable task object localization. By using object mediation representation information in subsequent robot motion data generation, robot motion data can be generated around task objects associated with the target operation task, improving the matching degree between robot motion data and the target operation task. This embodiment addresses the problem that related robot motion generation methods are easily affected by linguistic expression and visual scene complexity, leading to unstable task object associations. It employs the concept of projecting and matching visual and command representation information to generate object mediation representation information, thereby improving the stability of task object localization and reducing the impact of task object recognition deviations on robot motion generation results.

[0068] In one example, the robot motion generation method obtains visual representation information and instruction representation information. The visual representation information includes 196 visual sub-representations, each with a dimension of 768; the instruction representation information includes 8 instruction sub-representations, each with a dimension of 768. The task instruction is "Place the blue cup in the yellow tray." The robot motion generation method projects the 196 visual sub-representations, obtaining 196 visual projection sub-information pieces, each with a dimension of 256; the method also projects the 8 instruction sub-representations, obtaining 8 instruction projection sub-information pieces, each with a dimension of 256. The method calculates the matching degree between the 196 visual projection sub-information pieces and the 8 instruction projection sub-information pieces, obtaining 196×8 cross-modal matching information. Based on the cross-modal matching information, the method identifies 12 visual sub-representations with high relevance to "blue," "cup," "yellow," and "tray," and generates object mediation representation information based on these 12 visual sub-representations. Object mediation representation information is used to represent the spatial relationships between the blue cup, the yellow tray, and the blue cup and the yellow tray, and is used for subsequent robot motion data generation.

[0069] In some embodiments, determining the action mediation representation information based on the instruction representation information may include: performing global representation processing on the instruction representation information to obtain global instruction information; determining action distribution information based on the global instruction information; and performing projection processing on the action distribution information to obtain the action mediation representation information.

[0070] Global representation processing refers to the process of aggregating multiple instruction sub-representations from instruction representation information to obtain a representation of the overall semantics of the task instruction information. Global representation processing can include average pooling, max pooling, weighted pooling, attention-based aggregation, or feature concatenation followed by mapping. For example, for instruction representation information containing multiple instruction sub-representations, average pooling can be performed on these sub-representations to obtain a single overall semantic feature; alternatively, different weights can be assigned based on the importance of action words, object words, or target location words to obtain an overall semantic feature more biased towards action semantics. Through global representation processing, the task semantics scattered across multiple instruction sub-representations can be integrated into global instruction information.

[0071] Global instruction information refers to the overall semantic information obtained after globally representing instruction representation information. Global instruction information can be used to represent the overall operational intent, action type, action goal, and task context expressed by task instruction information. For example, in the task instruction information "Place the blue cup on the yellow tray," the global instruction information can include not only the object semantics of "blue cup" and "yellow tray," but also the action semantics corresponding to "place" and the task trend of "moving from the cup position to the tray position." Global instruction information provides overall semantic conditions for determining subsequent action distribution information.

[0072] Action distribution information refers to distributional information determined based on global instruction information, used to characterize the possible range of actions, trends of action changes, or implicit states of actions corresponding to a target operation task. Action distribution information can include the distribution of latent action variables, action mean information, action variance information, action stage probability information, action trajectory trend information, or action category probability information, etc. For example, for a target operation task of "grasping and placing," action distribution information can characterize that the robot's actions are more likely to include action stages such as approaching, gripping, lifting, moving, and releasing; for a target operation task of "pushing away an object," action distribution information can characterize that the robot's actions are more likely to include action stages such as contacting, applying force, and moving along a specified direction.

[0073] Determining action distribution information based on global instruction information refers to identifying distribution-type feature information that characterizes the possible range, trend, or implicit state of actions corresponding to a target operation task, based on the overall task semantics contained in the global instruction information. Specifically, global instruction information can be input into a distribution parameter generation network to obtain the mean, variance, standard deviation, or covariance information corresponding to the latent variables of the actions; it can also be input into an action phase prediction network to obtain probability information corresponding to action phases such as approaching, grasping, lifting, moving, and releasing; or it can be input into a trajectory pattern prediction network to obtain weight information corresponding to different candidate action trajectory patterns. Action distribution information can be the distribution parameters themselves, or it can be latent action features, action trend features, or action phase probability features determined based on the distribution parameters. In some implementations, the action generation process can directly use the mean information or the action pattern with the highest probability from the action distribution information; in other implementations, sampling or weighted fusion can be performed based on the action distribution information to obtain action-level features for subsequent projection processing. Therefore, motion distribution information is not limited to the final robot motion data, but is used to convert the motion intent in the global instruction information into intermediate features suitable for generating motion mediator representation information.

[0074] Projection processing refers to the feature mapping process that transforms action distribution information into action-mediated representation information. Projection processing can be implemented using methods such as linear mapping, multilayer perceptual mapping, normalized mapping, dimensionality transformation, or feature space alignment. Since action distribution information can be represented as distribution parameters, latent variables, or action trend features, projection processing can transform it into a representation form that is easily integrated with visual representation information, environmental representation information, instruction representation information, and object-mediated representation information. Through projection processing, the usability of action distribution information in subsequent action generation processes can be improved.

[0075] This embodiment improves the overall semantic integrity of task instruction information by performing global representation processing on instruction representation information, which aggregates multiple semantic fragments into global instruction information. By determining action distribution information based on the global instruction information, the action intent, action type, and action trend in the task instruction information can be converted into action-level distribution information, thereby improving the robot's ability to perceive action semantics in action data generation. Projection processing of the action distribution information converts it into action mediation representation information that is easy to integrate and use, thus improving the utilization efficiency of action distribution information in subsequent robot action data generation. By involving action mediation representation information in subsequent robot action data generation, robot action data can better match the action semantics or action distribution features associated with the target operation task, thereby improving the stability and task adaptability of robot action generation. This embodiment addresses the problem that some robot action generation methods tend to directly map the expression form in task instruction information to action results, leading to unstable action semantic extraction. It adopts the concept of performing global representation processing on instruction representation information, determining action distribution information, and projecting it into action mediation representation information, thereby enhancing the action semantic expression capability and reducing the impact of language expression deviation on the robot action generation results.

[0076] In one example, the robot motion generation method obtains instruction representation information comprising eight instruction sub-representations, each with a dimension of 768. The task instruction information is "Place the blue cup into the yellow tray." The robot motion generation method performs average pooling on the eight instruction sub-representations to obtain one global instruction information, also with a dimension of 768. Based on this global instruction information, the method determines motion distribution information, which can include 64-dimensional motion mean information and 64-dimensional motion variance information, used to represent the "grasp-move-place" motion trend. Subsequently, the method projects the motion distribution information, mapping the 128-dimensional motion distribution information to 768-dimensional motion mediation representation information. This mediation representation information is used to represent the semantic or distribution features of actions in the target operation task and participates in subsequent robot motion data generation along with visual representation information, environmental representation information, instruction representation information, and object mediation representation information.

[0077] In some embodiments, generating robot motion data based on the visual representation information, the environmental representation information, the instruction representation information, the object mediation representation information, and the action mediation representation information may include: fusing the visual representation information, the environmental representation information, the instruction representation information, the object mediation representation information, and the action mediation representation information to obtain motion generation condition information; and generating motion based on the motion generation condition information to obtain robot motion data of the robot within at least one target control cycle.

[0078] Fusion processing refers to the joint processing of visual representation information, environmental representation information, instruction representation information, object-mediated representation information, and action-mediated representation information to form a unified basis for action generation. Fusion processing can include splicing, weighted fusion, attention-based interaction, projection alignment, feature normalization, or multi-layer feature mapping. Through fusion processing, visual scene information, environmental context information, task instruction semantics, task object information, and action semantics can interact within the same feature space, thereby providing more complete conditional information for subsequent action generation.

[0079] Action generation condition information refers to the conditional feature information obtained after fusion processing, used to support robot action generation. Action generation condition information can simultaneously include information from visual observation dimensions, environmental context dimensions, linguistic semantic dimensions, object mediation dimensions, and action mediation dimensions. For example, action generation condition information can represent comprehensive conditions such as "the blue cup is on the left side of the table," "the yellow tray is on the right side of the table," "the current visual scene has lateral lighting and background clutter," "the target operation task is placement," and "the action trend is grasping, moving, and releasing." Action generation condition information serves as the data foundation for subsequent action generation.

[0080] Motion generation refers to the process of determining robot motion data within at least one target control cycle based on motion generation condition information. Motion generation can be implemented using methods such as sequence prediction, trajectory prediction, motion classification, control variable regression, cycle-by-cycle motion estimation, or multi-step motion planning. In different implementations, the motion generation process can directly output data conforming to the robot control interface, or it can first obtain intermediate calculation results and then convert them into robot motion data according to the target control cycle, robot control interface, or motion data format. Through motion generation, the conditional information after multimodal fusion can be converted into motion data that can be executed by the robot.

[0081] A target control cycle can refer to a control time unit used to generate or execute robot motion data. The target control cycle can be determined based on the robot's control frequency, such as 10 milliseconds, 20 milliseconds, 50 milliseconds, or 100 milliseconds. At least one target control cycle can represent a single control cycle or multiple consecutive control cycles. By generating robot motion data within at least one target control cycle, the robot motion generation method can support single-step control as well as multi-step motion sequence prediction, thereby improving the continuity of robot motion execution.

[0082] In some embodiments, the number of target control cycles and the duration of each target control cycle can be determined based on the target robot's control frequency, control interface requirements, motion prediction time domain, or actuator response characteristics. In other words, the amount of robot motion data generated by the robot motion generation method and the corresponding time intervals between adjacent robot motion data can vary depending on the robot platform.

[0083] This embodiment integrates visual representation information, environmental representation information, instruction representation information, object-mediated representation information, and action-mediated representation information to enable robot motion generation methods to simultaneously utilize visual scenes, environmental context, task instruction semantics, task objects, and action semantics, thereby improving the completeness of motion generation condition information. By generating motion based on action generation condition information, the multimodal fusion results can be converted into robot motion data, thus improving the matching degree between robot motion data and target operation tasks. By generating robot motion data within at least one target control cycle, the robot motion data can have temporal continuity and execution availability, thereby improving the stability of robot motion execution. By jointly introducing environmental representation information, object-mediated representation information, and action-mediated representation information into the robot motion data generation process, the impact of changes in environmental context and language expression on robot motion data generation can be reduced. This embodiment addresses the problem that related robot motion generation methods are prone to motion deviations when the visual environment changes, task object positioning is unstable, or action semantic expression is insufficient. It adopts the concept of fusing multiple types of representation information before generating robot motion data, thereby improving the robustness, continuity, and task adaptability of robot motion generation.

[0084] In one example, the robot motion generation method obtains visual representation information, environmental representation information, instruction representation information, object mediation representation information, and action mediation representation information, each with a dimension of 768. The method concatenates these five types of representation information to obtain a 3840-dimensional concatenated feature, and then projects this 3840-dimensional feature to obtain 768-dimensional motion generation condition information. The target task is "to place the blue cup into the yellow tray." Based on this 768-dimensional motion generation condition information, the method generates motion data for the next 10 target control cycles, each 20 milliseconds long. The motion data for each target control cycle includes the end effector's position increment in three dimensions, the end effector's attitude increment, and the gripper opening / closing command. The robot controls its robotic arm to move continuously based on the motion data from the 10 target control cycles, thus completing the target task of placing the blue cup into the yellow tray.

[0085] In a specific application scenario, the robot motion generation method can be applied to situations where a robot performs desktop operation tasks based on visual observation information and task instruction information. The robot may include a left-arm camera, a head camera, a right-arm camera, a robotic arm, and an end effector. The target operation task is "put the pen into the pen holder." At a target control moment, the robot motion generation method receives the left-arm view image captured by the left-arm camera, the head-view image captured by the head camera, and the right-arm view image captured by the right-arm camera, and receives the task instruction information "Put the pen into the pen holder." Each view image can be 224×224 pixels in size, and the task instruction information may include 8 text units.

[0086] The robot motion generation method uses a visual encoder to visually encode the left arm view image, the head top-down view image, and the right arm view image. Specifically, each view image can be divided into multiple image patches, and features are extracted from each patch separately. For example, each view image is divided into 196 image patches, and each patch corresponds to a 768-dimensional visual sub-representation, resulting in a total of 588 visual sub-representations across the three views. The robot motion generation method generates visual representation information based on these 588 visual sub-representations. This visual representation information can include visual elements such as a pen, pen holder, desktop, the area near the robotic arm's end effector, background objects, and differences in camera viewpoints.

[0087] The robot motion generation method uses a text encoder to encode task instruction information. For example, the task instruction "Put the pen into the pen holder" is encoded into eight instruction sub-representations, each with a dimension of 768. The robot motion generation method generates instruction representation information based on these eight instruction sub-representations. The instruction representation information can include the task object semantics corresponding to "pen," the action semantics corresponding to "put in," and the target position semantics corresponding to "pen holder."

[0088] The robot motion generation method determines environmental representation information based on visual representation information. Specifically, the method sets eight query messages, each with a dimension of 768, and performs cross-attention processing on 588 visual sub-representations based on these eight queries to obtain initial environmental representation information. This initial environmental representation information can represent environmental contexts such as desktop texture, lighting direction, background clutter, camera viewpoint, and non-task object areas. Further, the method reconstructs the initial environmental representation information based on 16 candidate context representation messages to obtain reconstructed environmental representation information. These 16 candidate context representation messages can represent environmental context types such as bright light environment, dark light environment, wood grain desktop, white desktop, cluttered background, clean background, close-up viewpoint, and top-down viewpoint. The method determines the environmental representation information based on the reconstructed environmental representation information, allowing this information to participate in subsequent fusion processing as the output of the visual backdoor learning branch.

[0089] The robot motion generation method determines object mediation representation information based on the cross-modal association between visual representation information and instruction representation information. Specifically, the method projects 588 visual sub-representations and 8 instruction sub-representations, mapping each to 256-dimensional projection information. Then, it calculates the matching degree between the 588 visual projection information and the 8 instruction projection information, obtaining 588×8 cross-modal matching information. Based on this cross-modal matching information, the method determines the correlation between each visual sub-representation and the task instruction information, and selects 16 target visual sub-representations with high correlation from the 588 visual sub-representations. These 16 target visual sub-representations primarily correspond to pens, pen holders, and the spatial relationship between pens and pen holders. Based on these 16 target visual sub-representations, the method generates object mediation representation information, which is used to represent the task objects associated with the target operation task.

[0090] The robot motion generation method determines intermediate motion representation information based on instruction representation information. Specifically, the method performs global representation processing on eight instruction sub-representations, such as average pooling, to obtain a 768-dimensional global instruction information. Based on this global instruction information, the method determines motion distribution information, which can include 64-dimensional motion mean information and 64-dimensional motion variance information, used to represent the motion trends of "approaching the pen, picking up the pen, moving above the pen holder, and releasing the pen." The method then projects this 128-dimensional motion distribution information into 768-dimensional intermediate motion representation information. This intermediate motion representation information is used to represent the semantic or distribution features of the actions associated with the target operation task.

[0091] The robot motion generation method fuses visual representation information, environmental representation information, instruction representation information, object-mediated representation information, and action-mediated representation information to obtain motion generation condition information. Specifically, the fusion process can jointly express the visual scene, environmental context, task instruction semantics, task object, and action semantics based on visual representation information, environmental representation information, instruction representation information, object-mediated representation information, and action-mediated representation information, outputting 768-dimensional motion generation condition information. The motion generation condition information can represent the position of the pen, the position of the pen holder, the desktop environment, background factors, the relationships between task objects, and the "placement" action trend in the current scene.

[0092] The robot motion generation method generates robot motion data for at least one target control cycle based on motion generation condition information. For example, the method generates robot motion data for the next 10 target control cycles, each target control cycle being 20 milliseconds. The robot motion data for each target control cycle can include the position increment of the end effector in three dimensions, the end effector attitude increment, and gripper opening and closing commands. The robot motion data for the first to third target control cycles is used to control the end effector to approach the pen; the robot motion data for the fourth target control cycle is used to control the gripper to close and pick up the pen; the robot motion data for the fifth to eighth target control cycles is used to control the end effector to move the pen above the pen holder; and the robot motion data for the ninth to tenth target control cycles is used to control the end effector to descend and open the gripper to allow the pen to enter the pen holder. Thus, the robot motion generation method completes the entire reasoning process from multi-view visual observation information and task command information to robot motion data.

[0093] In some embodiments, the robot motion data can be generated using a motion generation model.

[0094] In some embodiments, the training process of the action generation model may include: performing the following processes through the action generation model: receiving training samples, the training samples including training visual observation information, training task instruction information, and reference action data; extracting features from the training visual observation information and the training task instruction information respectively to obtain training visual representation information and training instruction representation information; determining training environment representation information based on the training visual representation information to represent the environmental context in the training visual observation information; determining training object mediation representation information based on the cross-modal association between the training visual representation information and the training instruction representation information, the training object mediation representation information being used to represent training task objects; determining training action mediation representation information based on the training instruction representation information, the training action mediation representation information being used to represent training action semantics or training action distribution features; and generating predicted action data based on the training visual representation information, the training environment representation information, the training instruction representation information, the training object mediation representation information, and the training action mediation representation information. The training process of the action generation model may further include: adjusting the model parameters of the action generation model based on the difference between the predicted action data and the reference action data to update the action generation model.

[0095] In some embodiments, determining training environment representation information for representing the environmental context in the training visual observation information based on the training visual representation information may include: performing cross-attention processing on the training visual representation information based on query information to obtain initial training environment representation information; performing environment reconstruction processing on the initial training environment representation information based on candidate context representation information to obtain reconstructed training environment representation information; determining intervention context representation information based on the candidate context representation information when environmental intervention conditions are met, and determining the training environment representation information based on the intervention context representation information and the reconstructed training environment representation information; or, determining the training environment representation information based on the reconstructed training environment representation information when the environmental intervention conditions are not met.

[0096] In some embodiments, the training samples may further include training robot state information.

[0097] In some embodiments, determining the intermediate representation information of training actions based on the training instruction representation information may include: performing global representation processing on the training instruction representation information to obtain global training instruction information; determining training action distribution information based on the global training instruction information; and performing projection processing on the training action distribution information to obtain the intermediate representation information of training actions.

[0098] In some embodiments, the training process of the action generation model may further include: performing action reconstruction processing based on the training action distribution information and the training robot state information to obtain reconstructed action data; and adjusting at least the model parameters in the action generation model used to determine the intermediate representation information of the training actions based on the difference between the reconstructed action data and the reference action data, so as to update the action generation model.

[0099] In some embodiments, the training process of the action generation model may further include: performing word masking or word discarding on the training task instruction information to obtain processed task instruction information; extracting features from the processed task instruction information to obtain processed instruction representation information; generating processed predicted action data based on the training visual representation information, the training environment representation information, the processed instruction representation information, the training object mediation representation information, and the training action mediation representation information; and adjusting the model parameters of the action generation model based on the difference between the processed predicted action data and the reference action data to update the action generation model.

[0100] In other embodiments, processed object mediation information and / or processed action mediation information may be re-determined based on processed instruction representation information, and processed predicted action data may be generated based on processed object mediation information and / or processed action mediation information.

[0101] In some embodiments, during the inference phase, when generating robot motion data through the motion generation model, no environmental intervention processing, word masking processing, or word discarding processing is performed.

[0102] In this embodiment, to distinguish between the training and inference phases of the action generation model, the prefix "training" can be added before certain technical terms. Technical terms with the "training" prefix indicate that the corresponding object is used in the training phase of the action generation model and can serve as input, intermediate representation, or supervisory data during model parameter adjustment. Except for differences in data sources, sample organization methods, whether they participate in model parameter adjustment, and whether they are combined with training intervention strategies, the meaning, data structure, and processing logic of technical terms with the "training" prefix can be the same as or similar to their corresponding counterparts without the "training" prefix. For example, training visual observation information can be understood with reference to visual observation information; training task instruction information can be understood with reference to task instruction information; training visual representation information can be understood with reference to visual representation information; training instruction representation information can be understood with reference to instruction representation information; training environmental representation information can be understood with reference to environmental representation information; training object mediation representation information can be understood with reference to object mediation representation information; and training action mediation representation information can be understood with reference to action mediation representation information. In other words, the "training" prefix is ​​mainly used to distinguish between data in the training phase and data in the inference phase, and does not necessarily indicate that the corresponding technical terms are different in their representational meaning or processing logic.

[0103] A motion generation model (MGM) can be defined as a model used to generate robot motion data based on visual observation information and task instruction information. During the inference phase, the MGM can generate robot motion data based on visual representation information, environmental representation information, instruction representation information, object mediation representation information, and action mediation representation information. During the training phase, the MGM can generate predicted motion data (i.e., an abbreviation for predicted robot motion data) based on training samples, and adjust model parameters based on the differences between the predicted motion data and reference motion data (i.e., an abbreviation for reference robot motion data) to update the MGM. The MGM can include components for extracting representation information, determining environmental representation information, determining object mediation representation information, determining action mediation representation information, and generating motion data.

[0104] Training samples refer to data samples used to train a motion generation model. Training samples can include training visual observation information, training task instruction information, and reference motion data, and may also include training robot state information. Training visual observation information and training task instruction information serve as input data during the training phase of the motion generation model, reference motion data serves as a supervisory benchmark for predicted motion data, and training robot state information can be used for motion reconstruction processing. Training samples can originate from human instruction, robot historical successful execution records, simulation environment data collection, manual annotation, or task data obtained through multi-source data fusion.

[0105] Reference motion data refers to the expected motion result corresponding to the training visual observation information and training task instruction information, used as a supervisory benchmark for predicted motion data during the training of the motion generation model. Reference motion data can originate from manual teleoperation teaching, expert trajectory acquisition, robot historical successful execution records, task trajectories generated in a simulation environment, or manually annotated motion sequences. Reference motion data may include end effector position increments, end effector posture increments, target end effector pose, joint angles, joint velocities, joint torques, gripper opening / closing commands, or motion stage markers within at least one control cycle. By calculating the difference between the predicted motion data and the reference motion data, the deviation of the motion result output by the motion generation model from the expected motion result can be determined. Based on this deviation, the model parameters of the motion generation model can be adjusted, enabling the motion generation model to gradually learn the correspondence between training visual observation information, training task instruction information, and robot motion data.

[0106] Environmental intervention conditions refer to the conditions used to determine whether to perform environmental intervention processing on the reconstructed environmental representation information during training. Environmental intervention conditions can include random probability conditions, training round conditions, sample type conditions, environmental category conditions, visual background complexity conditions, or training stage conditions. For example, the environmental intervention probability can be set to 30%, and the environmental intervention condition is considered met when the random number corresponding to the current training sample is less than 0.3; alternatively, the environmental intervention condition can be considered met when the training visual observation information belongs to cluttered background samples, strong light samples, or samples where the target object is partially occluded. Through environmental intervention conditions, the triggering timing of environmental intervention processing during the training stage can be controlled.

[0107] Intervention contextual representation information refers to the contextual feature information determined based on candidate contextual representation information and used in the process of determining training environment representation information, under the condition that environmental intervention conditions are met. Intervention contextual representation information can be randomly selected from candidate contextual representation information, or determined from candidate contextual representation information according to environment category, matching degree, training strategy, or preset sampling rules. Intervention contextual representation information can be replaced, fused, weighted fused, or perturbed with training reconstructed environment representation information, so that the training environment representation information does not completely depend on the real background features in the training visual observation information.

[0108] Motion reconstruction processing refers to the process of generating reconstructed motion data based on training motion distribution information and training robot state information. Training motion distribution information can provide the motion trend, motion stage, or potential motion distribution corresponding to the training task, while training robot state information can provide the robot's body state at the corresponding moment in the training sample, such as joint angles, joint velocities, end effector pose, and gripper state. Through motion reconstruction processing, the training motion distribution information and training robot state information can be combined to generate reconstructed motion data that matches the training task and the robot's current state.

[0109] Reconstructed motion data refers to motion data obtained through motion reconstruction processing. Reconstructed motion data can include end-effector position increments, end-effector posture increments, gripper opening and closing commands, joint angles, joint velocities, or motion phase information within at least one control cycle. Reconstructed motion data is used to calculate differences with reference motion data to determine whether the training motion distribution information and the intermediate representation information of the training motion can adequately express the motion patterns corresponding to the training task. By adjusting the model parameters used to determine the intermediate representation information of the training motion based on the differences between the reconstructed motion data and the reference motion data, the expressive power of the intermediate representation information of the training motion for motion semantics or motion distribution features can be enhanced.

[0110] Lexical masking or lexical discarding refers to the process of masking or deleting certain words in the training task instruction information during the training of an action generation model. Lexical masking replaces some words with preset labels, while lexical discarding removes some words from the training task instruction information. The words to be processed can be determined based on random probability, part-of-speech information, word position, semantic type, or training strategy. For example, masking or discarding can be performed on modifiers, function words, some object words, or some action words. Lexical masking or discarding can reduce the excessive reliance of the action generation model on specific expressions in the training task instruction information.

[0111] Post-processed task instruction information refers to task instruction information obtained after lexical masking or lexical discarding of training task instruction information. Post-processed task instruction information retains some semantic content of the training task instruction information while changing its expression. For example, "put the blue cup into the yellow tray" becomes "put [masked] cup into the yellow tray" after lexical masking, and "put the blue cup into the tray" after lexical discarding. Post-processed task instruction information is used to train the action generation model to still generate action results that match the reference action data even when the language expression is incomplete or changed.

[0112] Post-processed instruction representation information refers to semantic feature information obtained by feature extraction from processed task instruction information. The difference between post-processed instruction representation information and training instruction representation information is that post-processed instruction representation information originates from task instruction information that has undergone lexical masking or lexical discarding. Post-processed instruction representation information can be used to participate in the generation of processed predicted action data, enabling the action generation model to learn to generate action data based on perturbed language expressions during the training phase.

[0113] Post-processed predicted action data refers to action results generated based on training visual representation information, training environment representation information, post-processed instruction representation information, training object mediated representation information, and training action mediated representation information. Post-processed predicted action data is used to calculate differences with reference action data to adjust the model parameters of the action generation model. By analyzing the differences between post-processed predicted action data and reference action data, the action generation model can be constrained to still generate action results close to the reference action data even when lexical masking or word discarding occurs in the task instruction information.

[0114] Model parameters refer to adjustable parameters in an action generation model that affect the predicted action data or the processed predicted action data. Model parameters may include feature extraction parameters, environmental representation information determination parameters, object mediation representation information determination parameters, action mediation representation information determination parameters, feature fusion parameters, and action data generation parameters. Adjusting model parameters can refer to updating at least some of the model parameters in the action generation model based on the differences between the predicted action data and reference action data, the differences between the reconstructed action data and reference action data, or the differences between the processed predicted action data and reference action data. This updates the action generation model to generate action data that more closely approximates the target task during subsequent training or inference.

[0115] In the training process of the action generation model in this embodiment, after receiving training samples, the action generation model extracts features from the training visual observation information and the training task instruction information to obtain training visual representation information and training instruction representation information, respectively. Then, based on the training visual representation information, training environment representation information is determined; based on the cross-modal correlation between the training visual representation information and the training instruction representation information, training object intermediate representation information is determined; and based on the training instruction representation information, training action intermediate representation information is determined. Further, predicted action data is generated based on the training visual representation information, training environment representation information, training instruction representation information, training object intermediate representation information, and training action intermediate representation information. The model parameters of the action generation model are adjusted based on the difference between the predicted action data and the reference action data to update the action generation model. During training, if environmental intervention conditions are met, training environment representation information can also be determined based on intervention context representation information and training reconstructed environment representation information. Alternatively, action reconstruction processing can be performed based on training action distribution information and training robot state information, and the model parameters used to determine the training action intermediate representation information can be updated based on the difference between the reconstructed action data and the reference action data. It can also perform word masking or word discarding on the training task instruction information, generate processed predicted action data based on the processed instruction representation information, and update the action generation model based on the difference between the processed predicted action data and the reference action data.

[0116] This embodiment determines the training environment representation information based on training visual representation information, which can explicitly introduce environmental context features during the training phase, thereby reducing the excessive dependence of the action generation model on fixed backgrounds, specific lighting, or specific textures. By determining the training environment representation information based on intervention context representation information and training reconstructed environment representation information under the condition of environmental intervention, the variability of the environmental context during the training phase can be increased, thereby improving the generalization ability of the action generation model in the face of environmental changes. By determining the intermediate representation information of the training object based on the cross-modal association between training visual representation information and training instruction representation information, the correspondence stability between the training task object and the training visual observation information can be improved, thereby reducing the impact of task object localization deviation on model parameter updates. By determining the intermediate representation information of training actions based on training instruction representation information, the ability of the training process to express action semantics or action distribution features can be enhanced, thereby improving the matching degree between predicted action data and training tasks. By performing action reconstruction processing based on training action distribution information and training robot state information, the training action distribution information can be linked with the robot's body state, thereby improving the ability of the intermediate representation information of training actions to express action patterns in reference action data. By applying lexical masking or word discarding to the training task instruction information, the over-reliance of the action generation model on specific linguistic expressions can be reduced, thereby improving the stability of the action generation model in the face of changes in linguistic expression. This embodiment addresses the problem of easily learning superficial correlations between visual background, linguistic expression, and reference actions during the training of related robot action generation. It employs a concept that uses training environment representation information, intermediate representation information of training objects, intermediate representation information of training actions, environmental intervention, action reconstruction, and lexical masking or word discarding to jointly constrain model parameter updates, thereby improving the generalization ability of the action generation model in the face of changes in visual environment and linguistic expression.

[0117] In one example, the training samples include two training visual observation images, a training task instruction, a set of reference motion data, and training robot state information. The two training visual observation images are from the robot's head camera and wrist camera, respectively, and both images are 224×224 pixels in size. The training task instruction is "Place the blue cup into the yellow tray." The reference motion data includes the end effector position increment, end effector pose increment, and gripper opening / closing instructions for the next 10 control cycles, with each control cycle being 20 milliseconds. The training robot state information includes 7 joint angles, 7 joint velocities, the end effector's 3D position, the end effector's quaternion pose, and the gripper's opening / closing state, totaling 22 dimensions. The motion generation model extracts features from the two training visual observation images, obtaining 392 training visual sub-representations, each with a dimension of 768; it also extracts features from the training task instruction, obtaining 8 training instruction sub-representations, each with a dimension of 768. The action generation model determines 768-dimensional training environment representation information based on 392 training visual sub-representations, 768-dimensional training object mediation representation information based on cross-modal correlations between the 392 training visual sub-representations and 8 training instruction sub-representations, and 768-dimensional training action mediation representation information based on the 8 training instruction sub-representations. The action generation model generates predicted action data for the next 10 control cycles based on the training visual representation information, training environment representation information, training instruction representation information, training object mediation representation information, and training action mediation representation information, and adjusts the model parameters based on the differences between the predicted action data and the reference action data. Furthermore, when environmental intervention conditions are met, the action generation model can fuse the intervention context representation information and the training reconstructed environment representation information with a weight of 0.4:0.6 to obtain the training environment representation information. The action generation model can also generate reconstructed action data for the next 10 control cycles based on 128-dimensional training action distribution information and 22-dimensional training robot state information, and adjust the model parameters used to determine the intermediate representation information of training actions based on the difference between the reconstructed action data and the reference action data. The action generation model can also mask the "yellow" in the training task instruction information to obtain the processed task instruction information "Put the blue cup into the [masked] tray", and generate processed predicted action data based on the processed instruction representation information corresponding to the processed task instruction information, so as to continue to update the action generation model based on the difference between the processed predicted action data and the reference action data.

[0118] See Figure 2 , Figure 2 This is a schematic diagram of the structure of an action generation model provided in an embodiment of this application.

[0119] In a specific application scenario, robot motion data can be generated through a motion generation model. This model can include a visual encoding component, a text encoding component, an environmental representation branch, an object mediation representation branch, a motion mediation representation branch, a fusion processing component, and a motion generation component. Figure 2 This only shows one exemplary structure of the action generation model and does not constitute a limitation on the number of network layers, parameter scale, module connection method or specific implementation algorithm of the action generation model.

[0120] In one inference process example, the target operation task is "put the pen into the pen holder". The robot motion generation method receives visual observation information and task instruction information corresponding to the target operation task. The visual observation information can include visual data from the left arm perspective, head perspective, and right arm perspective, and the task instruction information can be "put the pen into the pen holder". The left arm perspective visual data can represent the local scene near the robot's left arm end effector, the head perspective visual data can represent the overall layout of the desktop, and the right arm perspective visual data can represent the local scene near the robot's right arm end effector.

[0121] The action generation model uses a visual encoding component to visually encode visual data from the left arm's perspective, the head's perspective, and the right arm's perspective, obtaining visual representation information. This visual representation information can include multiple visual sub-representations, which can respectively represent image patches, the task object region, the background region, the region near the robotic arm's end effector, the pen's location, the pen holder's location, and spatial relationships from different perspectives. The action generation model also uses a text encoding component to textually encode task instruction information, obtaining instruction representation information. This instruction representation information can include multiple instruction sub-representations, which can respectively represent semantic content such as "will," "pen," "place," and "pen holder."

[0122] In the environment representation branch, the action generation model performs cross-attention processing on the visual representation information based on query information to obtain initial environment representation information. The query information can be used to extract features related to the environment context from the visual representation information, such as desktop texture, background objects, lighting conditions, camera viewpoint, and non-task object regions. Subsequently, the action generation model performs environment reconstruction processing on the initial environment representation information based on candidate context representation information to obtain reconstructed environment representation information. Candidate context representation information can be used to represent different environment context types, such as a white desktop, a wood-grain desktop, a strong light environment, a low light environment, a cluttered background, or a simple background. The action generation model determines the environment representation information based on the reconstructed environment representation information, allowing the environment context to participate in subsequent fusion processing as an independent representation.

[0123] In the object mediation representation branch, the action generation model projects visual representation information and instruction representation information separately to obtain visual projection information and instruction projection information. Based on the visual projection information and instruction projection information, the action generation model determines cross-modal matching information and performs matching filtering based on the cross-modal matching information to obtain object mediation representation information. Object mediation representation information can be used to represent the task objects associated with the target operation task, such as a pen, a pen holder, and the spatial relationship between pens and pen holders. Through object mediation representation information, the object semantics in the task instruction information can establish a correspondence with specific visual regions in the visual observation information.

[0124] In the action mediation representation branch, the action generation model performs global representation processing on the instruction representation information to obtain global instruction information. This global instruction information represents the overall task semantics of the task instruction information. Based on this global instruction information, the action generation model determines action distribution information. This action distribution information can be used to represent the action semantics or action distribution features associated with the target operation task, such as action trends like approaching the pen, picking up the pen, moving above the pen holder, and releasing the pen. The action generation model then projects this action distribution information to obtain action mediation representation information, transforming it into a representation form suitable for fusion processing.

[0125] The action generation model fuses visual representation information, instruction representation information, environmental representation information, object mediation representation information, and action mediation representation information through a fusion processing component. Fusion processing can include stitching, attention interaction processing, weighted fusion processing, projection alignment processing, or feature mapping processing. Through fusion processing, the action generation model can simultaneously utilize visual scene information, linguistic semantic information, environmental context information, task object information, and action semantic information to obtain action generation conditional information.

[0126] The motion generation model generates motion based on motion generation condition information and outputs robot motion data. This data can include end-effector position increments, end-effector posture increments, joint control parameters, gripper opening / closing commands, or motion sequences within at least one target control cycle. For example, the motion generation model can generate robot motion data for the next 10 target control cycles, each lasting 20 milliseconds. Based on this motion data, the robot controls its robotic arm to first approach the pen, then close the gripper to pick it up, then move it above the pen holder, and finally open the gripper to allow the pen to enter the holder.

[0127] pass Figure 2The illustrated motion generation model structure allows for the fusion of visual representation information, environmental representation information, instruction representation information, object-mediated representation information, and action-mediated representation information before motion generation. This enables robot motion data to be generated not only based on direct representations of visual observation information and task instruction information, but also by incorporating environmental context, task objects, and action semantics. Consequently, the impact of environmental background, lighting, viewpoint changes, and differences in language expression on the robot's motion generation results can be reduced, improving the stability of motion generation in open operating scenarios.

[0128] See Figure 3 , Figure 3 This is a structural block diagram of a robot motion generation device provided in an embodiment of this application.

[0129] This application also provides a robot motion generation device, including a feature extraction module, an environment representation module, an object mediation representation module, a motion mediation representation module, and a motion generation module.

[0130] The feature extraction module is used to receive visual observation information and task instruction information corresponding to the target operation task, and to perform feature extraction based on the visual observation information and the task instruction information to obtain visual representation information and instruction representation information.

[0131] The environment characterization module is used to determine, based on the visual characterization information, environmental characterization information used to characterize the environmental context in the visual observation information.

[0132] The object mediation representation module is used to determine object mediation representation information based on the cross-modal association between the visual representation information and the instruction representation information. The object mediation representation information is used to represent the task object associated with the target operation task.

[0133] The action mediation representation module is used to determine action mediation representation information based on the instruction representation information. The action mediation representation information is used to represent the action semantics or action distribution features associated with the target operation task.

[0134] The motion generation module is used to generate robot motion data based on the visual representation information, the environmental representation information, the instruction representation information, the object mediation representation information, and the motion mediation representation information.

[0135] See Figure 4 , Figure 4 This is a structural block diagram of a robot provided in an embodiment of this application.

[0136] This application also provides a robot, which includes a control module for performing the steps of any of the above methods.

[0137] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0138] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0139] The computer program product may be a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the computer program product of the embodiments of this application is not limited thereto, and the computer program product may be any combination of one or more computer-readable media.

[0140] See Figure 5 , Figure 5 This is a structural block diagram of a computer device provided in an embodiment of this application.

[0141] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.

[0142] The embodiments of this application do not limit the computer device, which may be, for example, a local computer device, a cloud computer device, a distributed computer device, etc.

[0143] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected through internal connection paths.

[0144] The memory 110 is used to store computer programs, which in some implementations may include code for implementing the methods of the embodiments of this application.

[0145] The processor 120 executes the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information, and output operation results and other data. In some implementations, when the solutions of the embodiments of this application are implemented by software or firmware, the computer program used to implement the solutions of the embodiments of this application can be stored in the processor 120 and executed by the processor 120.

[0146] The memory 110 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any memory of these and other suitable types. As an example, the memory 110 includes random access memory (RAM), cache memory, and read-only memory (ROM). The memory 110 stores a computer program that can be executed by processor 120, causing processor 120 to implement the steps of any of the methods described above.

[0147] The processor 120 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 120 can be any conventional processor.

[0148] In implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware in the processor 120 or by instructions in software form. The method disclosed in the embodiments of this application can be directly implemented by the hardware processor, or by a combination of hardware and software modules in the processor 120. The software modules can be located in mature storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in the memory 110, and the processor 120 reads the information in the memory 110 and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are not provided here.

[0149] In some implementations, in addition to the hardware units described above, computer devices may also include software modules, such as operating systems, basic input / output systems (BIOS), and application software.

[0150] An operating system is used to manage the hardware and / or software resources of a computer device; it is the kernel and foundation of the computer. The operating system handles fundamental tasks such as managing and configuring memory, determining the priority of system resource allocation, controlling input and output devices, operating the network, and managing the file system. To facilitate user operation, most operating systems provide a user interface for interaction with the system.

[0151] The BIOS is used to perform hardware initialization during the power-on boot phase and to provide runtime services for the operating system and applications. In some implementations, the BIOS can also monitor and display processor temperature and execute temperature protection strategies.

[0152] Application software, also known as an application program, is software written for a specific user application purpose and is one of the main categories of computer software. For example, application software can be a program used to achieve purposes such as power control and temperature management.

[0153] It is understood that the specific examples in this application are only intended to help those skilled in the art better understand the implementation of this application, and are not intended to limit the scope of protection of this application.

[0154] It is understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0155] It is understood that the various implementation methods described in this application can be implemented individually or in combination, and this application does not limit them.

[0156] Unless otherwise stated, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0157] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design intervention conditions of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0158] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the embodiments described above can be referred to the corresponding processes in other embodiments, and will not be repeated here.

[0159] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0160] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the technical solution in this application, depending on actual needs.

[0161] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0162] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to related technologies, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0163] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating robot actions, characterized in that, include: Receive visual observation information and task instruction information corresponding to the target operation task, and perform feature extraction based on the visual observation information and the task instruction information respectively to obtain visual representation information and instruction representation information; Based on the visual representation information, environmental representation information is determined to characterize the environmental context in the visual observation information; Based on the cross-modal association between the visual representation information and the instruction representation information, object mediation representation information is determined, which is used to represent the task object associated with the target operation task. Based on the instruction representation information, action mediation representation information is determined, and the action mediation representation information is used to represent the action semantics or action distribution features associated with the target operation task. Based on the visual representation information, the environmental representation information, the instruction representation information, the object mediation representation information, and the action mediation representation information, robot motion data is generated; The step of determining environmental representation information for representing the environmental context in the visual observation information based on the visual representation information includes: performing cross-attention processing on the visual representation information based on query information to obtain initial environmental representation information; performing environmental reconstruction processing on the initial environmental representation information based on candidate context representation information to obtain reconstructed environmental representation information; and determining the environmental representation information based on the reconstructed environmental representation information. The step of determining object mediation representation information based on the cross-modal association between the visual representation information and the instruction representation information includes: performing projection processing on the visual representation information and the instruction representation information respectively to obtain visual projection information and instruction projection information; determining cross-modal matching information based on the visual projection information and the instruction projection information; and determining the object mediation representation information based on the cross-modal matching information. The step of determining the action mediation representation information based on the instruction representation information includes: performing global representation processing on the instruction representation information to obtain global instruction information; determining action distribution information based on the global instruction information; and performing projection processing on the action distribution information to obtain the action mediation representation information.

2. The robot motion generation method according to claim 1, characterized in that, The step of extracting features based on the visual observation information and the task instruction information to obtain visual representation information and instruction representation information includes: Visual encoding is performed on visual data from at least one viewpoint in the visual observation information to obtain the visual representation information; The task instruction information is text-encoded to obtain the instruction representation information.

3. The robot motion generation method according to claim 1, characterized in that, The process of generating robot motion data based on the visual representation information, the environmental representation information, the instruction representation information, the object mediation representation information, and the action mediation representation information includes: The visual representation information, the environmental representation information, the instruction representation information, the object mediation representation information, and the action mediation representation information are fused together to obtain action generation condition information; Actions are generated based on the action generation condition information to obtain robot action data within at least one target control cycle.

4. The robot motion generation method according to claim 1, characterized in that, The robot motion data is generated through a motion generation model, and the training process of the motion generation model includes: The action generation model performs the following processes: receiving training samples, which include training visual observation information, training task instruction information, and reference action data; extracting features from the training visual observation information and the training task instruction information to obtain training visual representation information and training instruction representation information, respectively; determining training environment representation information based on the training visual representation information to represent the environmental context in the training visual observation information; determining training object mediation representation information based on the cross-modal association between the training visual representation information and the training instruction representation information, which is used to represent the training task object; determining training action mediation representation information based on the training instruction representation information, which is used to represent the semantics or distribution features of the training action; and generating predicted action data based on the training visual representation information, the training environment representation information, the training instruction representation information, the training object mediation representation information, and the training action mediation representation information. Based on the difference between the predicted action data and the reference action data, the model parameters of the action generation model are adjusted to update the action generation model.

5. The robot motion generation method according to claim 4, characterized in that, The step of determining training environment representation information for representing the environmental context in the training visual observation information based on the training visual representation information includes: Based on the query information, cross-attention processing is performed on the training visual representation information to obtain the initial training environment representation information; Based on the candidate context representation information, the initial training environment representation information is reconstructed to obtain the reconstructed training environment representation information. Under the condition of environmental intervention, intervention context representation information is determined based on the candidate context representation information, and training environment representation information is determined based on the intervention context representation information and the training reconstructed environment representation information; or, If the environmental intervention conditions are not met, the training environment representation information is determined based on the training-reconstructed environment representation information.

6. The robot motion generation method according to claim 4, characterized in that, The training samples also include training robot status information; The step of determining the intermediate representation information of training actions based on the training instruction representation information includes: performing global representation processing on the training instruction representation information to obtain global training instruction information; determining training action distribution information based on the global training instruction information; and performing projection processing on the training action distribution information to obtain the intermediate representation information of training actions. The training process of the action generation model further includes: performing action reconstruction processing based on the training action distribution information and the training robot state information to obtain reconstructed action data; and adjusting at least the model parameters used to determine the intermediate representation information of the training action in the action generation model based on the difference between the reconstructed action data and the reference action data to update the action generation model.

7. The robot motion generation method according to claim 4, characterized in that, The training process of the action generation model also includes: The training task instruction information is subjected to word masking or word discarding to obtain processed task instruction information. Feature extraction is performed on the processed task instruction information to obtain processed instruction representation information; Based on the training visual representation information, the training environment representation information, the processed instruction representation information, the training object mediation representation information, and the training action mediation representation information, processed predicted action data is generated. Based on the difference between the processed predicted action data and the reference action data, the model parameters of the action generation model are adjusted to update the action generation model.

8. A robot motion generation device, characterized in that, include: The feature extraction module is used to receive visual observation information and task instruction information corresponding to the target operation task, and to perform feature extraction based on the visual observation information and the task instruction information respectively to obtain visual representation information and instruction representation information. An environmental characterization module is used to determine environmental characterization information for characterizing the environmental context in the visual observation information based on the visual characterization information. An object mediation representation module is used to determine object mediation representation information based on the cross-modal association between the visual representation information and the instruction representation information, wherein the object mediation representation information is used to represent the task object associated with the target operation task; An action mediation representation module is used to determine action mediation representation information based on the instruction representation information, wherein the action mediation representation information is used to represent the action semantics or action distribution features associated with the target operation task. The motion generation module is used to generate robot motion data based on the visual representation information, the environmental representation information, the instruction representation information, the object mediation representation information, and the motion mediation representation information; The step of determining environmental representation information for representing the environmental context in the visual observation information based on the visual representation information includes: performing cross-attention processing on the visual representation information based on query information to obtain initial environmental representation information; performing environmental reconstruction processing on the initial environmental representation information based on candidate context representation information to obtain reconstructed environmental representation information; and determining the environmental representation information based on the reconstructed environmental representation information. The step of determining object mediation representation information based on the cross-modal association between the visual representation information and the instruction representation information includes: performing projection processing on the visual representation information and the instruction representation information respectively to obtain visual projection information and instruction projection information; determining cross-modal matching information based on the visual projection information and the instruction projection information; and determining the object mediation representation information based on the cross-modal matching information. The step of determining the action mediation representation information based on the instruction representation information includes: performing global representation processing on the instruction representation information to obtain global instruction information; determining action distribution information based on the global instruction information; and performing projection processing on the action distribution information to obtain the action mediation representation information.

9. A robot, characterized in that, The robot includes a control module for performing the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images