Information generation method, apparatus, device, storage medium, and program product

Abstract

The application discloses an information generation method and device, equipment, a storage medium and a program product, and applies to the technical field of artificial intelligence. The information generation method comprises the following steps: acquiring a user instruction in a first time window, and a first environment image set and joint space state information collected by a robot in the first time window, wherein the user instruction is used for instructing the robot to perform an interactive operation on a target object; inputting the user instruction, the first environment image set and the joint space state information into an environment reasoning network, and acquiring environment reasoning information of a second time window output by the environment reasoning network, wherein the environment reasoning information comprises motion state information of a reference object, and the reference object comprises the target object and an interactive object which has an interactive behavior with the target object; and determining motion control information of the robot in the second time window according to the environment reasoning information. The method can improve the accuracy of an interactive task execution of an embodied intelligent robot in a complex dynamic environment.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to an information generation method, apparatus, device, storage medium and program product. Background Technology

[0002] An embodied intelligent robot is an intelligent robot with a physical entity, equipped with an artificial intelligence (AI) system. This AI system generates control commands by integrating technologies such as machine vision, natural language understanding, and motion control, and uses these commands to control the embodied intelligent robot to complete interactive tasks with a target object. However, during actual task execution, due to computational and processing delays in the AI ​​system, there is a temporal misalignment between the control commands generated by the AI ​​system based on historical perception data and the actual state of the target object at the time the control commands are executed. This causes the control commands to fail to accurately reflect the current state of the target object, thus affecting the accuracy of task execution. Summary of the Invention

[0003] This application provides an information generation method, apparatus, device, storage medium, and program product that can solve the problem of low accuracy in task execution by embodied intelligent robots.

[0004] In a first aspect, embodiments of this application provide an information generation method, which includes: acquiring a user instruction within a first time window, and a first set of environmental images and joint space state information collected by a robot within the first time window, wherein the user instruction is used to instruct the robot to perform an interactive operation on a target object; inputting the user instruction, the first set of environmental images, and the joint space state information into an environmental inference network, and acquiring environmental inference information of a second time window output by the environmental inference network, wherein the environmental inference information includes motion state information of a reference object, the reference object includes a target object and an interactive object that interacts with the target object, and the second time window is a time window located after the first time window; and determining motion control information of the robot in the second time window based on the environmental inference information, wherein the motion control information is used to control the robot to perform the interactive operation on the target object indicated by the user instruction in the second time window.

[0005] Secondly, embodiments of this application provide an information generation device, comprising: an acquisition module, configured to acquire user instructions within a first time window, and a first set of environmental images and joint space state information collected by the robot within the first time window, wherein the user instructions are used to instruct the robot to perform interactive operations on a target object; a spatiotemporal perception reasoning module, configured to input the user instructions, the first set of environmental images, and the joint space state information into an environmental reasoning network, and acquire environmental reasoning information for a second time window output by the environmental reasoning network, wherein the environmental reasoning information includes motion state information of a reference object, the reference object includes a target object and an interactive object that interacts with the target object, and the second time window is a time window located after the first time window; and a determination module, configured to determine motion control information of the robot in the second time window based on the environmental reasoning information, wherein the motion control information is used to control the robot to perform the interactive operations on the target object indicated by the user instructions in the second time window.

[0006] Thirdly, embodiments of this application provide an electronic device, which includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements an information generation method as described in any of the first aspects.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the information generation method as described in any of the first aspects.

[0008] Fifthly, embodiments of this application provide a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, implements the information generation method as described in any of the first aspects.

[0009] The information generation method, apparatus, device, and computer storage medium of this application embodiment can obtain user instructions within a first time window, as well as a first set of environmental images and joint space state information collected by the robot within the first time window, and input them into an environmental inference network to determine environmental inference information for a second time window. Based on this environmental inference information, the robot's motion control information for the second time window can be determined. This allows for the prediction of the motion trends of the target object and the interactive object in advance, enabling the robot to generate more forward-looking and adaptive motion control information. It effectively solves the problem of target object state misalignment caused by time delay, reduces the risk of task execution caused by time delay and environmental dynamism, and improves the accuracy of interactive task execution by embodied intelligent robots in complex dynamic environments. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating some embodiments of the information generation method provided in this application is shown; Figure 2 This illustration shows a schematic diagram of the attention flow of the query token attention mechanism in the information generation method provided in some embodiments of this application; Figure 3 A flowchart illustrating a specific implementation of step 120 provided in some embodiments of this application is shown; Figure 4 The diagram illustrates a specific implementation of step 130 provided in some embodiments of this application; Figure 5 A flowchart illustrating a method for determining motion control information provided in some embodiments of this application is shown. Figure 6 A flowchart illustrating a method for determining reference motion control information provided in some embodiments of this application is shown. Figure 7 A flowchart illustrating a method for determining joint space control commands using an information generation method provided in some embodiments of this application is shown. Figure 8 The present application provides schematic diagrams of the structure of an information generation apparatus according to some embodiments. Figure 9 The diagram shows a schematic representation of the structure of an electronic device provided in some embodiments of this application. Detailed Implementation

[0012] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0013] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0014] It should be noted that the acquisition, storage, use, and processing of data in this application embodiment all comply with the relevant provisions of national laws and regulations.

[0015] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0016] Before describing the technical solutions provided in the embodiments of this application, in order to facilitate understanding of the embodiments of this application, this application first provides a detailed description of the relevant technologies involved: In embodied intelligence systems, computational and processing delays in the artificial intelligence system lead to a temporal misalignment between the control commands generated from historical perception data and the actual state of the target object at the time the control commands are executed. This misalignment prevents the control commands from accurately reflecting the current state of the target object, thus affecting the accuracy of task execution. For example, in the application scenario of an embodied intelligent robot grasping a dynamically moving object, the artificial intelligence system acquires environmental images and perceives the position of the target object at time t0, and generates the grasping command for time t2 based on the perception data at time t0 at time t1. Furthermore, within the delay period from t1 to t2, the motion state of the target object changes, causing the target object to be no longer in the predetermined position when the command is executed at time t2, thus triggering interaction failure and ultimately leading to task execution failure.

[0017] To address the problems in the aforementioned related technologies, embodiments of this application provide an information generation method, apparatus, device, storage medium, and program product. The following description, in conjunction with the appendix... Figure 1 To be continued Figure 7 The information generation method provided in this application will be described in detail through specific embodiments and application scenarios.

[0018] Figure 1 A flowchart illustrating some embodiments of the information generation method provided in this application is shown. For example... Figure 1 As shown, the information generation method may include steps 110 to 130.

[0019] Step 110: Obtain the user command within the first time window, as well as the first set of environmental images and joint space state information collected by the robot within the first time window. The user command is used to instruct the robot to perform interactive operations on the target object. Step 120: Input the user command, the first set of environmental images, and the joint space state information into the environmental inference network to obtain the environmental inference information of the second time window output by the environmental inference network. The environmental inference information includes the motion state information of the reference object, which includes the target object and interactive objects that interact with the target object. The second time window is the time window located after the first time window. Step 130: Determine the motion control information of the robot in the second time window based on the environmental inference information. The motion control information is used to control the robot to perform the interactive operations on the target object indicated by the user command in the second time window.

[0020] Therefore, by acquiring user commands within the first time window, as well as the first set of environmental images and joint space state information collected by the robot within the first time window, and inputting them into the environmental inference network to determine the environmental inference information for the second time window, and then determining the robot's motion control information for the second time window based on this environmental inference information, the robot can predict the motion trends of the target object and the interactive object in advance. This enables the robot to generate more forward-looking and adaptive motion control information, effectively solving the problem of target object state misalignment caused by time delay, reducing the task execution risk caused by time delay and environmental dynamism, and improving the accuracy of embodied intelligent robots in performing interactive tasks in complex dynamic environments.

[0021] The steps described above are explained in detail below.

[0022] First, regarding step 110, the first time window in this embodiment refers to a specific time period. The length of the first time window can be set according to the needs of the actual application scenario of the embodied intelligent robot. For example, it can be a fixed short time interval. User instructions refer to instructions issued by the user to the robot to instruct the robot to perform specific tasks or operations. Specifically, it can include which target object the robot needs to perform what kind of interactive operation on. These user instructions can be obtained in various ways. For example, user instructions can be obtained through text commands typed by the user in a text input interface; another example is that user instructions can be obtained by listening to and recognizing the user's voice input through the speech recognition system built into the artificial intelligence system, and converting the voice commands into text data through speech recognition technology; yet another example is that user instructions can also be obtained by recognizing gesture commands issued by the user to the robot through a gesture recognition system.

[0023] The first environmental image set refers to the collection of multiple image data acquired by the robot through its vision sensors within the first time window. In specific implementation, the artificial intelligence system acquires static RGB images and depth images from the environment, showing the robot's global perspective and the robot gripper's perspective. The image size is uniformly (640, 480, 3). Furthermore, these image data are transmitted to the central processing unit via the acquisition interface and stored in a timestamp format for subsequent feature extraction and encoding.

[0024] Joint space state information refers to the current state data of each joint of the robot within the first time window, which is used to accurately reflect the robot's real-time posture and motion capabilities. Specifically, it can include the angle, angular velocity or torque values ​​of each joint of the head, waist and both arms obtained by joint sensors and torque sensors, as well as the opening and closing state of the end effector gripper.

[0025] In some embodiments of this application, after acquiring the aforementioned multimodal raw data, a unified feature encoding can be performed to convert it into a feature representation that can be processed by the environment inference network. This may include text instruction encoding, visual information encoding, and state information encoding. Text instruction encoding refers to cleaning, segmenting, lexical reconstruction, and stop word removal of the recognized text instructions, and then converting them into a word vector sequence. Visual information encoding refers to sampling the acquired multi-view images to obtain image sequences. The image frame at each time step is input into a Vision Transformer (ViT). The ViT divides the image into fixed-size image blocks, performs linear projection and position encoding on each block, and finally forms a visual token sequence. State information encoding refers to mapping the acquired joint spatial state information, such as joint angles and torques, into vector representations through a linear multilayer perceptron (MLP).

[0026] Secondly, regarding step 120, the environmental inference network refers to a computational model used to receive multimodal input data and predict and infer the future state of the robot's environment. The second time window refers to a time period following the first time window. This second time window is the target time period for the environmental inference network to predict the future state. The length of the second time window can be set according to the needs of the actual application scenario of the embodied intelligent robot; for example, it can be a fixed short time interval. Environmental inference information refers to the predicted data about the environmental state within the second time window output by the environmental inference network. The reference object refers to the set of objects that need to be considered during the robot's interactive operations. This set of objects can include the target object explicitly indicated in the user instructions and interactive objects that may interact with the target object. Motion state information refers to information including the dynamic attributes of the reference object within the second time window, such as its position, velocity, and attitude.

[0027] In this step, the environment inference network can be a pre-trained deep learning model, such as a recurrent neural network or a Transformer-based model. The environment inference network analyzes the semantic content of user commands, visual features in a first set of environmental images, and the robot's own state in joint space state information to infer the possible motion trends of key objects in the environment after the first time window and within a second time window. For example, the environment inference network can identify the target object and interactive objects that may interact with the cup, and predict the motion state information of these reference objects within the second time window, such as changes in position, velocity, or posture.

[0028] In some embodiments of this application, the above-described information generation method may further include step 140, which involves inputting the text features of the user instruction, the image features of the first environment image set, and the state features of the joint space state information into a multimodal neural network to obtain the multimodal fusion features output by the multimodal neural network. The multimodal neural network is used to interactively model the text features, image features, and state features using a multi-head self-attention algorithm to obtain the multimodal fusion features. Based on this, step 120 may specifically include: inputting the multimodal fusion features into an environment inference network to obtain the environment inference information of the second time window output by the environment inference network.

[0029] Multimodal neural networks (MNNs) are neural network models that use multi-head self-attention algorithms to cross-learn and fuse features from different modalities, such as text features, image features, and state features. This MNN can effectively learn the correlations and complementarities between data from different modalities, thus capturing the complex relationships between features more comprehensively. Multimodal fusion features are unified vector representations that effectively integrate feature information from different modalities after processing by the MNN, enabling a more comprehensive and accurate description of the overall situation of the current task and environment.

[0030] Therefore, by deeply fusing the text features of user commands, the image features of the first environment image set, and the state features of joint space state information, the semantic differences between different modal data are effectively compensated, enabling the environment reasoning network to receive more comprehensive and accurate input information, thereby significantly improving the accuracy of environment reasoning and enabling the robot to execute the interactive operations indicated by user commands more accurately.

[0031] In some embodiments of this application, the environment reasoning network is specifically implemented as a query-based world knowledge prediction module. This world knowledge prediction module utilizes a set of learnable, function-specific query tokens to extract different types of future environmental knowledge in parallel from unified multimodal fusion features.

[0032] Among them, the world knowledge query token, as the Query input in the attention process, can extract information from the multimodal fusion features using the cross-attention algorithm represented by the following formula (1): (1) like Figure 2 As shown, the rows and columns of this matrix are labeled in the same way, representing: text, joint state, image sequence, dynamic region, depth map, semantic map, subtask, and action query, respectively. The squares in the matrix indicate whether the node represented by the row label can pay attention to the node represented by the column label: gray squares indicate that attention connections are allowed, meaning information can flow from the column node to the row node; white squares indicate that attention connections are blocked, meaning the information flow is interrupted. Specifically, the three-layer information processing structure involved in this matrix is ​​shown below: The first layer is a unified multimodal information source, including text, joint states, and image sequences. For example... Figure 2 As shown, this layer of information is fully open to all upper-layer nodes. Dynamic regions, depth maps, semantic maps, subtasks, and action queries can all pay attention to this kind of basic information, thereby ensuring that all high-level reasoning is built on the common foundation of fused perception and instructions.

[0033] The second layer consists of specialized world knowledge prediction subqueries, comprising four nodes: dynamic region, depth map, semantic map, and subtask. Attention connections are prohibited between subqueries, achieving information isolation and effectively preventing cross-type knowledge leakage issues such as motion features contaminating geometric reasoning and semantic cues interfering with task decomposition. This ensures the independence of each prediction task and the purity of the output results.

[0034] The third layer is the final action planning query, including action query nodes. For example... Figure 2 As shown, this action query node can simultaneously focus on all basic information sources and all world knowledge prediction results, enabling the final action generation to integrate all underlying perceptions and high-level prediction knowledge to make globally optimal decisions.

[0035] so, Figure 2 The revealed attention mechanism, through conditional masking and connection, allows each sub-query in world knowledge prediction to extract features from a shared context without interfering with each other; simultaneously, the action planning query integrates all information. This ensures the accuracy, modularity, and consistency and synergy of the robot's AI system in multi-dimensional environmental reasoning, as well as in the final action planning.

[0036] In some embodiments of this application, the environmental inference information includes motion state information and subtask sequences. The motion state information includes region information, depth information, semantic category feature information, and instance segmentation information. The environmental inference network includes a first subnetwork, a second subnetwork, a third subnetwork, and a fourth subnetwork. This environmental inference network can be a structure including multiple decoders, with one decoder corresponding to one subnetwork. The environmental inference information environment can be obtained through the inference network. This can be achieved through the following formula (2): (2) in, M represents the environment reasoning information, and M represents the multi-decoder structure.

[0037] Based on this, such as Figure 3 As shown, step 120 above may specifically include steps 1201 to 1204.

[0038] Step 1201: Input the environmental change-related features from the multimodal fusion features into the first sub-network, and obtain the regional information of the second time window output by the first sub-network. The regional information includes information used to represent the region where the reference object is located in the second time window.

[0039] The first sub-network is specifically designed to process environmental change-related features in the multimodal fusion features and output regional information for the second time window. This first sub-network can be a detection model based on a convolutional neural network or a model based on a Transformer architecture, used to capture temporal environmental changes and predict future regions. The regional information includes information representing the region where the reference object is located in the second time window. This regional information can be presented in various forms; for example, it can be the two-dimensional bounding box coordinates of the reference object on the image plane, or its volume bounding box coordinates in three-dimensional space; or it can be the keypoint position of the reference object in a specific coordinate system.

[0040] In this step, cross-attention calculation can be performed first by querying the dynamic region token and multimodal fusion features. Based on the calculated attention weights, the feature components most relevant to dynamic changes are weighted and aggregated from the multimodal fusion features to obtain features related to environmental changes. Then, the features related to environmental changes are input into a lightweight visual Transformer decoder. This decoder performs spatial modeling and distribution reconstruction through multi-layer self-attention and feedforward networks, and finally outputs a binarized dynamic region prediction mask that is aligned with the input image space. This dynamic region prediction mask is the region information of the second time window.

[0041] Understandably, the dynamic region query token is a learnable, specialized query pointer or guiding signal used to indicate "please extract the spatial and spatiotemporal features related to scene dynamics, object motion trends, and potential interaction areas from the multimodal fusion features." During the training phase of the first sub-network, it can undergo supervised learning by comparing the predicted mask with the ground truth of the dynamic mask generated by analyzing pixel motion trajectories using optical flow tracing models, such as the Cooperative Tracker (CoTracker), thereby learning to infer the distribution of future dynamic regions from current observation information.

[0042] Step 1202: Input the geometrically related features from the multimodal fusion features into the second sub-network to obtain the depth information of the second time window output by the second sub-network. The depth information includes information used to represent the relative pose between the reference object and the robot within the second time window.

[0043] The second sub-network is specifically designed to process geometrically related features in the multimodal fusion and output depth information for the second time window. This second sub-network can be a depth estimation network or a neural network based on stereo matching or structured light principles to extract depth information from multi-view images. The depth information can be a depth map formed by the distances from each point on the reference object's surface to the robot's camera plane; alternatively, it can be the three-dimensional coordinates of the reference object relative to the robot; or it can be the relative distance and direction vector between the reference object and the robot's end effector.

[0044] In this step, cross-attention calculation can be performed first by querying the token and multimodal fusion features. Based on the calculated attention weights, the feature components most related to the geometric structure are weighted and aggregated from the multimodal fusion features to obtain the deep feature representation. Specifically, this can be achieved by the following formula (3). Then, the deep feature representation is fused with the spatial coding features to restore its spatial image structure and obtain the features related to the geometric structure. Subsequently, the features related to the geometric structure are input into the lightweight visual Transformer decoder for upsampling and reconstruction, and the decoded output is the predicted future depth map.

[0045] Understandably, a depth query token is a learnable, specialized query pointer or guiding signal used to represent the specific intent of "finding the 3D geometry, spatial layout, and object surface orientation information of the scene from the multimodal fusion features." During the training phase of the second sub-network, it utilizes real-world depth map data from depth sensors, such as RealSense cameras, as supervisory labels, optimizing network parameters by minimizing the difference between the predicted and true depth maps. This depth information is ultimately used to calculate the precise relative distance and orientation between the reference object and the robot's end effector, providing crucial geometric constraints for motion planning.

[0046] (3) in, The query vector is obtained by projecting the depth-query token through a linear transformation. Let d be the transpose of the key matrix. Dimensions This is the normalization function.

[0047] Step 1203: Input the semantically related features from the multimodal fusion features into the third sub-network to obtain the semantic category feature information and instance segmentation information of the second time window output by the third sub-network.

[0048] The third sub-network is specifically designed to process semantically relevant features in the multimodal fusion features and output semantic category feature information and instance segmentation information for the second time window. This third sub-network can be a semantic segmentation network capable of simultaneously performing pixel-level classification and instance segmentation, or it can be a network combined with a graph neural network for inferring semantic relationships between objects. The semantic category feature information is used to identify the category attributes of reference objects; for example, it can indicate whether the reference object belongs to a specific category such as "cup," "book," or "person." This semantic category feature information can be represented in the form of one-hot encoding or as a high-dimensional feature vector. The instance segmentation information is used to accurately separate each reference object instance in the image from the background. For example, a pixel-level mask can be generated for each individual object in the image. This instance segmentation information can be represented as a binary mask image, where the value of each pixel indicates whether it belongs to a specific instance.

[0049] In this step, cross-attention calculation can be performed first by semantic query token and multimodal fusion features. Based on the calculated attention weights, semantically related feature components are weighted and aggregated from the multimodal fusion features to obtain semantically related features. Then, the semantically related features are input into two parallel lightweight decoders: a DINO decoder is used to regress the dense semantic feature map of the image, i.e. semantic category feature information; a segmentation decoder is used to predict the precise mask of each object instance, i.e. instance segmentation information.

[0050] Understandably, this semantic query token is a learnable, specialized query pointer or guiding signal used to represent "please parse semantic features related to object category, part, and identity from multimodal fusion features." During the training phase of the third sub-network, the outputs of the DINO decoder and segmentation decoder can be optimized using visual features extracted by the Self-Distillation with No Labels version 2 (DINOv2) model and segmentation masks generated by the Segment Anything Model (SAM) model, respectively, as supervision signals. This allows the third sub-network to learn to predict future semantic scene understanding from current observations.

[0051] Step 1204: Input the features related to the task logic from the multimodal fusion features into the fourth sub-network to obtain the sub-task sequence of the second time window output by the fourth sub-network. The sub-task sequence is related to the motion state information of the reference object in the second time window.

[0052] The fourth sub-network is specifically designed to process task-logic-related features in the multimodal fusion features and output a sequence of subtasks within a second time window. This fourth sub-network can be a sequence generation model that predicts a series of discrete subtask steps based on the current task objective and environmental state, such as a recurrent neural network or a Transformer; or it can be a policy network based on reinforcement learning that directly outputs a sequence of actions related to the task logic.

[0053] In this step, cross-attention calculation can be performed first by querying the task token and multimodal fusion features. Based on the calculated attention weights, the feature components most relevant to task decomposition and sequential planning are weighted and aggregated from the multimodal fusion features to obtain features related to task logic. The features related to task logic and the semantic representation of user instructions are input into a language decoder based on the Transformer architecture. The language decoder optimizes the execution priority based on the task logic order and also combines key motion information obtained from dynamic region prediction, and decodes long task instructions step by step to generate a sub-task sequence corresponding to the environmental state and task objective.

[0054] Understandably, the task query token is a learnable, specialized query pointer or guiding signal used to indicate "Please extract the temporal and causal features related to action steps, execution logic, and task constraints from the multimodal fusion features".

[0055] Therefore, by decomposing the environmental reasoning network into multiple specialized sub-networks and inputting them into these sub-networks according to different feature attributes, the environmental reasoning network can process environmental information from different dimensions in parallel and professionally. This avoids the potential decrease in accuracy and inefficiency that might result from a single network processing all complex information. This multi-dimensional and refined environmental reasoning capability significantly improves the robot's understanding of complex dynamic environments and its prediction accuracy. It provides a richer, more accurate, and operable basis for determining subsequent motion control information, thereby enabling more effective planning and execution of interactive operations targeting the object and improving the success rate of robot task execution.

[0056] Then, regarding step 130, motion control information refers to the set of instructions used to directly control the robot to perform specific actions within the second time window. This motion control information may include target position, velocity, or torque commands for each joint of the robot, or target pose sequences for the end effector. Interactive operation refers to the physical or logical actions performed by the robot on a target object according to user instructions. Examples include grasping, placing, pushing, and observing.

[0057] In some embodiments of this application, such as Figure 4As shown, step 130 above may include steps 1301 and 1302.

[0058] Step 1301: Input the environmental reasoning information and multimodal fusion features into the action planning network to obtain the action execution sequence output by the action planning network. The action execution sequence includes the target pose information of the end effector of the robot at each time within the second time window. The multimodal fusion features are determined based on user instructions, the first environmental image set and joint space state information.

[0059] The motion planning network is a network used to transform task instructions and environmental states into a sequence of specific robot actions, i.e., a series of target poses for the robot's end effector. This motion planning network can be implemented based on a deep learning model, such as a Transformer, which learns from a large amount of robot operation data to establish a mapping relationship between environmental states and motion sequences. The motion execution sequence is a series of time-sequential target pose information for the robot's end effector, including the end effector position and orientation that the robot should achieve at each moment within a second time window.

[0060] Step 1302: Determine the motion control information of the robot in the second time window based on the action execution sequence.

[0061] Therefore, by introducing an action planning network, environmental reasoning information and multimodal fusion features are effectively integrated, thereby generating a refined sequence of action executions. This enables the robot to execute the interactive operations indicated by the user's instructions more accurately, significantly improving the accuracy of the robot in performing complex tasks.

[0062] In some embodiments of this application, the motion control information of the second time window includes the joint space control commands of the robot at each moment within the second time window; based on this, step 1302 may specifically include: inputting the action execution sequence, multimodal fusion features and joint space state information into a low-rank action execution network, and obtaining the joint space control commands of the robot at each moment within the second time window output by the low-rank action execution network; wherein, the low-rank action execution network is used to fuse the action execution sequence, multimodal fusion features and joint space state information to generate a latent action embedding representation, and to perform denoising processing on the latent action embedding representation to obtain the joint space control commands of the robot at each moment within the second time window.

[0063] Joint space control commands refer to control signals that directly act on the robot's joints, such as joint angle, joint angular velocity, and joint torque commands. These joint space control commands are used to drive the robot's joint movements, thereby achieving the desired pose of the end effector. Low-rank motion execution networks (LSINs) are networks that efficiently generate robot control commands by learning and utilizing the low-rank structure in motion data. Latent motion embedding representations are abstract intermediate representations learned internally by the LINs.

[0064] In this step, the low-rank action execution network can be a Diffusion Transformer (DiT-B) model based on the Transformer architecture. In this Diffusion Transformer model, a learnable action query token is used to perform global attention aggregation on the multimodal fusion features, extracting latent action embeddings related to the current task execution. These latent action embeddings encode the core action intent related to the current task execution. Next, the Transformer DiT-B model iteratively recovers a high-quality action vector sequence from Gaussian noise through a progressive denoising process, ultimately mapping it to joint space control commands. In other words, by learning to reverse a progressively adding noise process, a clear and reasonable action sequence is synthesized from pure noise.

[0065] This step can be achieved using the following formula (4): (4) in, Let D(.)wei be a sequence of joint space control instructions that lasts for n time steps starting from time t, and let D(.)wei be a low-rank motion execution network. This represents the joint space state information at time t. For user commands, For environmental observations at time t,<word query> These are learnable word query vectors used as guiding signals to extract task-relevant semantic information from multimodal input. <action>It is a learnable action query vector used to extract guiding signals related to action execution features from multimodal inputs and world knowledge.

[0066] Therefore, the low-rank motion execution network can effectively integrate multi-source information and learn the essential low-dimensional representation of motion. It transforms the target pose information of the high-level end effector into joint space control commands that the robot can directly execute, and performs denoising processing on the latent motion embedding representation. This ensures that the generated joint space control commands are more stable and accurate, enabling the robot to perform complex interactive operations with higher precision and significantly improving the accuracy of robot task execution.

[0067] In some embodiments of this application, the second time window includes a first sub-time window and a second sub-time window located after the first sub-time window; wherein, the second time window can be divided into two or more consecutive time periods, wherein the first sub-time window is used for initial execution of instructions and data acquisition, and the second sub-time window is used for adjusting the control instructions of the second sub-time window based on the feedback information of the first sub-time window. Based on this, after step 130 above, as Figure 5 As shown, the above information generation method may further include steps 1501 to 1503.

[0068] Step 1501: Obtain the second set of environmental images collected by the robot in the first sub-time window.

[0069] In this step, after the robot begins to execute the motion control information of the second time window, within the first sub-time window, the robot uses its onboard visual sensors, such as an RGB camera, a depth camera, or a stereo camera, to capture image data of the current environment again.

[0070] Step 1502: If the pose of the reference object determined by the second environmental image set and the first environmental image set has not changed within the first sub-time window, the motion control information is adjusted according to the reference motion control information to obtain the adjusted motion control information. The reference motion control information is determined based on the first environmental image set, joint space state information and user commands.

[0071] The reference motion control information can be an ideal motion trajectory or control sequence based on the initial environmental state and user instructions.

[0072] In one example, a feature point matching algorithm can be used to identify and track key feature points of a reference object in two sets of images, and the relative displacement of the feature points can be used to determine whether the reference object has moved.

[0073] In another example, object detection and pose estimation algorithms are used to identify reference objects in two sets of images and estimate their 3D poses. Then, the difference between the two estimated poses is compared to see if it is less than a preset threshold to determine if it has not changed.

[0074] In this step, when it is confirmed that the reference object remains stationary within the first sub-time window, the motion control information currently planned for the entire second time window is corrected using a pre-determined reference motion control information, or reference motion control information generated based on the initial input information, namely the environmental image set, joint space state information, and user commands of the first time window. For example, by calculating the deviation between the robot end effector pose indicated by the current motion control information and the ideal pose indicated by the reference motion control information, a controller, such as a proportional-integral-derivative controller, is applied to generate a correction amount. This correction amount is then superimposed on the original motion control information to obtain the adjusted motion control information.

[0075] Step 1503: The adjusted motion control information is determined as the motion control information of the robot in the second sub-time window.

[0076] In this step, a new, more precise sequence of motion control commands, after adjustments, will be assigned to the actions the robot will perform next in the second sub-time window. This ensures that the robot can complete the remaining interactive operations with higher precision based on the latest environmental observations and confirmation of the stationary state of the reference object.

[0077] The following is a specific example to illustrate this.

[0078] If the user instructs the robot to grasp a water glass placed on a table, in the first time window, the robot acquires a first set of environmental images of the water glass and its own joint space state information. Based on the user's instruction to grasp the water glass, it plans the grasping trajectory for the next 5 seconds, i.e., the second time window. These 5 seconds can be divided into the first 2 seconds, i.e., the first sub-time window, and the last 3 seconds, i.e., the second sub-time window; the robot begins executing the movement for the first 2 seconds. At the 1.5-second mark, the robot takes another image of the water glass, i.e., the second set of environmental images, and compares this image with the first set of environmental images to confirm that the position and orientation of the water glass have not changed within the first sub-time window. At this point, an ideal grasping trajectory determined based on the initial position of the water glass and the robot's joint states, i.e., reference motion control information, is used to fine-tune the originally planned motion control information for the last 3 seconds, enabling the robot to more accurately approach and grasp the water glass in the next 3 seconds. Finally, the robot will use this adjusted motion control information to complete the grasping action in the second sub-time window.

[0079] Therefore, by subdividing the second time window into a first sub-time window and a second sub-time window, and acquiring a new set of environmental images within the first sub-time window, the existing motion control information can be adjusted using reference motion control information determined based on the initial observation information when the pose of the reference object does not change within the first sub-time window. This avoids the time-consuming and computationally intensive environmental reasoning and motion planning that occurs every time a small deviation occurs. Furthermore, by making local adjustments to the existing motion control information, the accumulated errors in the robot's execution or perception can be corrected, thereby ensuring that the robot can more accurately execute the interactive operations indicated by the user's instructions and effectively improving the accuracy of the robot's operation.

[0080] In some embodiments of this application, before performing step 1502 above, such as Figure 6 As shown, the above information generation method may further include steps 1601 to 1603.

[0081] Step 1601: Based on the first environmental image set, determine the first relative pose information of the robot and the target object in the first time window.

[0082] In one example, computer vision technology can be used to process the first set of environmental images acquired in the first time window, identify the robot end effector and the target object through target detection and pose estimation algorithms, and calculate their relative positions and poses.

[0083] In another example, by combining the robot's own kinematic model and visual information, the robot obtains the pose of its end effector through its internal sensors, while using visual sensors to identify the position of the target object in the image, and then obtains the relative pose through coordinate transformation and geometric calculation.

[0084] Step 1602: Based on user instructions, determine the second relative pose information that the robot and the target object need to achieve in the second time window.

[0085] In this step, the semantics of user commands can be parsed and mapped to a predefined library of interactive actions using natural language processing technology. Each interactive action corresponds to one or a set of desired relative poses of the robot and the target object.

[0086] Step 1603: Determine reference motion control information based on the deviation information between the first relative pose information and the second relative pose information, as well as the joint space state information.

[0087] In this step, the deviation information between the first and second relative pose information is used, combined with the robot's kinematic and dynamic characteristics, to generate control commands that guide the robot from its current state to a desired state. For example, a model-based control method, such as an inverse kinematics solver, can be used. This method takes the deviation information between the first and second relative pose information as input, combines it with the robot's kinematic model and the current joint space state information, and calculates the joint angle change that can eliminate the deviation, thereby generating joint space control commands.

[0088] Therefore, based on the actual relative pose of the robot and the target object in the first time window, the expected relative pose indicated by the user command, and the current joint space state of the robot, the reference motion control information is accurately calculated. This allows the robot to generate accurate motion control commands based on the clear deviation between the current state and the target state, even when the pose of the reference object has not changed. This effectively adjusts subsequent motion control information, ensuring that the robot can stably and accurately execute the interactive operations indicated by the user command, thus improving the accuracy of robot task execution.

[0089] In some embodiments of this application, the second time window includes m sub-time windows. The motion control information of the second time window includes the joint space control commands of the robot at each moment within the second time window. Here, m sub-time windows refer to subdividing a relatively long second time window into m consecutive, shorter time intervals. For example, the second time window can be uniformly divided into m sub-time windows of fixed length based on factors such as task complexity, robot speed, and environmental dynamics. After step 130 above, as... Figure 7 As shown, the above information generation method may further include steps 1701 to 1704.

[0090] Step 1701: Execute the joint space control instructions corresponding to each of the m sub-time windows according to the timing sequence of the sub-time windows to control the robot's movement.

[0091] In this step, the robot controller can receive and parse the joint space control instructions corresponding to each sub-time window, and drive each joint of the robot to move according to the instructions.

[0092] Step 1702: After executing the control command for the nth sub-time window in each of the m sub-time windows, acquire the actual environmental image collected by the robot in the nth sub-time window, where n∈[1,2,…,m-1].

[0093] In this step, the robot can immediately trigger the sensors to collect image data of the current environment after completing the movement of the nth sub-time window.

[0094] Step 1703: Calculate the actual relative pose between the robot's end effector and the target object based on the actual environment image.

[0095] In this step, image processing and computer vision algorithms, such as feature point matching, target detection and tracking, or deep learning models, can be used to identify target objects and robot end effectors from real-world images and estimate their relative positions and orientations in three-dimensional space.

[0096] Step 1704: Based on the pose error information between the actual relative pose and the expected relative pose pre-planned for the nth sub-time window, determine the joint space control commands for the robot in the (n+1)th sub-time window.

[0097] In this step, if the error value of the pose error information between the actual relative pose and the expected relative pose pre-planned for the nth sub-time window is greater than or equal to a preset error threshold, the joint space control command of the robot in the (n+1)th sub-time window is adjusted according to the error value of the pose error information to obtain the adjusted joint space control command; the adjusted joint space control command is determined as the joint space control command of the robot in the (n+1)th sub-time window.

[0098] The pose error information can include position error and / or attitude error. The error value of the pose error information is a quantitative indicator representing the degree of deviation between the actual relative pose and the desired relative pose between the robot's end effector and the target object. A preset error threshold is a pre-defined value used to determine whether adjustments to the robot control commands are needed. This preset error threshold can be empirically set based on factors such as the accuracy requirements of the specific task, the performance characteristics of the robot system, the complexity of the environment, and safety considerations. Adjustments to the joint space control commands can be based on inverse kinematics principles, calculating the amount of joint angle change required to compensate for the pose error; alternatively, feedback control algorithms such as proportional-integral-derivative (PID) controllers can be used to generate correction values ​​based on the error values.

[0099] Therefore, after the robot determines the initial motion control information based on environmental reasoning information, by subdividing the entire task execution time window into multiple sub-time windows, and performing environmental perception, pose calculation and error evaluation after each sub-time window is executed, the deviation between the robot's motion and the target can be detected and corrected in a timely manner, enabling the robot to adapt to the dynamically changing environment and significantly improving the success rate of the robot in performing complex interactive tasks.

[0100] Based on the information generation method provided in the above embodiments, this application also provides specific implementations of the information generation apparatus. Please refer to the following embodiments.

[0101] See Figure 8 The information generation device 200 provided in this application embodiment includes an acquisition module 201, a spatiotemporal perception reasoning module 202, and a determination module 203.

[0102] The acquisition module 201 is used to acquire user instructions within a first time window, as well as a first set of environmental images and joint space state information collected by the robot within the first time window. The user instructions are used to instruct the robot to perform interactive operations on the target object. The spatiotemporal perception reasoning module 202 is used to input the user instructions, the first set of environmental images, and the joint space state information into the environmental reasoning network, and acquire the environmental reasoning information of the second time window output by the environmental reasoning network. The environmental reasoning information includes the motion state information of the reference object, which includes the target object and interactive objects that interact with the target object. The second time window is a time window located after the first time window. The determination module 203 is used to determine the motion control information of the robot in the second time window based on the environmental reasoning information. The motion control information is used to control the robot to perform the interactive operations on the target object indicated by the user instructions in the second time window.

[0103] Therefore, the robot can acquire user commands within the first time window, as well as the first set of environmental images and joint space state information collected by the robot in the first time window, through the acquisition module 201. These are then input into the environmental reasoning network in the spatiotemporal perception reasoning module 202 to determine the environmental reasoning information for the second time window. The determination module 203 then determines the robot's motion control information for the second time window based on this environmental reasoning information. This allows the robot to anticipate the motion trends of the target object and the interactive object in advance, enabling it to generate more forward-looking and adaptive motion control information. This effectively solves the problem of target object state misalignment caused by time delay, reduces the risk of task execution caused by time delay and environmental dynamism, and improves the accuracy of interactive task execution by the embodied intelligent robot in complex dynamic environments.

[0104] In some embodiments of this application, the information generation device 800 in this application may further include a feature fusion module.

[0105] The feature fusion module is used to input the text features of the user command, the image features of the first environment image set, and the state features of the joint space state information into the multimodal neural network to obtain the multimodal fusion features output by the multimodal neural network. The multimodal neural network is used to perform interactive modeling of text features, image features, and state features through a multi-head self-attention algorithm to obtain the multimodal fusion features. Based on this, the spatiotemporal perception reasoning module 202 can be used to input the multimodal fusion features into the environment reasoning network to obtain the environment reasoning information of the second time window output by the environment reasoning network.

[0106] In some embodiments of this application, the environmental reasoning information includes motion state information and sub-task sequences. The motion state information includes region information, depth information, semantic category feature information, and instance segmentation information. The environmental reasoning network includes a first sub-network, a second sub-network, a third sub-network, and a fourth sub-network. The aforementioned spatiotemporal perception reasoning module 202 can be used to: input the features related to environmental changes in the multimodal fusion features into the first sub-network to obtain the region information of the second time window output by the first sub-network. The region information includes information representing the region where the reference object is located in the second time window. Input the features related to geometric structure in the multimodal fusion features into the second sub-network to obtain the depth information of the second time window output by the second sub-network. The depth information includes information representing the relative pose between the reference object and the robot within the second time window. Input the semantically related features in the multimodal fusion features into the third sub-network to obtain the semantic category feature information and instance segmentation information of the second time window output by the third sub-network. Input the task logic related features in the multimodal fusion features into the fourth sub-network to obtain the sub-task sequence of the second time window output by the fourth sub-network. The sub-task sequence is related to the motion state information of the reference object in the second time window.

[0107] In some embodiments of this application, the determining module 203 is used to: input environmental reasoning information and multimodal fusion features into the motion planning network, obtain the motion execution sequence output by the motion planning network, the motion execution sequence including the target pose information of the end effector of the robot at each moment in the second time window, the multimodal fusion features being determined based on user instructions, the first environmental image set and joint space state information; and determine the motion control information of the robot in the second time window based on the motion execution sequence.

[0108] In some embodiments of this application, the motion control information of the second time window includes the joint space control commands of the robot at each moment within the second time window; the determination module 203 is specifically used to: input the action execution sequence, multimodal fusion features and joint space state information into the low-rank action execution network, and obtain the joint space control commands of the robot at each moment within the second time window output by the low-rank action execution network; wherein, the low-rank action execution network is used to fuse the action execution sequence, multimodal fusion features and joint space state information to generate a latent action embedding representation, and to perform denoising processing on the latent action embedding representation to obtain the joint space control commands of the robot at each moment within the second time window.

[0109] In some embodiments of this application, the second time window includes a first sub-time window and a second sub-time window located after the first sub-time window; the aforementioned determining module 203 can also be used to, after performing the step of determining the robot's motion control information in the second time window based on environmental reasoning information, acquire a second set of environmental images collected by the robot in the first sub-time window; if the pose of the reference object determined based on the second set of environmental images and the first set of environmental images has not changed within the first sub-time window, adjust the motion control information based on the reference motion control information to obtain the adjusted motion control information, wherein the reference motion control information is determined based on the first set of environmental images, joint space state information, and user instructions; and determine the adjusted motion control information as the robot's motion control information in the second sub-time window.

[0110] In some embodiments of this application, the determination module 203 described above can also be used to: determine the first relative pose information of the robot and the target object in a first time window based on the first environmental image set; determine the second relative pose information that the robot and the target object need to achieve in a second time window based on user instructions; and determine reference motion control information based on the deviation information between the first and second relative pose information and the joint space state information.

[0111] In some embodiments of this application, the second time window includes m sub-time windows, and the motion control information of the second time window includes the joint space control commands of the robot at each moment within the second time window; the aforementioned determining module 203 can also be used to: after performing the step of determining the motion control information of the robot in the second time window based on environmental reasoning information, execute the joint space control commands corresponding to each sub-time window according to the timing of the m sub-time windows to control the robot's motion; after executing the control command of the nth sub-time window in each of the m sub-time windows, obtain the actual environmental image collected by the robot in the nth sub-time window, n∈[1,2,…,m-1]; calculate the actual relative pose between the robot's end effector and the target object based on the actual environmental image; determine the joint space control commands of the robot in the (n+1)th sub-time window based on the pose error information between the actual relative pose and the expected relative pose pre-planned for the nth sub-time window.

[0112] The various modules of the information generation apparatus 200 provided in this application embodiment can realize Figures 1 to 7 It provides the functions for each step of the information generation method and can achieve the corresponding technical effects. For the sake of brevity, it will not be described in detail here.

[0113] Figure 9 The diagram shows a schematic representation of the hardware structure of an electronic device provided in some embodiments of this application.

[0114] The electronic device 30 may include a processor 301 and a memory 302 storing computer program instructions.

[0115] Specifically, the processor 301 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0116] Memory 302 may include mass storage for data or instructions. For example, and not limitingly, memory 302 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 302 may include removable or non-removable (or fixed) media. Where appropriate, memory 302 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 302 is non-volatile solid-state memory.

[0117] In certain embodiments, memory 302 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory 302 includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the information generation method in the above embodiments according to this application.

[0118] The processor 301 implements any of the information generation methods described in the above embodiments by reading and executing computer program instructions stored in the memory 302.

[0119] In one example, the aforementioned electronic device 30 may further include a communication interface 303 and a bus 310. Wherein, as... Figure 9 As shown, the processor 301, memory 302, and communication interface 303 are connected through bus 310 and complete communication with each other.

[0120] The communication interface 303 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0121] Bus 310 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count Bus (LPC), a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VESA Local Bus, VLB), or other suitable buses, or a combination of two or more of these. Where appropriate, bus 310 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0122] The electronic device can execute the information generation method in the embodiments of this application, thereby achieving the combination Figures 1 to 8 The described information generation method and apparatus.

[0123] Furthermore, in conjunction with the information generation methods in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the information generation methods in the above embodiments. Examples of computer-readable storage media include non-transitory computer-readable storage media, such as portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, etc.

[0124] Furthermore, in conjunction with the information generation methods in the above embodiments, this application embodiment can provide a computer program product for implementation. This program product is stored in a storage medium and may specifically include a computer program or instructions. When executed by a processor, the computer program or instructions implement any of the information generation methods in the above embodiments. This program product is executed by at least one processor to implement the various processes of the information generation method embodiments described above, and can achieve the same technical effects. To avoid repetition, further details are omitted here.

[0125] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0126] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this disclosure are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable read-only memory (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0127] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0128] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable information generation apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable information generation apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0129] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.< / action>

Claims

1. An information generation method, characterized in that, include: The robot acquires user commands within a first time window, as well as a first set of environmental images and joint space state information collected by the robot within the first time window. The user commands are used to instruct the robot to perform interactive operations on the target object. The user instruction, the first environmental image set, and the joint space state information are input into the environmental inference network to obtain the environmental inference information of the second time window output by the environmental inference network. The environmental inference information includes the motion state information of the reference object, the reference object includes the target object and the interactive object that has interactive behavior with the target object, and the second time window is the time window located after the first time window. Based on the environmental reasoning information, motion control information of the robot in the second time window is determined. The motion control information is used to control the robot to perform the interactive operation on the target object as instructed by the user command in the second time window.

2. The information generation method according to claim 1, characterized in that, The method further includes: The text features of the user instruction, the image features of the first environment image set, and the state features of the joint space state information are input into a multimodal neural network to obtain the multimodal fusion features output by the multimodal neural network. The multimodal neural network is used to perform interactive modeling of the text features, the image features, and the state features through a multi-head self-attention algorithm to obtain the multimodal fusion features. The step of inputting the user command, the first environmental image set, and the joint space state information into the environmental inference network, and obtaining the environmental inference information of the second time window output by the environmental inference network, includes: The multimodal fusion features are input into the environment inference network to obtain the environment inference information of the second time window output by the environment inference network.

3. The information generation method according to claim 2, characterized in that, The environmental reasoning information includes motion state information and sub-task sequences. The motion state information includes region information, depth information, semantic category feature information, and instance segmentation information. The environmental reasoning network includes a first sub-network, a second sub-network, a third sub-network, and a fourth sub-network. The step of inputting the multimodal fusion features into the environmental reasoning network to obtain the environmental reasoning information of the second time window output by the environmental reasoning network includes: The environmental change-related features in the multimodal fusion features are input into the first sub-network to obtain the regional information of the second time window output by the first sub-network. The regional information includes information representing the region where the reference object is located in the second time window. The geometrically related features in the multimodal fusion features are input into the second sub-network to obtain the depth information of the second time window output by the second sub-network. The depth information includes information representing the relative pose between the reference object and the robot within the second time window. The semantically related features in the multimodal fusion features are input into the third sub-network to obtain the semantic category feature information and instance segmentation information of the second time window output by the third sub-network; The task logic-related features in the multimodal fusion features are input into the fourth sub-network to obtain the sub-task sequence of the second time window output by the fourth sub-network. The sub-task sequence is related to the motion state information of the reference object in the second time window.

4. The information generation method according to any one of claims 1 to 3, characterized in that, Determining the robot's motion control information within the second time window based on the environmental reasoning information includes: The environmental reasoning information and multimodal fusion features are input into the action planning network to obtain the action execution sequence output by the action planning network. The action execution sequence includes the target pose information of the end effector of the robot at each moment within the second time window. The multimodal fusion features are determined based on the user command, the first environmental image set, and the joint space state information. Based on the action execution sequence, the motion control information of the robot in the second time window is determined.

5. The information generation method according to claim 4, characterized in that, The motion control information of the second time window includes the joint space control commands of the robot at each moment within the second time window; determining the motion control information of the robot in the second time window based on the action execution sequence includes: The action execution sequence, the multimodal fusion features, and the joint space state information are input into a low-rank action execution network to obtain the joint space control commands of the robot at each moment within the second time window output by the low-rank action execution network. The low-rank motion execution network is used to fuse the motion execution sequence, the multimodal fusion features, and the joint space state information to generate a latent motion embedding representation, and to denoise the latent motion embedding representation to obtain the joint space control commands of the robot at each time point within the second time window.

6. The information generation method according to claim 1, characterized in that, The second time window includes a first sub-time window and a second sub-time window located after the first sub-time window; after determining the robot's motion control information in the second time window based on the environmental reasoning information, the method further includes: Acquire the second set of environmental images collected by the robot during the first sub-time window; If it is determined from the second environmental image set and the first environmental image set that the pose of the reference object has not changed within the first sub-time window, the motion control information is adjusted according to the reference motion control information to obtain the adjusted motion control information, wherein the reference motion control information is determined based on the first environmental image set, the joint space state information and the user command; The adjusted motion control information is determined as the motion control information of the robot in the second sub-time window.

7. The information generation method according to claim 6, characterized in that, The method further includes: Based on the first environmental image set, determine the first relative pose information of the robot and the target object within the first time window; Based on the user instructions, determine the second relative pose information that the robot and the target object need to achieve in the second time window; The reference motion control information is determined based on the deviation information between the first relative pose information and the second relative pose information, as well as the joint space state information.

8. The information generation method according to claim 1, characterized in that, The second time window includes m sub-time windows, and the motion control information of the second time window includes the joint space control commands of the robot at each moment within the second time window; after determining the motion control information of the robot in the second time window based on the environmental reasoning information, the method further includes: According to the timing sequence of the m sub-time windows, execute the joint space control instructions corresponding to each sub-time window to control the robot's movement; After executing the control command for the nth sub-time window out of the m sub-time windows, the robot acquires the actual environmental image collected within the nth sub-time window, where n∈[1,2,…,m-1]. Based on the actual environment image, calculate the actual relative pose between the robot's end effector and the target object; Based on the pose error information between the actual relative pose and the expected relative pose pre-planned for the nth sub-time window, the joint space control command of the robot in the (n+1)th sub-time window is determined.

9. An information generation device, characterized in that, include: The acquisition module is used to acquire user instructions within a first time window, as well as a first set of environmental images and joint space state information collected by the robot within the first time window. The user instructions are used to instruct the robot to perform interactive operations on the target object. The spatiotemporal perception reasoning module is used to input the user command, the first environmental image set and the joint space state information into the environmental reasoning network, and obtain the environmental reasoning information of the second time window output by the environmental reasoning network. The environmental reasoning information includes the motion state information of the reference object, the reference object includes the target object and the interactive object that has interactive behavior with the target object, and the second time window is a time window located after the first time window. The determination module is used to determine the motion control information of the robot in the second time window based on the environmental reasoning information. The motion control information is used to control the robot to perform the interactive operation on the target object indicated by the user instruction in the second time window.

10. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the information generation method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the information generation method as described in any one of claims 1-8.

12. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device is able to perform the information generation method as described in any one of claims 1-8.

Images