Robot control method and system for autonomous placement of large models on a body

Abstract

The present disclosure provides a robot control method and system for large model-driven autonomous object placement, relating to the technical field of robot control, comprising: obtaining attribute information of a candidate object and an initial state of the candidate object; generating a user instruction specifying a preferred placement of the candidate object according to the initial state of the candidate object; inputting the attribute information and the user instruction into a diffusion model based on Transformer to obtain a predicted target state of object re-placement conditioned on the user instruction and the initial state; inputting the target state and the initial state into a trajectory generator based on LLM to determine the sequence of object re-placement to the target state, and then using an RRT-Connect motion planning method to generate a collision-free trajectory for the robot to re-place the object to the target state in a recursive manner.

Description

Technical Field

[0001] This disclosure relates to the field of robot control technology, specifically to a robot control method and system for autonomously placing large-scale models of objects. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] In daily life, object rearrangement involves a wide variety of tasks, such as tidying a desk, storing objects in a cabinet, and cleaning a room. This is also a fundamental skill of embodied intelligence. For robots, the purpose of object rearrangement is to move objects from their initial positions to their desired positions according to specific rules or instructions.

[0004] Most robotic object repositioning methods employ supervised learning frameworks to learn placement rules to predict target states. However, on one hand, creating training data requires extensive manual annotation, limiting the scalability and applicability of these methods. On the other hand, such methods only predict the target state of the object as a guide for object repositioning, ignoring the order and trajectory of the objects to achieve the required setup. Therefore, they can only handle relatively simple scenarios. For example, some only handle four scenarios, including circular, tower-shaped, straight-line, and tabletop settings, or only introduce three task scenarios with objects from five different categories.

[0005] Recently, large models have demonstrated a powerful ability to extract rich common-sense reasoning knowledge, leading to various applications in data generation and robotics. It's natural to consider using large models to address the two shortcomings of existing robotic object rearrangement methods mentioned above. However, existing methods either develop a framework for handling rearrangement tasks without a clear objective specification, or propose language-guided object rearrangement frameworks to predict the target state based on initial observations and language instructions; or, many researchers have applied diffusion models to object rearrangement tasks, proposing a diffusion model called Lego-Net to rearrange furniture in a room, such as tables, chairs, and sofas, and training a graph-based diffusion model to rearrange arbitrarily placed objects from a random layout to a normalized target distribution. Attempts have been made to extend diffusion models for behavioral policy learning, but these approaches do not focus on the object rearrangement task and rely heavily on large amounts of expert demonstration data. These framework methods only focus on rearranging objects into specific structures such as shape, size, and position, treating object rearrangement merely as a task of predicting the target state while ignoring the operational trajectory, thus failing to meet the diverse needs of people's daily lives. Summary of the Invention

[0006] To address the aforementioned issues, this disclosure proposes a robot control method and system for autonomous object placement driven by a large model. It proposes the DynamicDiffusion large model framework and utilizes a Transformer-based diffusion model to generate trajectory sequences for robot object repositioning. Specifically, it designs a Transformer-based diffuser to predict the target state for object repositioning and proposes an LLM-based trajectory generator to determine the order of object repositioning and generate corresponding motion trajectories for the robot.

[0007] According to some embodiments, the present disclosure adopts the following technical solutions:

[0008] A robot control method for autonomously placing objects driven by a large model includes:

[0009] Obtain the attribute information of the candidate object and the initial state of the candidate object;

[0010] Based on the initial state of the candidate objects, generate user instructions specifying the placement preferences of the candidate objects;

[0011] The attribute information and the user command are input into the Transformer-based diffusion model to obtain the predicted target state of the object rearrangement based on the user command and the initial state.

[0012] The target state and initial state are then input into an LLM-based trajectory generator to determine the order in which the objects are rearranged to the target state. Then, the RRT-Connect motion planning method is used to recursively generate a collision-free trajectory for the robot to rearrange the objects to the target state.

[0013] According to some embodiments, the present disclosure adopts the following technical solutions:

[0014] A robot control system for autonomously placing objects using a large model includes:

[0015] The data acquisition module is used to acquire the attribute information of the candidate object and the initial state of the candidate object;

[0016] The initialization module is used to generate user instructions specifying the placement preferences of candidate objects based on their initial states.

[0017] The prediction module is used to input the attribute information and the user command into the Transformer-based diffusion model to obtain the predicted target state of the object rearrangement based on the user command and the initial state.

[0018] The sequence and trajectory generation module is used to input the target state and initial state into the LLM-based trajectory generator to determine the order in which the objects are rearranged to the target state. Then, the RRT-Connect motion planning method is used to recursively generate a collision-free trajectory for the robot to rearrange the objects to the target state.

[0019] According to some embodiments, the present disclosure adopts the following technical solutions:

[0020] A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the robot control method and system for autonomous placement of a large model-driven object.

[0021] According to some embodiments, the present disclosure adopts the following technical solutions:

[0022] An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute the robot control method and system for autonomous placement of large model-driven objects.

[0023] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0024] This paper discloses a robot control method for autonomous object placement driven by a large model. It proposes the DynamicDiffusion framework, a novel diffusion model-based approach that generates robot trajectory sequences to perform language-guided object repositioning tasks. Specifically, a transformer-based framework is proposed as the diffusion model for prediction, which predicts the target state conditioned on the initial state and user commands. An LLM-based trajectory generator is also designed to determine the object repositioning order and generate corresponding motion trajectories. This method not only predicts the target state of the objects but also generates robot motion trajectories in a reasonable order. Experiments demonstrate that DynamicDiffusion outperforms all baseline methods in both simulation and real-world scenarios.

[0025] This disclosure presents a robot control method for autonomous object placement driven by a large model. Addressing the problem that existing methods only treat object repositioning as a task for predicting the target state while ignoring the operation trajectory, the method proposed in this disclosure not only predicts the target state but also generates the trajectory and sequence for achieving the target state. Attached Figure Description

[0026] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0027] Figure 1 This is a schematic diagram illustrating the implementation process of the method according to an embodiment of this disclosure;

[0028] Figure 2 This is an example of an object library according to an embodiment of this disclosure;

[0029] Figure 3 This is the architecture of the transformer-based diffusion model in this embodiment of the disclosure;

[0030] Figure 4 This is a schematic diagram of a method for generating sequential trajectories according to an embodiment of the present disclosure;

[0031] in, Figure 4 (a) in the text describes the specific rules for discretizing the workspace and setting state values ​​for each cell;

[0032] , Figure 4 (b) in the diagram represents the process of generating a collision-free trajectory;

[0033] Figure 5 This is a diagram illustrating the prompt fine-tuning of embodiments of this disclosure from difficult to easy.

[0034] Figure 6 The position and rotation prediction results are from the simulation experiments of this embodiment.

[0035] Figure 7 This is a qualitative example of the predicted target in an embodiment of this disclosure;

[0036] Figure 8 Visualization of sequential trajectories generated in the OmniGibson simulator by the LLM-based trajectory generator of this disclosure embodiment;

[0037] Figure 9 This is a real-world robot system as described in the embodiments of this disclosure;

[0038] Figure 10 The images show the real-world robot rearrangement results generated by DynamicDiffusion in this embodiment of the disclosure, with each row representing a test case. Detailed Implementation

[0039] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0040] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0041] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0042] Terminology Explanation

[0043] GPT-4 is an AI language model that allows you to converse and ask questions. For example, if you ask GPT-4 to write a description of a city, it will tell you the city's location, notable historical figures associated with it, and some historical information.

[0044] DynamicDiffusion Framework: The overall dynamicdiffusion framework uses GPT-4 to help parse some difficult-to-understand instructions and generate usable datasets. GPT-4 can be regarded as part of the entire dynamicdiffusion framework.

[0045] Example 1

[0046] One embodiment of this disclosure provides a robot control method for autonomously placing large-scale models of objects, including:

[0047] Step 1: Obtain the attribute information of the candidate object and the initial state of the candidate object;

[0048] Step 2: Based on the initial state of the selected object, generate user instructions specifying the placement preference for the candidate object;

[0049] Step 3: Input the attribute information and the user command into the Transformer-based diffusion model to obtain the predicted target state of the object rearrangement based on the user command and the initial state.

[0050] Step 4: Input the target state and initial state into the LLM-based trajectory generator to determine the order in which the objects are rearranged to the target state. Then, use the RRT-Connect motion planning method to recursively generate a collision-free trajectory for the robot to rearrange the objects to the target state.

[0051] As one embodiment, this disclosure proposes the DynamicDiffusion framework, a novel diffusion-based approach that generates robot trajectory sequences to perform language-guided object repositioning tasks, such as... Figure 1 As shown, a transformer-based framework is proposed as a diffusion model for prediction, which predicts the target state conditioned on the initial state and user instructions. Then, an LLM-based trajectory generator is designed to determine the order of object rearrangement and generate the corresponding motion trajectory.

[0052] The DynamicDiffusion framework constructs a diffusion model-based framework for instruction-guided object repositioning tasks. It can not only predict the target state of the object but also generate the robot's motion trajectory in a reasonable order. First, this disclosure provides a brief definition of the robot object repositioning problem. Then, it discusses the specific implementation process of a robot control method driven by a large model for autonomous object repositioning.

[0053] Problem definition:

[0054] Robotic object repositioning aims to take the initial state of the scene as input and output its target state. The state is typically represented as the attribute encoding of candidate objects, including position, rotation direction, category, and 3D bounding box. Existing work mainly focuses on predicting the target state, while neglecting how to execute effective actions to achieve the target state. In contrast, this disclosure focuses on target state prediction and operation trajectory generation from the perspective of robot repositioning applications. This disclosure represents the initial state of the object scene as a set of object attribute information:

[0055] O = {o1, o2, ..., o N},o i =(p i ,r i ,c i ,b i )

[0056] Among them, o i Represents the i-th object. Let r represent the 3D position of the i-th object. i = (cosθ, sinθ) represents the rotation of the object about the z-axis, c i o i Category This indicates the size of the 3D bounding box. i Standardized to the range [-1, 1]. Given the initial state O0 of the scene, the goal is to predict the target state and operation trajectory conditioned on language instructions, which can be modeled as follows:

[0057] Seq,Traj,O t =Γ{O0,I}

[0058] Where Γ represents the method of rearranging the robot object, I∈R k These are language instructions. Seq = (o3, o5, o2, ...) represents the order in which the objects are manipulated, and Traj = (τ3, τ5, τ2, ...) consists of the motion trajectory corresponding to each object. t Represents the predicted target state s t Observations.

[0059] The following describes the specific implementation process of a robot control method for autonomously placing objects driven by a large model, as disclosed in this paper:

[0060] 1. Data Generation

[0061] The DynamicDiffusion framework views the prediction of object repositioning guided by instructions as a diffusion process. This diffusion process involves gradually adding noise to the initial state of candidate objects, and then recovering the original state from the noisy data through a denoising process.

[0062] Essentially, object repositioning methods based on diffusion models learn repositioning knowledge from training data. Therefore, the performance of object repositioning largely depends on the quantity and diversity of the training data. DynamicDiffusion provides a novel data generation method to generate diverse repositioning samples. Such repositioning allows us to automatically generate large-scale datasets. Furthermore, the data generation method is built upon a simulator with a realistic physics engine, ensuring the physical validity of the generated samples. Specifically, the dataset is created through three steps:

[0063] (I): Selecting an object

[0064] OmniGibson was used as the simulation environment for data generation, and more than 400 household items from BEHAVIOR-1K were used as candidate objects (see attached). Figure 2 In the OmniGibson simulator, object attribute information, such as position, size, category, and 3D bounding box, can be easily obtained. GPT-4, as an object nomination generator, takes the classification information of all candidate objects as input and then automatically selects N semantically relevant objects to generate the initial scene.

[0065] (II): Generating instructions and placing layout

[0066] For each scenario state O = {o1, o2, ..., o...} NThe process involves using GPT-4 to generate various user-defined placement preferences. Each instruction I and object attributes (category and 3D bounding box) are then input into GPT-4, which outputs the desired placement layout L(O,I)={o1,O2,…,O2}. N}

[0067] (III): Checking physical validity in the simulator

[0068] Existing experiments have revealed that GPT-4 sometimes generates physically illogical placement layouts. This is because GPT-4 has limited spatial reasoning capabilities, making realistic physical interactions impossible. Therefore, this disclosure proposes further examining the physical validity of layouts generated by GPT-4 within the OmniGibson simulator. Specifically, the generated placement layout is rendered in OmniGibson, and the simulator's API interface is used to check whether all objects in the scene are stably placed and without collisions. If conflicts occur, the corresponding placement layout is not recommended. Thus, invalid samples are filtered out, thereby improving object repositioning performance.

[0069] Finally, the dataset was divided into three groups: 80% for training, 10% for validation, and 10% for testing.

[0070] 2. A Transformer-based diffusion model predicts the target state of object rearrangement given instructions and initial states.

[0071] To predict the target state of object rearrangement conditioned on instructions and initial states, a transformer-based diffusion model was constructed. On one hand, the transformer can learn to pay more attention to relevant objects specified in the linguistic instructions through an attention mechanism. On the other hand, it can learn the semantic relationships between objects. Figure 3 The architecture of the diffusion model, consisting of three main components, is explained. These include:

[0072] (I) User instruction encoder

[0073] To encode the input language user instructions, a representative text token is extracted from the user instructions using an instruction encoder, represented as follows. In practice, a pre-trained text encoder from CLIP is used as the instruction encoder because CLIP, in terms of network-scale processing and training, can largely align features, thus providing strong priors for the underlying semantics, such as color, text, and category. The input language instructions are ultimately encoded as a 512-dimensional vector.

[0074] (II) Object Attribute Encoder

[0075] To obtain each object o i The attribute information is encoded and processed independently using a position encoder.i The attributes, including category c i Position p i Rotation r i and 3D bounding box b i Because of o i Different attributes have different dimensions, so an additional linear layer is used to project the encoding onto the same dimension. The extracted object attribute features are then concatenated and processed through a two-layer MLP to finally output object attribute labels.

[0076] (III) Conditional Target State Diffusion Model

[0077] The conditional target state diffusion model is a Transformer model, which includes a forward diffusion process and a backward diffusion process. Forward diffusion is the process of adding Gaussian noise, and backward diffusion is the process of denoising.

[0078] The forward diffusion process is modeled as a Markov chain, where the initial sample x t Gradually adding Gaussian noise: The reverse process is the denoising process, in which noisy samples are gradually denoised according to the learned Gaussian posterior.

[0079]

[0080] Where, β t Let be the variance of a Gaussian distribution, and satisfy β1 < β2 < ... < β T .

[0081] The reverse process is the denoising process, in which noisy samples are gradually denoised according to the learned Gaussian posterior:

[0082]

[0083] During the forward pass, the initial state of the object is noise-added and then encoded by the object attribute encoder. Language user commands are processed by the language encoder. Then, object encoding F is executed. o and language encoding F t The Hadamard product is used to align the two types of features; the Hadamard product is used to multiply corresponding elements of the two matrices; following the denoising process, a noisy kernel is used. Let's learn the denoising process, where σ is the noise parameter. Therefore, the loss function is defined as:

[0084]

[0085] Joint distribution Where n(σ) is the distribution of the noise parameters, and q0(X) is the discrete uniform distribution of the training samples.θ This represents the model disclosed herein, X and L represents the target state and the disturbed state. dn It is defined using the equilibrium parameter λ:

[0086]

[0087] 3. An LLM-based trajectory generator determines the order in which objects are rearranged to the target state and the collision-free trajectory, including:

[0088] Step 1: In order to plan a reasonable motion trajectory for the robot, the initial scene is represented as a 3D voxel map. The 3D voxel map is created as a W×L×H workspace and uniformly discretized into w×l×h units according to the discretization resolution. Each unit is assigned a specific value of 0, 1 or 2, corresponding to 3 different spatial states.

[0089] like Figure 4 As shown in (a), the specific rules for setting state values ​​are: 1) Spaces without any objects are assigned a value of 0 (displayed in gray); 2) Spaces occupied by the manipulated object are assigned a value of 1 (displayed in blue); 3) Spaces occupied by obstacles are assigned a value of 2 (displayed in red). A 3D voxel map is created by projecting RGB-D observations onto the 3D work area using hand-eye calibration.

[0090] Step 2: Determine the order of object operations based on the object's initial and target states.

[0091] Specifically, using GPT-4 as the inference engine, it initializes the object's initial state O. init ={o1,o2,…,o N} and target state O goal ={o1,o2,…,o N The order in which the manipulated objects are taken as input and output. i ={p i ,r i ,c i ,b i} is the i-th object to be operated on, where {p i ,r i ,c i ,b i} represent its position, rotation, category, and 3D bounding box, respectively.

[0092] Step 3: Recursively generate collision-free trajectories

[0093] To enable the robot to move a single object from an initial state to a target state without collision, RRT-Connect, a motion planning method that has demonstrated good performance in object repositioning tasks, is employed. Unlike other methods, this disclosure aims to generate trajectories for all objects requiring manipulation using RRT-Connect before execution. To this end, a motion planning strategy is designed to recursively generate collision-free trajectories for robot object repositioning.

[0094] like Figure 4 As shown in (b), given a number of objects to be manipulated, RRT-Connect first generates a collision-free trajectory for the first object to be manipulated based on the 3D voxel map of the initial scene created in the above process. Next, the previously moved object is projected from its initial state to its target state, and the voxel map is updated. Simultaneously, the second object to be manipulated now becomes the current object to be manipulated, so the value of the cell occupied by this object is changed from 2 to 1, and the voxel map is updated accordingly. RRT-Connect can now recursively generate collision-free trajectories for all objects to be manipulated based on the updated voxel map, thus planning a new trajectory for the second object.

[0095] As one embodiment, this disclosure uses a VLM (Virtual Model) to fine-tune the diffusion model from difficult to easy prompts. The proposed diffusion model is trained using automatically generated data. During inference, the initial state and specific instructions are input into the diffusion model to predict the target state. In the simulation environment, object properties can be easily obtained through an API interface. In real-world scenarios, OWLv2 is used as an object detector to obtain object properties. Although this strategy is widely used in recent object repositioning methods, it performs poorly in real-world repositioning tasks due to the limited inference capabilities of object detectors. In fact, open-vocabulary object detectors can only understand relatively simple words (e.g., apple, "cup," "knife"), and their performance deteriorates further if the prompts are more complex (e.g., selecting the most potentially dangerous object). Notably, existing instruction-guided object repositioning methods cannot well understand complex instructions.

[0096] VLMs have demonstrated powerful reasoning capabilities. Therefore, this disclosure proposes using GPT-4V to enhance our method's ability to rearrange different instructions and objects. Figure 5As shown, a difficulty-to-easy cue adjustment strategy was designed based on GPT-4V. Its core idea is to reinterpret difficult-to-understand cuees into easier-to-understand ones using GPT-4V. The cue adjustment strategy consists of two components. First, the input cue of the diffusion model is adjusted through language inference using GPT-4V. Given the original cue and object attributes (including category, position, rotation, and size, provided on demand), a query cue is constructed according to a hand-designed template and then forwarded to GPT-4V, ultimately outputting an adjusted cue. Second, the input cue of the open-vocabulary object detector OWLv2 is adjusted through visual inference using GPT-4V. In the inference case, if OWLv2 outputs a prediction with a low confidence value, the input cue will be refined through GPT-4V. Figure 5 As shown in the right half, the scene image and the designed cues are input into the GPT-4V. The GPT-4V reinterprets the complex cues as simple representations of four object attributes (size, shape, position, and color).

[0097] Simulation Experiment

[0098] Experimental setup

[0099] Simulation and real-world experiments were conducted to evaluate the method disclosed herein from the perspectives of target state prediction and repositioning execution. For simulation testing, different test scenarios were created in the OmniGibson simulator, while in real-world testing, the UR5 robotic arm was used to perform the repositioning task.

[0100] In the experiments, four types of rearrangement tasks were designed: linear structure, circular structure, table setting, and unconstrained rearrangement. In the linear and circular structure tasks, the goal was to arrange a group of objects into a linear or circular structure according to specific instructions. In the table setting task, the goal was to arrange tableware into an organized structure according to language instructions. In everyday life scenarios, object rearrangement tasks are more complex than the above types. Therefore, a wide range of test cases with different organization rules (excluding the test cases mentioned above) were designed, referred to as unconstrained rearrangement. Samples from the test dataset were used in the simulation experiments, while different rearrangement tasks were manually created in the real world. For each test, high-level language instructions were first randomly created, and then selected candidate objects were used and the initial state was set accordingly.

[0101] Evaluation indicators

[0102] This disclosure uses both subjective and objective metrics to evaluate the proposed method. In a simulation scenario, the predicted object pose P is measured. pred ={p i ,r i} and true posture P gt ={pj ,r j The deviation between} is expressed as

[0103]

[0104]

[0105] Where N represents the number of objects in a scene, p i and r i These represent the position and rotation angle of an object, respectively. Note that smaller distances indicate better prediction performance.

[0106] In real-world scenarios, a human-assisted strategy is employed to evaluate task performance. Humans judge task success based on their own observations. The rearrangement task is considered successful if the object manipulations occur without collisions and the post-processing scene state satisfies the given instructions.

[0107] Benchmark Method

[0108] StructFormer and StructDiffusion are two representative language-guided object repositioning methods. StructFormer is a Transformer-based learning network that generates repositioning plans based on language commands. StructDiffusion employs a diffusion-based model to generate target placements conditioned on language instructions. These two methods are used as benchmarks for comparison, and comparative experiments are conducted on various repositioning tasks.

[0109] result

[0110] Extensive simulation experiments were conducted to evaluate our method. Figure 6 The evaluation results of our method's prediction of the target state are shown, including the positional and rotational deviations between the predicted target state and the true value. It can be seen that our method consistently outperforms the comparison methods in all three types of tasks, demonstrating the effectiveness of the repositioning method in predicting the target's placement state.

[0111] To conduct a more in-depth analysis of the repositioning performance, the success rates of target state prediction and repositioning execution were further evaluated separately. In simulation testing, the success of target state prediction was determined using a threshold-based scheme, where predictions with distances less than a threshold were considered successful. Specifically, the threshold for position distance was set to 0.03m, and the threshold for rotation distance was set to 10. Based on successful target state predictions, physical manipulations were performed on the objects in the OmniGibson simulator using trajectories generated by an LLM-based trajectory generator. If all objects moved to the correct positions without any collisions, the repositioning was considered successful.

[0112] The comparative results for target state prediction and rearrangement execution are reported in different columns of Table 1. The results clearly demonstrate that our method outperforms the baseline method in both target state prediction and rearrangement action execution. Furthermore, a significant drop in the success rate of the baseline method was observed from the target state prediction stage to the rearrangement execution stage. In contrast, our method shows a slight decrease in success rate, demonstrating the effectiveness of the LLM-based trajectory generator in generating sequential trajectories to complete the desired operation.

[0113] Table 1 shows the success rates of different methods in target state prediction and repositioning execution.

[0114]

[0115] The method disclosed herein performs well on all four types of tasks, while the baseline method fails to handle unconstrained rearrangement tasks. These tasks include Line structure (straight lines), Circle structure (circles), Dining tablesetting (dining tables), and Unconstrained rearrangement. This is because the baseline method uses relatively simple data to train the model, which only contains rearrangement samples with specific structures (e.g., lines, circles). In contrast, an LLM-based data generation method automatically generates diverse rearrangement samples, increasing the quantity and diversity of training data, thereby improving the generalization ability of the method disclosed herein across different rearrangement scenarios.

[0116] exist Figure 7 The image shows a qualitative example of predicted target placement generated by the method of this disclosure, which is then rendered in the OmniGibson simulator. It can be seen that this method facilitates high-quality target placement prediction across different scenes and demonstrates good generalization ability in handling instruction-guided object repositioning tasks. Furthermore, in Figure 8 The image visualizes the sequential trajectory generated by the LLM-based trajectory generator, demonstrating the ability of the method disclosed herein to generate reasonable and collision-free trajectories in different scenarios.

[0117] ablation experiment

[0118] Ablation experiments were conducted to evaluate the effects of different components of DynamicDiffusion.

[0119] Random operations and LLM-based operations

[0120] A highlight of this disclosed method is its LLM-based approach design for generating sequential trajectories and determining the order of object operations based on the initial and target states of the objects. To demonstrate its effectiveness, a variant of our method was created, replacing the LLM-based trajectory generator simply with the RRT-Connect planner. This variant does not consider the order of object operations and randomly selects objects for operation. Our method was compared with this variant in a series of simulated repositioning tasks performed in a relatively chaotic scene, where multiple objects needed to be manipulated. Finally, the full method achieved an 83% success rate, while the variant achieved a 63% success rate. These results demonstrate that the LLM-based sequential trajectory generator can produce reasonable sequence of trajectories, thus ensuring more reliable object repositioning.

[0121] Generalize to unknown objects and instructions

[0122] To verify the generalization ability of DynamicDiffusion, we created a set of test cases in OmniGibson using objects and instructions that were seen or not seen in the training dataset. Our method achieved an overall success rate of 78%, very close to the 80% success rate reported in Table 1. The results demonstrate that our method has good generalization ability across different objects and instructions. This cross-object and instruction generalization ability is mainly attributed to the proposed difficulty-to-easy cue adjustment strategy. Therefore, a simplified version of the full method was created, in which the difficulty-to-easy cue adjustment strategy was removed. Comparative results are provided in Table 2, demonstrating the effectiveness of the proposed difficulty-to-easy cue adjustment strategy in handling scenarios with unknown objects and instructions.

[0123] Table 2 shows the effect of the LLM-based trajectory generator and the adjustment of the prompts from difficult to easy.

[0124] Random manipulation 63％ LLM-based trajectory generator 83％ w / o hard-to-easy prompt tuning 61％ w / hard-to-easy prompt tuning 78％

[0125] Here, Random manipulation represents random operation, LLM-based trajectory represents an LLM-based path generator, w / o hard-to-easy prompt tuning means no hard-to-easy prompt tuning is used, and w / hard-to-easy prompt tuning means hard-to-easy prompt tuning is used.

[0126] Real experiment

[0127] Robotic rearrangement system.

[0128] To conduct real-world object repositioning tests, a repositioning system equipped with a UR5 robotic arm and suction cups was constructed, such as... Figure 9As shown. For real-world repositioning tasks, OWLv2 is used to obtain the category and position of each object, and DenseFusion is used to obtain the 3D bounding box and rotation. To complete the repositioning task, two primitive actions, pick and place, are predefined. SuctionNet is used to implement the suction cup action. For the placement action, pos = (x, y, z + 3 cm) is defined as the position of the object being placed, where (x, y, z) is the predicted target position of the object. Furthermore, the rotation along the z-axis for both pick and place actions is calculated based on the object's initial state and the predicted target state.

[0129] Performance evaluation of object repositioning.

[0130] To evaluate in real-world scenarios, different test cases were manually created. Similar to simulation testing, our method was quantitatively compared to two benchmark methods by executing four types of tasks designed for real-world use. The results listed in Table 3 demonstrate the superiority of our method over the baselines. Furthermore, our method exhibits good simulation-to-real-world transfer capabilities. Taking the most challenging of the four tasks as an example: our method achieved an overall success rate of 80% in the simulated unconstrained rearrangement task and 75% in real-world testing. Figure 10 This presentation showcases examples of real-world repositioning tests, demonstrating reliable robot repositioning performance across various scenarios. In the demonstrations, we provide an in-depth evaluation of our method through a series of test cases, showcasing not only overall repositioning performance but also demonstrating that our method handles unknown objects and complex instructions well, generating sequential trajectories in relatively cluttered scenes to enable stable robot execution.

[0131] Table 3 Success Rate of Different Methods

[0132]

[0133] Example 2

[0134] One embodiment of this disclosure provides a robot control system for autonomously placing large-scale model-driven objects, including:

[0135] The data acquisition module is used to acquire the attribute information of the candidate object and the initial state of the candidate object;

[0136] The initialization module is used to generate user instructions specifying placement preferences for candidate objects based on the initial state of the selected objects.

[0137] The prediction module is used to input the attribute information and the user command into the Transformer-based diffusion model to obtain the predicted target state of the object rearrangement based on the user command and the initial state.

[0138] The sequence and trajectory generation module is used to input the target state and initial state into the LLM-based trajectory generator to determine the order in which the objects are rearranged to the target state. Then, the RRT-Connect motion planning method is used to recursively generate a collision-free trajectory for the robot to rearrange the objects to the target state.

[0139] Example 3

[0140] One embodiment of this disclosure provides a non-transitory computer-readable storage medium for storing computer instructions. When these computer instructions are executed by a processor, they implement the robot control method for autonomous placement of a large model-driven object.

[0141] Example 4

[0142] One embodiment of this disclosure provides an electronic device, including a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute a robot control method for autonomously placing a large model-driven object.

[0143] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0144] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0145] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A robot control method for autonomously placing objects driven by a large model, characterized in that, include: Obtain the attribute information of the candidate object and the initial state of the candidate object; The attribute information of the candidate object includes position, 3D bounding box, rotation and category, and the initial state of the candidate object is represented as a set of attribute information of the candidate object; Based on the initial state of the candidate objects, generate user instructions specifying the placement preferences of the candidate objects; The attribute information and the user command are input into a Transformer-based diffusion model to obtain the predicted target state of object repositioning based on the user command and the initial state. The Transformer-based diffusion model includes a speech command encoder and an object attribute encoder. The input speech user command is encoded, and the command encoder extracts text tags from the user command. The input attribute information is encoded, and the position encoder independently extracts object attribute features. Different attributes have different dimensions. An additional linear layer is used to project the encoding onto the same dimension. The extracted object attribute features are concatenated and then processed by an MLP to finally output object attribute tags. The target state and initial state are then input into an LLM-based trajectory generator to determine the order in which the objects are rearranged to the target state. Then, the RRT-Connect motion planning method is used to recursively generate a collision-free trajectory for the robot to rearrange the objects to the target state.

2. The robot control method for autonomously placing a large model-driven object as described in claim 1, characterized in that, The process of repositioning objects guided by instructions to predict the target is regarded as a diffusion model. The diffusion process is to gradually add noise to the initial state of the candidate object, and then restore the original state from the noisy data through the denoising process.

3. The robot control method for autonomously placing a large model-driven object as described in claim 1, characterized in that, The Transformer-based diffusion model includes forward diffusion and backward diffusion. Forward diffusion is the process of adding Gaussian noise, while backward diffusion is the process of denoising. In the forward process, the object's attribute information is denoised and then encoded by the object attribute encoder. The user's command is processed by the language encoder. Then, the Hadamard product of the object encoding and the language encoding is performed to align the two types of features and output the predicted target state of the object repositioning conditioned on the user command and the initial state.

4. The robot control method for autonomously placing a large model-driven object as described in claim 1, characterized in that, The steps for generating collision-free trajectories include: First, representing the initial scene as a 3D voxel map, which is then created as... The workspace is uniformly discretized according to the discretization resolution. Each unit is assigned a specific value of 0, 1, or 2, corresponding to three different spatial states.

5. The robot control method for autonomously placing a large model-driven object as described in claim 4, characterized in that, The initial and target states of the objects are taken as input and the order of the manipulated objects is output. In the created 3D voxel map, given all the objects to be manipulated, RRT-Connect is used to generate the collision-free motion trajectory of the first object to be manipulated. Then, the previous moving object is projected from its initial state to its target state and the voxel map is updated. At the same time, the second object to be manipulated becomes the current object to be manipulated. This process is repeated to generate collision-free trajectories for all the objects to be manipulated.

6. A robot control system for autonomously placing objects using a large model, characterized in that, Specifically, the robot control method for autonomous placement of a large model-driven object as described in any one of claims 1-5 includes: The data acquisition module is used to acquire the attribute information of the candidate object and the initial state of the candidate object; The initialization module is used to generate user instructions specifying the placement preferences of candidate objects based on their initial states. The prediction module is used to input the attribute information and the user command into the Transformer-based diffusion model to obtain the predicted target state of the object rearrangement based on the user command and the initial state. The sequence and trajectory generation module is used to input the target state and initial state into the LLM-based trajectory generator to determine the order in which the objects are rearranged to the target state. Then, the RRT-Connect motion planning method is used to recursively generate a collision-free trajectory for the robot to rearrange the objects to the target state.

7. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement a robot control method for autonomous placement of a large model-driven object as described in any one of claims 1-5.

8. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform a robot control method for autonomous placement of a large model driven object as described in any one of claims 1-5.

Images