Robot simulation task generation method, medium, and device

Abstract

This application provides a method, medium, and device for generating robot simulation tasks, belonging to the field of data processing technology. The method includes: acquiring a natural language task description; extracting task requirement information of the simulation task from the natural language task description, the task requirement information including the target object, associated objects, and positional relationships between the target object and associated objects that the robot needs to grasp; querying a target object model matching the target object from a preset object library; querying a target scene matching the associated objects and the target workspace of the target scene from a preset scene library; forming layout constraint information of the target object model in the target workspace based on the positional relationships and the size of the target object and interfering objects; and generating at least one simulation task conforming to the task requirement information under the layout constraint information. This application can improve the efficiency and quality of robot simulation task generation.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, medium and device for generating robot simulation tasks. Background Technology

[0002] In the field of robotics, building task-driven intelligent systems is the core of achieving autonomous decision-making and operation for robots, and this process heavily relies on a large number of diverse, high-quality simulation tasks as a training foundation. These simulation tasks need to simulate the interaction relationships between objects, spatial layout, and target conditions in real-world scenarios, providing data support for the robot's reinforcement learning, imitation learning, and motion planning training.

[0003] Traditional methods for generating simulation tasks typically employ several approaches, including relying on fixed templates, manual construction, or behavior learning sampling. Fixed template generation is based on a pre-defined scene framework and task pattern, generating new tasks by replacing a few parameters; for example, in the "item handling" template, only the item type is changed. Manual construction relies on full human involvement, from scene selection and object placement to setting target conditions, all done manually to ensure the task meets specific requirements. Behavior learning sampling involves recording existing task execution processes in real-world or simulation environments, extracting and reproducing similar tasks.

[0004] However, all of these modes require a lot of manual labor to set the robot's grasping objects and the spatial environment layout of the grasping scene one by one. Their task construction efficiency is low, and it is difficult to generate diverse scenes in batches, which seriously restricts the large-scale application of simulation tasks. Summary of the Invention

[0005] The purpose of this application is to provide a method, medium, and device for generating robot simulation tasks to solve at least one of the above-mentioned technical problems.

[0006] In a first aspect, this application provides a method for generating robot simulation tasks, the method comprising:

[0007] Obtain a natural language task description, and extract the task requirement information of the simulation task from the natural language task description. The task requirement information includes the target object, associated objects, and positional relationship between the target object and associated objects that the robot needs to grasp.

[0008] Retrieve target object models that match the target object from a pre-defined item database;

[0009] Retrieve the target scene that matches the associated object and the target workspace of the target scene from the preset scene library;

[0010] Based on the positional relationship, the layout constraint information of the target object model in the target work space is formed, and at least one simulation task that meets the task requirements information is generated under the layout constraint information.

[0011] Optionally, the step of forming the layout constraint information of the target object model in the target work space based on the positional relationship includes:

[0012] A first number of interfering object models are determined based on the target object and the associated object;

[0013] Based on the positional relationship, layout constraint information for the target object model and the interference object model is generated in the target work space.

[0014] Optionally, determining a first number of interfering object models based on the target object and the associated objects includes:

[0015] The projected area of ​​the target object model on the surface of the associated object is determined based on the target object, associated objects, and associated relationships.

[0016] The constraint area of ​​the interfering object is determined based on the projected area and the associated object;

[0017] Based on the target object and the associated object, multiple candidate objects matching the target scene are identified;

[0018] Based on the constraint area, a first number of interference models are determined from multiple candidates.

[0019] Optionally, the layout constraint information requires that there be no collision or spatial overlap between the target object and the interfering object;

[0020] The step of generating at least one simulation task that conforms to the task requirements under the layout constraint information includes:

[0021] Based on the positional relationship, the first pose information of the target object model and the second pose information of each interference object are set in the target work space to form the spatial layout information of the target object and interference objects in the target work space.

[0022] Detect whether the spatial layout information satisfies the layout constraint information; if it does, generate the corresponding simulation task.

[0023] If the conditions are not met, the process returns to re-execute the setting of the first pose information of the target object model and the second pose information of each interfering object in the target work space based on the positional relationship, until the layout constraint information is met.

[0024] Optionally, the layout constraint information requires that there be no collision or spatial overlap between the target object and the associated object;

[0025] The step of generating at least one simulation task that conforms to the task requirements under the layout constraint information includes:

[0026] Based on the positional relationship, the target object model and the interference object model are randomly set in the target work space to form multiple sets of spatial layout information of target objects and interference objects;

[0027] Detect whether each group of spatial layout information satisfies the layout constraint information;

[0028] The spatial layout information that satisfies the layout constraints is stored in a structured format to generate the corresponding simulation task.

[0029] Optionally, the associated objects include initial associated objects and target associated objects; the positional relationships include initial positional relationships and target positional relationships; the workspace includes the size parameters and position center point parameters of each associated object;

[0030] The step of extracting the task requirement information of the simulation task from the natural language task description includes:

[0031] The pre-defined large language model is invoked to perform dependency parsing on the natural language task description and extract the predicates.

[0032] Based on the predicate, the target object, initial associated object, target associated object, initial positional relationship between the target object and the initial associated object, and target positional relationship between the target object and the target associated object are determined from the natural language task description;

[0033] Based on the target object, initial associated object, target associated object, initial positional relationship, and target positional relationship, a structured language file is formed to reflect the task requirements.

[0034] Optionally, retrieving a target object model matching the target object from a preset item database includes:

[0035] Retrieve item tag information that matches the target item from a preset item database;

[0036] When multiple matching item tag information exists, obtain the item description information for each item tag information, and analyze the scene matching between each item description information and the natural language task description;

[0037] The target object model is determined from multiple item tag information based on the scene matching.

[0038] Optionally, retrieving the target scene matching the associated object and the target workspace of the target scene from a preset scene library includes:

[0039] Based on the semantic understanding of the natural language task description, a target scene matching the task requirement information is retrieved from the scene database;

[0040] Retrieve a basic workspace from a pre-defined scene library that matches the associated object and the target scene;

[0041] Based on the semantic understanding of the natural language task description, the location and size information of the basic task space are configured to form the target task space.

[0042] In a second aspect, this application provides a computer-readable storage medium storing executable instructions that, when executed by a processor, cause the processor to perform the method described in any embodiment of this application.

[0043] In a third aspect, this application provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to perform the method as described in any embodiment of this application.

[0044] The robot simulation task generation method, medium, and device in this application automatically extract task requirements, match models, and layout constraints based on natural language task descriptions, reducing the manual scene setting and layout adjustments required in traditional methods and improving the efficiency and quality of simulation task generation. Secondly, the extraction process from natural language to target objects, related objects, and positional relationships ensures semantic matching between the virtual model and the task description, avoiding semantic discrepancies and guaranteeing semantic consistency. Furthermore, by using boundary constraints and layout rules of the target workspace, combined with positional relationships, this application ensures that the virtual scene conforms to physical logic (e.g., objects do not float or penetrate), enhancing the realism of the simulation task and strengthening the scene's rationality. Additionally, this application can generate multiple constraint-compliant simulation tasks based on the same natural language description (e.g., the layout of cups and tables in different positions), providing the robot with diverse training data and enhancing its adaptability. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation on the scope of this application.

[0046] Figure 1 This is a flowchart illustrating a robot simulation task generation method in one embodiment;

[0047] Figure 2This is a flowchart illustrating the process of extracting task requirement information of a simulation task from a natural language task description in one embodiment.

[0048] Figure 3 This is a flowchart illustrating the process of retrieving a target object model that matches the target object from a preset item database in one embodiment.

[0049] Figure 4 This is a schematic diagram of the process of querying a target scene that matches an associated object from a preset scene library and the target workspace of that target scene in one embodiment.

[0050] Figure 5 This is a flowchart illustrating the process of determining a first number of interfering object models based on the target object and related objects in one embodiment.

[0051] Figure 6 This is a flowchart illustrating the process of generating at least one simulation task that meets task requirements under layout constraint information in one embodiment.

[0052] Figure 7 This is a schematic diagram of the structure of an electronic device in one embodiment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] All terms used in this application (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0055] For example, the terms "first," "second," etc., used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element.

[0056] For example, the terms "comprising" or "including" used in this application indicate the presence of features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0057] This application provides a method for generating robot simulation tasks, combined with Figure 1 As shown, the method includes:

[0058] Step 110: Obtain the natural language task description and extract the task requirement information of the simulation task from the natural language task description.

[0059] In this embodiment, the natural language task description is descriptive information expressed in natural language to instruct the robot to perform relevant simulation tasks. It includes the object to be manipulated (i.e., the target object) and the specific manipulation task performed on that object. Furthermore, it may include additional information such as a description of the target object itself and environmental characteristics. This descriptive information may include the target object's color, size, and dimensions, while the environmental characteristics may include the target object's location and information about other objects in its environment. The natural language task description can be actively input by the user or provided by a pre-set task library, with the electronic device automatically retrieving the description from the library.

[0060] A simulation task represents a task that simulates a real-world scenario in a virtual environment for robot training and testing. This simulation task can be a training task for a robot to grasp and place objects. The simulation task includes a virtual scene, virtual objects (such as target objects, associated objects, and interfering objects), and the positional relationships between these objects, which can be loaded and executed by the simulator. The task requirements include the target object the robot needs to grasp, the associated objects, the positional relationships between the target object and its associated objects, and the target actions the robot needs to perform. The target object is the core object that the robot directly manipulates (e.g., grasps or moves) in the task; associated objects are auxiliary objects that have spatial relationships or interactions with the target object, used to support the target object's actions; the positional relationships describe the relative positional states of the target object and its associated objects in space (including initial and target states).

[0061] For example, in the natural language task description of "put the cup on the table", "cup" is the target object and "table" is the associated object, and the positional relationship between the two is "the cup is on the table"; in the natural language task description of "insert the key into the lock", "key" is the target object and "lock" is the associated object, and the positional relationship between the two is "the key is in the lock".

[0062] Specifically, the system can receive a corresponding natural language task description from a preset input interface, and use language parsing technology to extract the core elements of the task requirements (such as the target object, related objects, interference objects, positional relationships, and execution actions) to obtain the task requirement information. The natural language task description can be generated manually or automatically through a corresponding natural language generation model.

[0063] After receiving user input or system-preset natural language text, semantic analysis tools are used to identify nouns (objects), verbs (actions), and prepositions (relationships) in the text, separating the target object, related objects, and their positional relationships to form task requirement information. This positional relationship may include the initial positional relationship and / or the target positional relationship.

[0064] For example, the natural language task description is "pick the red cup off the table and put it in the drawer". Through parsing, we can determine that the target object is "red cup", the related objects are "table" and "drawer", the initial positional relationship is "red cup is on the table", and the target positional relationship is "red cup is in the drawer".

[0065] In one embodiment, when one or more of the associated elements, positional relationships, etc., are missing in the natural language task description, semantic analysis can automatically complete the missing elements such as associated elements and positional relationships.

[0066] Step 120: Query the target object model that matches the target object from the preset item library.

[0067] In this embodiment, the item library is a database that stores models (item models) of virtual objects (such as target objects) used to construct simulation tasks. It includes 3D models of various types and attributes of items (such as 3D models stored in .obj and .glb formats). Furthermore, corresponding item tag information is set for each 3D model. This item tag information is used to identify structured keywords or symbols of the item model's core features (such as name, physical attributes, and functions). For example, the item tag information identifies the item name (such as "cup"), and can also identify the item's physical attributes (such as "material: ceramic" and "color: red") and functions (such as "containing liquids"), for rapid retrieval and matching. The target object model is a virtual model selected from the item library that semantically matches the "target object" in the task requirements. The target object model can specifically be a 3D virtual model. For example, the tag information for a cup model is "Category: Cup; Material: Ceramic; Color: White; Function: Hot water container".

[0068] Based on the target object features in the task requirements information, a search and matching process is performed in a pre-set item database to determine the corresponding 3D model, and the determined 3D model is used as the target object model.

[0069] Optionally, by calculating the matching degree between the target object and the item tag information (such as "cup") of each model in the item library, the model with the highest matching degree is selected as the target object model; if there are multiple models with the highest matching degree, one or more can be selected as the target object model by combining the task scenario (such as choosing a heat-resistant material model in a kitchen scenario) and the item description information of the model.

[0070] Sub-description: For example, for the target object "red cup", the item database retrieves "cup_model_003 (item tag information: "cup")" and "cup_model_005 (item tag information: "cup")". By comparing semantic similarity, one or both of "cup_model_003" and "cup_model_005" can be selected as the target object model. Alternatively, by analyzing the task scenario and the item description information of the two models, "cup_model_003" can be ultimately determined as the target object model.

[0071] Step 130: Query the target scene that matches the associated object and the target workspace of the target scene from the preset scene library.

[0072] In this embodiment, the scene library refers to a database that stores virtual environment templates used to construct simulation tasks. It includes different types of scenes (such as indoor scenes, outdoor scenes, bedrooms, kitchens, living rooms, supermarkets, hospitals, offices, factories, etc.). Each scene is adapted to the workspaces of various robots. The workspace is configured with corresponding spatial structures and fixed facilities (such as cabinets and dining tables in a kitchen, and desks and chairs in an office). The workspace is a designated area for performing task operations, including spatial boundaries (such as length × width × height), coordinates of the center point, and other parameters, defining the layout range of the target object and related objects. The fixed facilities in the workspace may include 3D models of related objects related to the natural language task description. The model parameters of these 3D models can be fixed or adaptively adjusted according to the natural language task description.

[0073] Based on the attributes of the associated objects and the task logic, a matching target scene is selected from the scene library, and the workspace within that scene is determined. Specifically, target scenes can be first filtered based on common usage scenarios of the associated objects (e.g., "tables" often appear in kitchens and living rooms); then, a workspace that meets the task operation requirements is selected from the target scenes (e.g., "placing cups" requires selecting a flat area on the tabletop). For example, for the associated objects "table" and "drawer," the "kitchen scene" is selected from the scene library; this scene contains a 3D model of a dining table with drawers (i.e., the associated object model), and the "dining table area" is selected as the target workspace (center point [1.5,2,0], size 1.2m×0.8m), where the dining table may have drawers.

[0074] Optionally, the scene and target job space can be automatically selected based on the large language model, or they can be manually selected from the scene and job space selection interface.

[0075] Step 140: Based on the positional relationship, form the layout constraint information of the target object model in the target work space, and generate at least one simulation task that meets the task requirements under the layout constraint information.

[0076] In this embodiment, the layout constraint information is a set of rules used to regulate the placement and posture of the target object model in the work space. It may include one or more of the following: spatial positional relationship constraints (such as "the target object is above the associated object"), physical constraints (such as "no collision or overlap"), and boundary constraints (such as "not exceeding the work space range"). Taking the above natural language task description "pick up the red cup from the table and put it into the drawer" as an example, the resulting layout constraint information may be: "The cup model must be placed on the upper surface of the table model, and the distance from the edge of the table must be no less than 5cm, and there must be no overlap with other objects."

[0077] The generated simulation task can arrange the target object model and related object models in the work space according to the layout constraints, and output the task file in a standardized format that can be loaded by the simulator. The simulation task contains the pose data of the target object model, such as position coordinates, orientation angle, and relationship, to ensure that the task is reproducible.

[0078] Based on the positional relationship between the target object and its associated objects, the relative placement rules of the two in the workspace are determined. The layout is then adjusted using physical collision detection to ultimately generate an executable simulation task. Specifically, the target object model and associated object models are first placed according to their initial positional relationship (e.g., "a cup on a table") to form spatial layout information. Then, a collision detection algorithm (e.g., the separating axis theorem) is used to check for spatial overlap. If overlap exists, the position or orientation of the target object model is adjusted until the constraints are met. Finally, the layout result (spatial layout information) is saved in a structured format (e.g., JSON) to form at least one simulation task. Optionally, multiple spatial layout information can be generated simultaneously. The relative positions of the target object model and associated object model in each spatial layout information differ. For spatial layout information that meets the constraints, no adjustment is needed, and the corresponding simulation task can be directly generated. This allows multiple simulation tasks to be generated simultaneously for a single natural language task description, improving the efficiency of simulation task generation.

[0079] The robot simulation task generation method in this application automatically extracts task requirements, matches models, and establishes layout constraints based on natural language task descriptions, reducing the manual scene setting and layout adjustments required in traditional methods and improving the efficiency of simulation task generation. Secondly, the extraction process from natural language to target objects, related objects, and positional relationships ensures semantic matching between the virtual model and the task description (e.g., "drawer" will not match a "playground" scene), avoiding semantic discrepancies and guaranteeing semantic consistency. Furthermore, this application ensures that the virtual scene conforms to physical logic (e.g., objects will not float or penetrate) by using boundary constraints and layout rules of the target workspace, combined with positional relationships, enhancing the realism of the simulation task and strengthening the scene's rationality. Additionally, this application can generate multiple constraint-compliant simulation tasks based on the same natural language description (e.g., the layout of cups and tables in different positions), providing the robot with diverse training data and enhancing its adaptability.

[0080] In one embodiment, the associated objects include an initial associated object and a target associated object; the positional relationship includes an initial positional relationship and a target positional relationship; the workspace includes the size parameters and position center point parameters of each associated object. For example, Figure 2 As shown, the task requirement information for the simulation task is extracted from the natural language task description, including:

[0081] Step 210: Call the preset large language model to perform dependency parsing on the natural language task description and extract the predicates;

[0082] Step 220: Based on the predicates, determine the target object, initial associated object, target associated object, initial positional relationship between the target object and the initial associated object, and target positional relationship between the target object and the target associated object from the natural language task description;

[0083] Step 230: Based on the target object, initial associated object, target associated object, initial positional relationship, and target positional relationship, form a structured language file to reflect the task requirements.

[0084] In this embodiment, the size parameters and center point parameters of the associated object are spatial attribute data of the associated object. The size parameter refers to the three-dimensional spatial size of the associated object (e.g., length × width × height), and the center point parameter refers to the three-dimensional coordinates of the associated object in the work space (e.g., [x, y, z]), used to constrain the layout range of the target object. By setting appropriate size parameters and center point parameters of the associated object, the initial / target positional relationship between the target object and the associated object can be quantified into specific spatial constraints, ensuring that the layout of the target object in the work space conforms to physical logic (e.g., a cup will not exceed the spatial range of a table or drawer).

[0085] Initial associated objects refer to objects in the simulation task scenario that have a direct spatial relationship with the initial position of the target object. They serve as the carrier or reference point for the target object's initial position. In other words, initial associated objects are the associated objects at the target object's position before it performs an action, used to define the starting state of the task. Target associated objects refer to objects in the simulation task scenario that have a direct spatial relationship with the final position (target position) of the target object. They serve as the carrier or reference point for the target object's target position, which is the destination to which the target object needs to be moved.

[0086] Initial positional relationship represents the spatial relative state between the target object and its initial associated object at the start of the task. It is represented by an initial state predicate and reflects the spatial constraints between the target object and its initial associated object before the action is executed (i.e., the initial positional relationship, such as "on..." or "beside..."). Target positional relationship represents the spatial relative state between the target object and its target associated object when the task is completed, reflecting the spatial constraints between the target object and its target associated object after the action is executed (i.e., the target positional relationship, such as "inside..." or "on...").

[0087] Taking "take the red cup off the table and put it in the drawer" as an example, "take" is the predicate, "red cup" is the target object; "table" is the initial related object (the cup is initially on the table); "drawer" is the target related object (the cup will eventually be put in the drawer); "the cup is on the table" (OnTop) is the initial positional relationship; "the cup is in the drawer" (Inside) is the target positional relationship.

[0088] This method utilizes a large language model to parse the grammatical structure of natural language task descriptions, identify and output predicates. This large language model can be any suitable model based on the GPT series, LLaMA, or similar frameworks. The natural language task description is input into a predefined large language model. The model first uses dependency parsing to clarify the logical structure of objects, actions, and spatial relationships within the sentence. Then, combining predicate extraction rules (such as semantic constraints based on a BDDL (Behavior Domain Definition Language) template library) and the natural language description, it separates related objects, target objects, descriptions of target objects, and various states between related objects, forming the predicate relations of the natural language. For example, predicate relations can be extracted using BDDL-based prompts.

[0089] The predicate relation can be represented as: Initial state: OnTop(cup, table); Target state: Inside(cup, drawer).

[0090] After the predicate relation is formed, a corresponding structured language file is constructed based on this relation. This structured language file can be a BDDL language file. An example BDDL language file can be represented as follows:

[0091] objects:

[0092] - cup

[0093] - table

[0094] - drawer

[0095] initial_conditions:

[0096] - OnTop(cup, table)

[0097] goal_conditions:

[0098] - Inside (cup, drawer)

[0099] This embodiment extracts predicates by combining a large language model with dependency parsing, which can accurately separate the initial state and target state of the target object, avoiding relationship confusion caused by ambiguity in natural language (such as clearly distinguishing the different associated objects corresponding to "from the table" and "put into the drawer"); by converting the spatial positional relationships in natural language into structural language files, it provides a clear logical basis for the layout constraints of subsequent simulation tasks (such as collision detection and spatial layout).

[0100] In one embodiment, such as Figure 3 As shown, the system retrieves target object models that match the target object from a pre-defined item database, including:

[0101] Step 310: Retrieve item tag information that matches the target item from the preset item library.

[0102] Step 320: When multiple matching item tag information exists, obtain the item description information for each item tag information and analyze the scene matching between each item description information and the natural language task description.

[0103] Step 330: Determine the target object model from multiple item tag information based on scene matching.

[0104] In this embodiment, the item description information is a supplementary description of the item model's detailed features, applicable scenarios, functional details, and other dimensions. It is a supplementary extension to the item label information, containing more specific scenario-based descriptions. For example, the item description information describes the corresponding item's size, material, weight, suitable workspace, price, and other dimensions. For instance, the description information for a ceramic mug item model is "white ceramic mug with handle, high temperature resistant, suitable for use in the kitchen or office for holding hot water, coffee, etc., capacity 350ml."

[0105] Scene matching refers to the degree of fit between the item description information and the scene requirements implied in the natural language task description. Specifically, it can be quantified through semantic similarity calculations (such as keyword overlap and scene attribute matching). The higher the similarity value, the more suitable the item is for the task scene. Taking the natural language task description as "get a cup that can hold hot water in the kitchen" as an example, then "ceramic cup (heat resistant, suitable for the kitchen)" has a higher scene matching than "glass cup (not heat resistant)".

[0106] Optionally, when the natural language description contains not only the target object but also its key features, the target object and its key features (such as category and attributes) can be extracted, converted into search keywords, and matched with the tag information of models in the item database to filter out models whose tag overlap reaches a preset threshold. For example, if the target object is a "cup" and its color is red, the keywords "category: cup; color: red" can be extracted, and models with the tags "cup" and "color: red" can be searched in the item database to obtain candidate model A (tag: "cup, red, plastic"), model B (tag: "cup, red, ceramic"), and model C (tag: "cup, red, glass").

[0107] For multiple candidate models, the suitability of each model is determined by analyzing the fit between its detailed description information and the requirements of the task scenario. For example, text analysis tools (such as semantic similarity algorithms) can be used to compare the item description information of each model with the scenario (such as "kitchen" or "office") and functional requirements (such as "filling hot water" or "lightweight and easy to carry") implied in the natural language task description, and a scenario matching score is calculated. One or more models with the highest scores are selected; if there are models with the same score, other rules (such as model accuracy and commonness) can be further combined to determine the final result, or all models with the same score can be used as target object models, and simulation tasks can be generated independently based on each target object model.

[0108] For example, the natural language task description is "get a red cup in the kitchen to fill with hot water": extract the task scenario requirements: "kitchen scenario" and "fill with hot water". If the description information of the above candidate models are as follows: Model A: "red plastic cup, lightweight, suitable for the office, not heat resistant"; Model B: "red ceramic cup, heat resistant, suitable for the kitchen, capacity 300ml"; Model C: "red glass cup, transparent, suitable for the living room, easily broken by sudden temperature changes".

[0109] Analyzing scene matching: Model B has the highest matching degree with "kitchen" and "filling hot water" (score 90); Model C has the second highest matching degree (score 60, material is not suitable for hot water); Model A has the lowest matching degree (score 30, scene and function do not match). Therefore, Model B is determined as the target object model that matches the target object "red cup".

[0110] In this embodiment, the two-layer screening mechanism of "label initial screening + description information scene matching" can not only quickly locate the most suitable target object model and improve retrieval efficiency, but also avoid mismatches caused by relying on a single label. It ensures that the selected model is consistent with the task in terms of function and applicable scenario (such as prioritizing high-temperature resistant material models in kitchen scenarios), and can more realistically simulate the characteristics of objects in real-world scenarios (such as the rationality of using ceramic cups to hold hot water), providing a more realistic environment for robot training.

[0111] In one embodiment, such as Figure 4 As shown, the system retrieves the target scene matching the associated object and the target workspace of that target scene from the preset scene library, including:

[0112] Step 410: Based on the semantic understanding of the natural language task description, query the scene library to find the target scene that matches the task requirement information.

[0113] Step 420: Query the basic workspace that matches the associated objects and target scene from the preset scene library.

[0114] Step 430: Based on the semantic understanding of the natural language task description, configure the location and size information of the basic task space to form the target task space.

[0115] In this embodiment, it can detect whether the natural language directly mentions a scene. If it does, the scene is taken as the target scene. If no scene information is mentioned, semantic understanding is used to analyze the scene suitable for the natural language task description, and the analyzed scene is taken as the target scene. For example, if the natural language is "put the clothes on the bed into the closet in the bedroom," the directly mentioned scene is "bedroom." For the natural language "put the clothes on the bed into the closet," semantic understanding can analyze that the matching target scene is "bedroom."

[0116] Each scene contains one or more basic operation spaces. These basic operation spaces are predefined general operation area templates that match the functions of related objects in the scene (such as "bookshelf operation area" or "refrigerator operation area"). They contain basic space attributes (such as planar / three-dimensional areas and function labels), but one or more parameters such as the location and size can be fixed or adjustable values.

[0117] Based on the type (e.g., "wardrobe" or "bookshelf") and function (e.g., "store clothes" or "store books") of the associated objects, retrieve the basic work space that matches the function tag of the associated object from the set of basic work spaces associated with the target scene (e.g., "wardrobe" corresponds to "storage area in front of the wardrobe").

[0118] Furthermore, after determining the basic work space, it is necessary to further consider whether the size of the basic work space meets the requirements of the corresponding task. Specifically, after determining the basic work space, it is also possible to further combine the operational range of the actions in the task (such as "putting clothes" requires sufficient floor space), the size of related objects (such as the width of the wardrobe being 1.2m), and the position of other facilities in the scene (such as no obstacles within 0.5m next to the wardrobe) to configure or modify the position (such as the position center point parameter mentioned above) and size information (such as the size parameter mentioned above) of each related object in the basic work space. For example, configure the position center point coordinates and length × width × height dimensions of each related object to meet the interaction convenience between the target object and related objects (such as the work space covering the wardrobe door opening range).

[0119] Similar to the method of selecting targets, the final determined target scene and target work space can also be one or more. Electronic devices can arrange and combine each determined target, target scene and target work space, and generate simulation tasks for each combination, thereby increasing the number of simulation tasks generated and the diversity of tasks.

[0120] In one embodiment, forming layout constraint information of the target object model in the target work space based on the positional relationship includes: determining a first number of interfering object models based on the target object and related objects; and forming layout constraint information of the target object model and the interfering object model in the target work space based on the positional relationship.

[0121] In this embodiment, in addition to the target object and related objects mentioned in natural language, one or more interfering objects can be appropriately added to the generated simulation task. Interfering objects are virtual objects introduced to simulate the complexity of the real environment and are not the objects of the task operation. Interfering objects have a certain semantic relationship with the target object, related objects, or task scene (e.g., in a "kitchen scene," interfering objects are mostly tableware and ingredients, rather than tools). By occupying space and obstructing parts of the path, they pose a challenge to the robot's perception or operation, increasing the scene complexity. The positional relationship between interfering objects and related objects can be the same as the positional relationship between the target object and related objects; for example, both may be located on top of the interfering objects.

[0122] The first quantity can be a fixed quantity preset by the system (such as 2 or 3), or it can be a quantity adaptively calculated based on the natural language task description.

[0123] Specifically, based on the item tag information of the target object, models associated with the tags can be retrieved from the model library to ensure that the distractors and the task scenario are semantically matched. For the retrieved models, a first number of models can be randomly selected as distractor models. The selected models can be the same or different; for example, two identical distractor models can be selected. For instance, two candidate models (such as "banana" and "orange") can be randomly selected as distractor models, as they belong to the same fruit category as the target object "apple," thus conforming to the scenario semantics.

[0124] For the selected target object model, interference object model, and target work space, the spatial constraint information is formed by combining the positional relationship between the target object and its associated objects within the target work space as the first constraint, the positional relationship between the interference object and its associated objects as the second constraint, and the positional relationship between the interference object and the target object as the third constraint. The positional relationship between the interference object and its associated objects can be the same as that between the target object and its associated objects (e.g., both objects are on top of each other). The positional relationship between the interference object and the target object is such that there is no collision or spatial overlap between them. Furthermore, the spatial constraint information also includes the requirement that the target object model and the interference object model must not exceed the work area of ​​the target work space.

[0125] Taking the positional relationship as "apple inside the fruit plate", the target working space as "operating table area (center point [1.2, 0.8, 0], size 1.5m × 0.6m × 0.75m)", and the interfering objects as "banana" and "orange", for example, the first constraint (core constraint) is: the coordinates of all vertices of the apple model must be within the internal space of the fruit plate model (the z-coordinate of the bottom of the fruit plate ≤ the z-coordinate of the bottom of the apple ≤ the z-coordinate of the top of the fruit plate, and the x and y coordinates are within the plane of the fruit plate); the second constraint (interfering object constraint) is: the banana and orange models must be placed around the fruit plate model (distance from the edge of the fruit plate ≥ 5cm); the third constraint... The constraints (collision constraints) are: the apple model, banana model, and orange model have no overlap in 3D space; in addition, the longest side of the banana model must not occlude the area above the apple model (to ensure the robot's grasping path is unobstructed); the fourth constraint (boundary constraints) is: the x-coordinates of all vertices of the apple, fruit plate, banana, and orange must be within the range of [1.2-0.75, 1.2+0.75] (operating table length 1.5m), the y-coordinates must be within the range of [0.8-0.3, 0.8+0.3] (operating table width 0.6m), and the z-coordinates must be within the range of [0, 0.75] (operating table height 0.75m).

[0126] This embodiment introduces a first number of interference object models to simulate non-target objects (such as other items on the kitchen countertop) that exist in the real environment, making the robot training closer to the actual scenario and enhancing its anti-interference ability training.

[0127] In one embodiment, such as Figure 5 As shown, a first number of interference models are determined based on the target object and related objects, including:

[0128] Step 510: Determine the projected area of ​​the target object model on the surface of the associated object based on the target object, associated objects, and the relationship between them.

[0129] Based on the 3D model parameters of the target object and its associated objects, and their spatial relationship, the orthographic projection area of ​​the target object on a specific surface of the associated object is calculated. This surface can be determined based on the association relationship, such as a horizontal plane. In terms of distance, "OnTop" corresponds to the top surface of the associated object, and "Inside" corresponds to the bottom surface inside the associated object. The target object model contains the size of the target object; based on the position, orientation, and size between the target object model and the associated object model, the corresponding projected area can be calculated.

[0130] Step 520: Determine the constraint area of ​​the interfering object based on the projected area and the associated objects.

[0131] The constraint area is a threshold (maximum allowable value) used to limit the total projected area of ​​all interfering object models on the surface of the associated object, ensuring that interfering objects do not occupy excessive space. Optionally, the constraint area (such as a certain proportion of the total surface area of ​​the associated object minus the projected area of ​​the target object) can be calculated based on the projected area of ​​the target object and the available surface area of ​​the associated objects, and is a key indicator for screening interfering objects.

[0132] Optionally, the difference between the usable area of ​​the surface of the associated object and the projected area of ​​the target object can be used first, and the product of this difference and a preset first coefficient can be used as the corresponding constraint area. This coefficient is less than 1, and can be set to any suitable value such as 0.4, 0.5, 0.6, 0.7, etc. For example, the top surface area of ​​the associated object "plate" is 300 cm². 2 The projected area of ​​the target object is 50cm². 2 The constrained area can be set as (300-50)×60%=150cm² 2 (That is, the total projected area of ​​all interfering objects must not exceed 150cm²) 2 ).

[0133] In another embodiment, the constraint area = the projected area of ​​the target object × a second coefficient. The second coefficient can be any preset suitable value, such as any suitable value between 1 and 2 (e.g., the second coefficient is 1.5).

[0134] Step 530: Based on the target object and related objects, identify multiple candidate objects that match the target scene.

[0135] Specifically, the system obtains the tag information of the target object (e.g., "mug" tag: tableware, cup), related objects (e.g., "desk" tag: furniture, office), and target scene (e.g., "office" tag: office scene); it filters models from the item library whose tags intersect with the above tags (e.g., in the "office" scene, models related to "office" and "tableware"), excludes models that are exactly the same as the target object, and retains models that conform to semantic association as candidates.

[0136] For example, the target item is "mug" (label: tableware, cup), the associated item is "desk" (label: office, furniture), and the target scene is "office": the model with the label "office" and "tableware" is retrieved from the item library to obtain "pen model", "notebook model", "folder model" and "small potted plant model"; all of the above models are matched with the office scene to form a candidate set.

[0137] Step 540: Determine a first number of interfering object models from multiple candidates based on the constraint area.

[0138] In this embodiment, a first number of models are selected from the candidate set to ensure that their total projected area does not exceed the constraint area. Understandably, there may be more than one feasible candidate selection scheme. The electronic device can randomly select one or more schemes that meet the conditions, and combine each scheme with the target object, target scene, and target workspace to generate subsequent simulation tasks. Each selection scheme generates one or more corresponding simulation tasks.

[0139] In this embodiment, by quantitatively calculating the projected area and the constraint area, the total space of the interfering objects is avoided from being too large, ensuring that the core operating space of the target object and related objects is not squeezed, which conforms to the spatial distribution law of the real scene. By selecting candidate objects based on the target object, related objects and the scene, irrelevant models are avoided (such as not selecting "frying pan" as an interfering object in an office scene), ensuring the scene rationality of the simulation task.

[0140] In one embodiment, the layout constraint information requires that there be no collision or spatial overlap between the target object and interfering objects. For example... Figure 6 As shown, generating at least one simulation task that meets the task requirements under layout constraint information includes:

[0141] Step 610: Based on the positional relationship, set the first pose information of the target object model and the second pose information of each interference object in the target work space to form the spatial layout information of the target object and interference objects in the target work space.

[0142] Specifically, the first pose information of the target object model can be generated first. Based on the first pose information, the second pose information of each interference object model can be generated sequentially. The generation method of the first and second pose information satisfies the positional relationship between the target object, interference object, and related objects. For example, the generated first pose information indicates that the target object is on top of the interference object.

[0143] Taking the positional relationship as "apple on plate" and the target workspace as "kitchen workbench (range [x:0.4-0.6, y:0.2-0.4, z:0.7-0.7])" as an example, the first pose information is represented as: apple model position [0.5, 0.3, 0.7] (center of plate), pose [0°, 0°, 0°]; the second pose information is represented as: two interfering objects (banana, orange) in the workbench within a ≥5cm area around the apple, banana position [0.55, 0.3, 0.7], pose [0°, 45°, 0°] (tilted along the length direction); orange position [0.5, 0.35, 0.7], pose [0°, 0°, 0°]; the spatial layout information is integrated into the pose information of the three objects.

[0144] Step 620: Check whether the spatial layout information meets the layout constraints. If yes, proceed to step 630; otherwise, return to step 610.

[0145] Step 630: Generate the corresponding simulation task.

[0146] In this embodiment, the spatial layout of the target object and the interfering objects is verified to meet the "no collision, no overlap" constraint. If it does, a simulation task is output. A collision detection algorithm (such as the Separating Axis Theorem, SAT) can be used to check whether the three-dimensional space of the target object model and each interfering object model overlaps (i.e., whether there is an intersection). If there is no overlap between all models, it is determined that the constraint is met. The spatial layout information is saved in a structured format (such as JSON) to generate a simulation task for easy reuse and sharing.

[0147] If a collision is detected, the pose of the target object or interfering object is readjusted, and the detection is repeated until the constraints are met. Specifically, the cause of the collision can be analyzed (e.g., the interfering object is too close); pose adjustment is performed based on the analyzed cause of the collision, thereby improving adjustment efficiency. For example, if the collision is caused by the interfering object being too close, the position of the interfering object can be slightly moved (e.g., offset by 2cm along the x-axis) or its posture can be rotated (e.g., changing the tilt angle of the banana) while keeping the first pose of the target object unchanged (prioritizing the core positional relationship). The number of pose adjustment iterations does not exceed a preset threshold (e.g., 200 times).

[0148] Steps 610 to 630 above can be repeated multiple times to generate multiple simulation tasks. For example, they can be repeated three times to generate three simulation tasks, as follows:

[0149] jsontask1.jsontask2.jsontask3.json.

[0150] Each JSON file (task1.json, task2.json, task3.json) is a simulation task file stored in a structured format, and its file can be represented as follows:

[0151] {

[0152] "cup": {"position": [0.2, 0.8, 0.5], "orientation": [0,0,0], "model":"cup_model_003"},

[0153] "table": {...},

[0154] "relations": ["OnTop(cup, table)", "Inside(cup, drawer)"]

[0155] }

[0156] In one embodiment, generating at least one simulation task that meets the task requirements under layout constraint information includes: randomly setting target object models and interference object models in the target work space based on positional relationships to form multiple sets of spatial layout information for target objects and interference objects; detecting whether each set of spatial layout information meets the layout constraint information; storing the spatial layout information that meets the layout constraint information in a structured format to generate the corresponding simulation task.

[0157] Within the target workspace, multiple sets of pose combinations of target objects and interfering objects are randomly generated according to the positional relationship framework (combinations of the first pose information and the second pose information mentioned above), forming various spatial layout information. During the random generation process, the random range is determined according to the aforementioned positional relationship, ensuring that both the target object and the interfering object are within a reasonable area of ​​the associated objects (e.g., "on the table" requires the target object's x and y coordinates to be within the tabletop area, and the z coordinate value to equal the tabletop height; the interfering object's position must be within the workspace and maintain a basic safe distance from the target object).

[0158] Within this random range, the first pose information of the target object model and the second pose information of the interference object model are generated. The layout constraint information can also be detected using the collision detection algorithm (such as the Separating Axis Theorem SAT) mentioned above, and all layout constraint information that simultaneously satisfies collision detection and positional relationship verification is recorded.

[0159] For the generated simulation task, it can be loaded into a preset simulation environment, and the simulator can be started to train the robot and collect simulation training result data. For example, the scene can be loaded into psi-lab via a JSON file to start the simulator; in the simulator, the robot can be trained through reinforcement learning, imitation learning, motion planning, etc., and the training result data can be collected.

[0160] Based on the simulation training results data, the robot's simulation task difficulty, task completion rate and other evaluation indicators are evaluated. Based on these evaluation indicators, the selection of the natural language task description, target object model matching, target scene and target work space is optimized, and the layout constraint information is optimized, so that the newly generated simulation tasks can further improve the robot's simulation training effect.

[0161] This embodiment generates the first pose of multiple target object models and the second pose of interference object models at one time, forming multiple spatial layout information. For each spatial layout information, layout constraint information is detected, and spatial layout information that meets the constraint conditions is retained to improve the efficiency of simulation task generation.

[0162] In one embodiment, a computer-readable storage medium is provided having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the steps in the above method embodiments.

[0163] In one embodiment, an electronic device is also provided, including one or more processors; and a memory storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the steps in the above method embodiments.

[0164] In one embodiment, such as Figure 7The diagram illustrates the structure of an electronic device used to implement an embodiment of this application. The electronic device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0165] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0166] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer-readable medium carrying instructions that, in such embodiments, can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the instructions are executed by central processing unit (CPU) 701, the various method steps described in this application are performed.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0168] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of this application and form different embodiments. For example, all the embodiments above can be used in any combination. The information disclosed in this background section is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for generating robot simulation tasks, characterized in that, The method includes: Obtain a natural language task description, and extract the task requirement information of the simulation task from the natural language task description. The task requirement information includes the target object, associated objects, and positional relationship between the target object and associated objects that the robot needs to grasp. Retrieve target object models that match the target object from a pre-defined item database; Retrieve the target scene that matches the associated object and the target workspace of the target scene from the preset scene library; Based on the positional relationship, the layout constraint information of the target object model in the target work space is formed, and at least one simulation task that meets the task requirements information is generated under the layout constraint information. The step of forming layout constraint information of the target object model in the target work space based on the positional relationship includes: determining the projected area of ​​the target object model on the surface of the associated object based on the target object, associated objects and the relationship; determining the constraint area of ​​the interfering object based on the projected area and the associated object; determining multiple candidate objects that match the target scene based on the target object and the associated object; determining a first number of interfering object models from the multiple candidate objects based on the constraint area; and forming layout constraint information of the target object model and the interfering object model in the target work space based on the positional relationship.

2. The method according to claim 1, characterized in that, The layout constraint information requires that there be no collision or spatial overlap between the target object and the interfering object; The step of generating at least one simulation task that conforms to the task requirements under the layout constraint information includes: Based on the positional relationship, the first pose information of the target object model and the second pose information of each interference object are set in the target work space to form the spatial layout information of the target object and interference objects in the target work space. Detect whether the spatial layout information satisfies the layout constraint information; if it does, generate the corresponding simulation task. If the conditions are not met, the process returns to re-execute the setting of the first pose information of the target object model and the second pose information of each interfering object in the target work space based on the positional relationship, until the layout constraint information is met.

3. The method according to claim 1, characterized in that, The layout constraint information requires that there be no collision or spatial overlap between the target object and the associated object; The step of generating at least one simulation task that conforms to the task requirements under the layout constraint information includes: Based on the positional relationship, the target object model and the interference object model are randomly set in the target work space to form multiple sets of spatial layout information of target objects and interference objects; Detect whether each group of spatial layout information satisfies the layout constraint information; The spatial layout information that satisfies the layout constraints is stored in a structured format to generate the corresponding simulation task.

4. The method according to claim 1, characterized in that, The associated objects include initial associated objects and target associated objects; the positional relationships include initial positional relationships and target positional relationships; the workspace includes the size parameters and position center point parameters of each associated object; The step of extracting the task requirement information of the simulation task from the natural language task description includes: The pre-defined large language model is invoked to perform dependency parsing on the natural language task description and extract the predicates. Based on the predicate, the target object, initial associated object, target associated object, initial positional relationship between the target object and the initial associated object, and target positional relationship between the target object and the target associated object are determined from the natural language task description; Based on the target object, initial associated object, target associated object, initial positional relationship, and target positional relationship, a structured language file is formed to reflect the task requirements.

5. The method according to claim 1, characterized in that, The step of retrieving a target object model that matches the target object from a preset item database includes: Retrieve item tag information that matches the target item from a preset item database; When multiple matching item tag information exists, obtain the item description information for each item tag information, and analyze the scene matching between each item description information and the natural language task description; The target object model is determined from multiple item tag information based on the scene matching.

6. The method according to claim 1, characterized in that, The step of retrieving the target scene and the target workspace of the target scene that match the associated object from the preset scene library includes: Based on the semantic understanding of the natural language task description, a target scene matching the task requirement information is retrieved from the scene database; Retrieve a basic workspace from a pre-defined scene library that matches the associated object and the target scene; Based on the semantic understanding of the natural language task description, the location and size information of the basic task space are configured to form the target task space.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores executable instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1 to 6.

8. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to perform the method as described in any one of claims 1 to 6.

Images