Robot simulation task generation method, medium and equipment

Abstract

The invention provides a robot simulation task generation method, a medium and equipment, and belongs to the technical field of data processing. The method comprises the steps that natural language task description is obtained, task requirement information of a simulation task is extracted from the natural language task description, and the task requirement information comprises a target object needing to be grabbed by a robot, a related object and the position relation between the target object and the related object; querying a target object model matched with the target object from a preset object library; querying a target scene matched with the related object and a target operation space of the target scene from a preset scene library; and forming layout constraint information of the target object model in the target operation space based on the position relationship and the volumes of the target object and the interferent, and generating at least one simulation task meeting the task requirement information under the layout constraint information. The robot simulation task generation efficiency and the robot simulation task generation quality can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a robot simulation task generation method, medium and device. BACKGROUND

[0002] In the field of robots, building a task-driven intelligent system is the core link to realize autonomous decision and operation of robots, and this process highly depends on a large number of diversified and high-quality simulation tasks as a training basis. These simulation tasks need to simulate the object interaction relationship, spatial layout and target condition in the real scene, and provide data support for the training of robot reinforcement learning, imitation learning and action planning.

[0003] Traditional simulation task generation methods usually adopt several modes such as fixed template, manual construction or behavior learning sampling. Among them, the fixed template generation is based on a preset scene framework and task mode, and a new task is generated by replacing a small number of parameters, for example, only changing the object type in the "object carrying" template; manual construction relies on human full-time participation, from scene selection, object placement to target condition setting, which are all completed by human to ensure that the task meets the specific requirements; behavior learning sampling is to record the existing task execution process in the real world or simulation environment, and to extract and reproduce similar tasks from it.

[0004] However, these modes all need to consume a lot of manpower to set the grasping object and the spatial environment layout of the grasping scene of the robot one by one, and the efficiency of task construction is low, it is difficult to batch generate diversified scenes, which seriously restricts the large-scale application of simulation tasks. SUMMARY

[0005] The purpose of the present application is to provide a robot simulation task generation method, medium and device to solve at least one of the above technical problems.

[0006] In a first aspect of the present application, a robot simulation task generation method is provided, the method comprising: obtaining a natural language task description, extracting task requirement information of a simulation task from the natural language task description, the task requirement information including a target object, an associated object, and a position relationship between the target object and the associated object that the robot needs to grasp; querying a target object model matching the target object from a preset object library; querying a target scene matching the associated object and a target work space of the target scene from a preset scene library; forming layout constraint information of the target object model in the target work space based on the position relationship, and generating at least one simulation task meeting the task requirement information under the layout constraint information.

[0007] Optionally, the forming of the layout constraint information of the target object model in the target working space based on the position relationship comprises: determining a first number of interference object models based on the target object and the associated object; forming the layout constraint information of the target object model and the interference object model in the target working space based on the position relationship.

[0008] Optionally, the determining of the first number of interference object models based on the target object and the associated object comprises: determining a projection area of the target object model on the surface of the associated object based on the target object, the associated object and the associated relationship; determining a constraint area of the interference object based on the projection area and the associated object; determining a plurality of candidate objects matching the target scene based on the target object and the associated object; determining the first number of interference object models from the plurality of candidate objects based on the constraint area.

[0009] Optionally, the layout constraint information requires that there is no collision and spatial overlap between the target object and the interference object; the generating of at least one simulation task meeting the task requirement information under the layout constraint information comprises: setting first pose information of the target object model and second pose information of each interference object in the target working space based on the position relationship, to form spatial layout information of the target object and the interference object in the target working space; detecting whether the spatial layout information meets the layout constraint information, and if so, generating a corresponding simulation task; if not, returning to re-perform the setting of the first pose information of the target object model and the second pose information of each interference object in the target working space based on the position relationship, until the layout constraint information is met.

[0010] Optionally, the layout constraint information requires that there is no collision and spatial overlap between the target object and the associated object; the generating of at least one simulation task meeting the task requirement information under the layout constraint information comprises: randomly setting the target object model and the interference object model in the target working space based on the position relationship, to form a plurality of groups of spatial layout information of the target object and the interference object; detecting whether each group of spatial layout information meets the layout constraint information; storing spatial layout information meeting the layout constraint information in a structured format, to generate a corresponding simulation task.

[0011] Optionally, the association objects include an initial association object and a target association object; the position relationships include an initial position relationship and a target position relationship; and the work space includes a size parameter and a position center point parameter of each association object. The task requirement information of the simulation task is extracted from the natural language task description, including: A preset large language model is called to perform dependency syntax analysis on the natural language task description, and a predicate is extracted; Based on the predicate, a target object, an initial association object, a target association object, an initial position relationship between the target object and the initial association object, and a target position relationship between the target object and the target association object are determined from the natural language task description; According to the target object, the initial association object, the target association object, the initial position relationship, and the target position relationship, a structural language file for embodying the task requirement is formed.

[0012] Optionally, the target object model matching the target object is queried from the preset object library, including: An object tag information matching the target object is queried from the preset object library; When there are multiple matching object tag information, object description information of each object tag information is obtained, and scene matching between each object description information and the natural language task description is analyzed; Based on the scene matching, a target object model is determined from the multiple object tag information.

[0013] Optionally, the target scene matching the association object and the target work space of the target scene are queried from the preset scene library, including: Based on semantic understanding of the natural language task description, a target scene matching the task requirement information is queried from the scene library; A basic work space matching the association object and the target scene is queried from the preset scene library; Based on semantic understanding of the natural language task description, position and size information is configured for the basic work space to form a target work space.

[0014] In a second aspect, a computer readable storage medium is provided, and the computer readable storage medium stores executable instructions. When the executable instructions are executed by a processor, the processor performs the method described in any one of the embodiments of the present application.

[0015] In a third aspect, the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to perform the method according to any one of the embodiments of the present application.

[0016] The robot simulation task generation method, medium and device in the present application can automatically extract task requirements, match models and layout constraints according to natural language task descriptions, reduce the operation of manually setting a scene and adjusting a layout in the traditional method, and improve the simulation task generation efficiency and generation quality. Secondly, the extraction process from natural language to target objects, associated objects and position relationships can ensure semantic matching between virtual models and task descriptions, avoid semantic disconnection, and ensure semantic consistency. In addition, the boundary constraints and layout rules of the target work space, combined with the position relationship, can ensure that the virtual scene conforms to the physical logic (such as objects not floating or penetrating), improve the authenticity of the simulation task, and enhance the scene rationality. In addition, the present application can generate multiple simulation tasks (such as different positions of a cup and a table layout) that meet the constraints based on the same natural language description, provide diversified training data for robots, and enhance their adaptability. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope of the present application.

[0018] Figure 1 A flowchart of a robot simulation task generation method in an embodiment; Figure 2 A flowchart of extracting task requirement information of a simulation task from a natural language task description in an embodiment; Figure 3 A flowchart of querying a target object model matched with a target object from a preset object library in an embodiment; Figure 4 A flowchart of querying a target scene matched with an associated object and a target work space of the target scene from a preset scene library in an embodiment; Figure 5 A flowchart of determining a first number of interference object models based on a target object and an associated object in an embodiment; Figure 6 A flowchart of generating at least one simulation task meeting the task requirement information under layout constraint information in an embodiment; Figure 7 A structural diagram of an electronic device in an embodiment. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] All terms used in this application (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0021] For example, the terms "first," "second," etc., used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element.

[0022] For example, the terms "comprising" or "including" used in this application indicate the presence of features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0023] This application provides a method for generating robot simulation tasks, combined with Figure 1 As shown, the method includes: Step 110: Obtain the natural language task description and extract the task requirement information of the simulation task from the natural language task description.

[0024] In this embodiment, the natural language task description is descriptive information expressed in natural language to instruct the robot to perform relevant simulation tasks. It includes the object to be manipulated (i.e., the target object) and the specific manipulation task performed on that object. Furthermore, it may include additional information such as a description of the target object itself and environmental characteristics. This descriptive information may include the target object's color, size, and dimensions, while the environmental characteristics may include the target object's location and information about other objects in its environment. The natural language task description can be actively input by the user or provided by a pre-set task library, with the electronic device automatically retrieving the description from the library.

[0025] The simulation task is used to simulate a real scene in a virtual environment, and is a task for robot training and testing. The simulation task can be a training task for the robot to grasp and place an object. The simulation task includes a virtual scene, virtual objects (such as target objects, associated objects, interference objects, etc.), and the positional relationship between the objects, and can be loaded and executed by a simulator. The task requirement information includes the target object, the associated object, the positional relationship between the target object and the associated object, and the target action to be performed by the robot. The target object is a core object that the robot directly operates (such as grasping and moving) in the task. The associated object is an auxiliary object that has a spatial relationship or interaction with the target object, and is used to support the action execution of the target object. The positional relationship is used to describe the relative position state (including the initial state and the target state) of the target object and the associated object in space.

[0026] For example, in the natural language task description “put the cup on the table”, the “cup” is the target object, and the “table” is the associated object, and the positional relationship between the two is that the “cup is on the table”. In the natural language task description “insert the key into the door lock”, the “key” is the target object, and the “door lock” is the associated object, and the positional relationship between the two is that the “key is in the door lock”.

[0027] Specifically, the corresponding natural language task description can be received from a preset input interface, and language parsing technology can be used to extract the core elements of the task requirements (such as target objects, associated objects, interference objects, positional relationships, and execution actions), to obtain the task requirement information. The natural language task description can be artificially generated, or automatically generated by a corresponding natural language generation model.

[0028] After receiving the natural language text input by the user or preset by the system, a semantic analysis tool is used to identify the nouns (objects), verbs (actions), and prepositions (relationships) in the text, to separate the target object, the associated object, and the positional relationship between the two, to form the task requirement information. The positional relationship can include an initial positional relationship and / or a target positional relationship.

[0029] For example, the natural language task description is “pick up the red cup from the table and put it into the drawer”, and through analysis, it is determined that the target object is “the red cup”, the associated objects are “the table” and “the drawer”, the initial positional relationship is “the red cup is on the table”, and the target positional relationship is “the red cup is in the drawer”.

[0030] In one embodiment, when one or more of the associated objects and the positional relationship are missing in the natural language task description, the missing factors such as the associated objects and the positional relationship can be automatically completed by semantic analysis.

[0031] In step 120, a target object model matching the target object is queried from a preset object library.

[0032] In this embodiment, the item library is a database storing models (item models) of virtual objects (such as target objects) used to build a simulation task, which contains three-dimensional models (such as three-dimensional models stored in.obj,.glb format files) of items of various categories and attributes, and further sets corresponding item tag information for each three-dimensional model, which is used to identify the structured keywords or symbols of the core features (such as name, physical attribute, function, etc.) of the item model. For example, the item tag information identifies the name of the item (such as “cup”), and can also identify the physical attributes (such as “material: ceramic” and “color: red”) and functions (such as “holding liquid”) of the item, for quick retrieval and matching. The target object model is a virtual model selected from the item library and matched with the semantic of the “target object” in the task requirement. The target object model can be a three-dimensional virtual model. For example, the tag information of a cup model is “category: cup; material: ceramic; color: white; function: holding hot water”.

[0033] Based on the target object features in the task requirement information, the pre-set item library is searched and matched to determine the corresponding three-dimensional model, and the determined three-dimensional model is used as the target object model.

[0034] Optionally, the matching degree between the target object and the item tag information (such as “cup”) of each model in the item library is calculated to select the model with the highest matching degree as the target object model. If there are multiple models with the highest matching degree, one or more of them can be further selected as the target object model in combination with the task scene (such as the kitchen scene preferentially selecting a heat-resistant material model) and the item description information of the model.

[0035] Lower description: For example, for the target object “red cup”, the item library retrieves “cup_model_003 (item tag information: “cup”)” and “cup_model_005 (item tag information: “cup”)”. Through semantic similarity comparison, one or both of “cup_model_003” and “cup_model_005” can be selected as the target object model. Alternatively, by analyzing the task scene and the item description information of the two models, “cup_model_003” is finally determined as the target object model.

[0036] Step 130, querying the target scene matched with the associated object and the target work space of the target scene from the pre-set scene library.

[0037] In this embodiment, the scene library refers to a database storing virtual environment templates for constructing simulation tasks, which contains different types of scenes (such as indoor and outdoor scenes, bedrooms, kitchens, living rooms, supermarkets, hospitals, offices, factories, etc.), each of which is adapted to the work space of multiple robots, and the work space is configured with corresponding space structures, fixed facilities (such as cabinets and tables in the kitchen, and desks and chairs in the office), etc. The work space is a work area set for performing task operations, which contains parameters such as space boundaries (such as length x width x height) and position center point coordinates, and limits the layout range of target objects and associated objects. The fixed facilities in the work space (worksapce) can include an associated object three-dimensional model related to the natural language task description, and the model parameters of the associated object three-dimensional model can be fixed parameters, or can be adjusted adaptively according to the natural language task description.

[0038] According to the attributes of the associated object and the task logic, a matching target scene is selected from the scene library, and a work space in the scene for performing the task is determined. Specifically, the target scene can be first filtered based on the common use scene of the associated object (such as "table" often appearing in the kitchen and living room); and then a work space (such as "putting a cup" requiring selecting an area with a flat table surface) that meets the task operation requirements is selected from the target scene. For example, for the associated objects "table" and "drawer", a "kitchen scene" is selected from the scene library; a three-dimensional model of a dining table with a drawer (i.e. an associated object model) is included in the scene, and a "dining table area" is selected as the target work space (center point [1.5, 2, 0], size 1.2m x 0.8m), wherein the dining table can have a drawer.

[0039] Alternatively, the scene and the target work space can be automatically selected according to a large language model, or can be manually selected from a scene and work space selection interface.

[0040] In step 140, layout constraint information of the target object model in the target work space is formed based on the position relationship, and at least one simulation task meeting the task requirement information is generated under the layout constraint information.

[0041] In this embodiment, the layout constraint information is a set of rules for regulating the placement position and posture of the target object model in the work space, which can include one or more of spatial position relationship constraints (such as "the target object is above the associated object"), physical constraints (such as "no collision overlap"), boundary constraints (such as "not exceeding the work space range"), etc. Taking the natural language task description "put the red cup from the table into the drawer" as an example, the formed layout constraint information can be: "the cup model needs to be placed on the upper surface of the table model, and the distance to the edge of the table is not less than 5 cm, and there is no overlap with other objects".

[0042] The generated simulation task can be arranging the target object model and the associated object model in the layout constraint in the work space, and outputting a task file in a standardized format which can be loaded by a simulator. The simulation task includes the position coordinates, the orientation angle, the association relationship and other pose data of the target object model, and ensures the task reproducibility.

[0043] According to the position relationship between the target object and the associated object, the relative placement rules of the two in the work space are determined, and the layout is adjusted combined with the physical collision detection, and finally an executable simulation task is generated. Among them, the target object model and the associated object model are placed according to the "initial position relationship" (such as "the cup on the table") to form the space layout information, and then the space layout information is checked by the collision detection algorithm (such as the separation axis theorem) to check whether there is spatial overlap; if there is overlap, the position or attitude of the target object model is adjusted until the constraint is met; finally, the layout result (space layout information) is saved in a structured format (such as JSON) to form at least one simulation task. Optionally, multiple space layout information can be generated at the same time, and the relative positions between the target object model and the associated object model in each space layout information are different. For the space layout information that meets the constraint condition, the corresponding simulation task can be directly generated without adjustment. In this way, multiple simulation tasks can be generated at the same time for one natural language task description, which can improve the efficiency of simulation task generation.

[0044] The robot simulation task generation method in the present application automatically extracts task requirements, matches models and layout constraints according to natural language task descriptions, reduces the operation of manually setting scenes and adjusting layouts in traditional methods, and improves the efficiency of simulation task generation. Secondly, the extraction process from natural language to target object, associated object and position relationship in the present application can ensure the semantic matching of virtual models and task descriptions (such as "playground" will not be matched to "playground" scene), avoid semantic disconnection, and ensure semantic consistency. In addition, the present application ensures that the virtual scene conforms to the physical logic (such as objects cannot be suspended or penetrated) through the boundary constraint and layout rule of the target work space combined with the position relationship, improves the authenticity of the simulation task, and enhances the scene rationality. In addition, the present application can generate multiple simulation tasks that meet the constraints based on the same natural language description (such as different positions of the cup and the table layout), provide diversified training data for the robot, and enhance its adaptability.

[0045] In one embodiment, the associated object includes an initial associated object and a target associated object; the position relationship includes an initial position relationship and a target position relationship; the work space includes the size parameter and the position center point parameter of each associated object. Among them, as shown in the figure, the task requirement information of the simulation task is extracted from the natural language task description, including: Figure 2 ​Step 210, calling a preset large language model to perform dependency syntax analysis on the natural language task description to extract a predicate; Step 220, determining a target object, an initial associated object, a target associated object, an initial position relationship between the target object and the initial associated object, and a target position relationship between the target object and the target associated object from the natural language task description based on the predicate; Step 230, forming a structured language file for embodying task requirements according to the target object, the initial associated object, the target associated object, the initial position relationship, and the target position relationship.

[0046] In this embodiment, the size parameter and the position center point parameter of the associated object are spatial attribute data of the associated object. The size parameter refers to the three-dimensional spatial size (such as length x width x height) of the associated object, and the position center point parameter refers to the three-dimensional coordinates (such as [x, y, z]) of the associated object in the work space, which is used to constrain the layout range of the target object. By setting appropriate size parameters and position center point parameters of the associated object, the initial / target position relationship between the target object and the associated object can be quantified as specific spatial constraints, ensuring that the layout of the target object in the work space conforms to physical logic (such as a cup cannot exceed the spatial range of a table or a drawer).

[0047] The initial associated object represents an object that has a direct spatial association with the initial position of the target object in the simulation task scene to be performed, and is a carrier or reference body of the initial position of the target object. That is, the initial associated object is an associated object of the position of the target object before performing an action, and is used to define the starting state of the task. The target associated object represents an object that has a direct spatial association with the final position (target position) of the target object in the simulation task scene to be performed, and is a carrier or reference body of the target position of the target object. The target position is the destination to which the target object needs to be moved.

[0048] The initial position relationship represents the spatial relative state of the target object and the initial associated object at the beginning of the task, which is represented by the initial state predicate and reflects the spatial constraint (i.e., the initial position relationship, such as "on" or "next to") of the target object and the initial associated object before performing an action. The target position relationship represents the spatial relative state of the target object and the target associated object at the completion of the task, which reflects the spatial constraint (i.e., the target position relationship, such as "inside" or "on") of the target object and the target associated object after performing an action.

[0049] Taking "pick up the red cup from the table and put it into the drawer" as an example, "pick up" is the predicate, and "red cup" is the target object; "table" is the initial associated object (the cup is initially on the table); "drawer" is the target associated object (the cup finally needs to be put into the drawer); "cup on table" (OnTop) is the initial position relationship; and "cup in drawer" (Inside) is the target position relationship.

[0050] A large language model is used to parse the syntax structure of a natural language task description, identify and output a predicate. The large language model can be any suitable model such as a GPT series model, an LLaMA model, etc. The natural language task description is input into a preset large language model. The model first determines the logical structure of objects, actions, and spatial position relationships in the sentence through dependency syntax analysis, and then separates the associated objects, target objects, and various states between the associated objects and target objects by combining predicate extraction rules (such as semantic constraints based on a BDDL (Behavior Domain Definition Language) template library) and natural language descriptions to form the predicate relationship of the natural language, such as extracting the predicate relationship based on a BDDL prompt.

[0051] The predicate relationship can be represented as: initial state: OnTop(cup, table); target state: Inside(cup, drawer).

[0052] After forming the predicate relationship, a corresponding structured language file is constructed based on the predicate relationship. The structured language file can be a BDDL language file. An exemplary BDDL language file can be represented as: objects: - cup - table - drawer initial_conditions: - OnTop(cup, table) goal_conditions: - Inside(cup, drawer) This embodiment extracts the predicate by combining a large language model with dependency syntax analysis, which can accurately separate the initial state and target state of the target object and avoid relationship confusion caused by natural language ambiguity (such as distinguishing different associated objects corresponding to "from the table" and "put into the drawer"); by converting the spatial position relationship in natural language into a structured language file, it provides a clear logical basis for subsequent simulation task layout constraints (such as collision detection, spatial layout).

[0053] In one embodiment, as shown in Figure 3 the target object model matching the target object is queried from the preset object library, including: Step 310, querying the object label information matching the target object from the preset object library.

[0054] Step 320, when there are multiple matching item label information, obtain the item description information of each item label information, and analyze the scene matching between each item description information and the natural language task description.

[0055] Step 330, determine the target object model from the multiple item label information based on the scene matching.

[0056] In this embodiment, the item description information is a supplementary description of the detailed features, applicable scenarios, functional details, and other dimensional characteristics of the item model, which is a supplement and expansion of the item label information, and contains more specific scenario-based descriptions, such as the size, material, weight, suitable working space, price, and other dimensional information of the corresponding item described in the item description information. For example, the description information of a certain ceramic cup item model is "white ceramic material mug with handle, resistant to high temperature, suitable for holding hot water, coffee, etc. in the kitchen or office, capacity 350ml".

[0057] The scene matching represents the degree of fit between the item description information and the scene requirements implied by the natural language task description. Specifically, it can be quantified by semantic similarity calculation (such as keyword overlap degree, scene attribute matching degree), and the higher the similarity value, the more suitable the item is for the task scenario. For example, the natural language task description is "take a cup that can hold hot water in the kitchen", then the scene matching of "ceramic cup (resistant to high temperature, suitable for kitchen)" is higher than that of "glass cup (not resistant to high temperature)".

[0058] Optionally, when the natural language description contains not only the target object but also the key features of the target object, the target object and its key features (such as category, attribute) can be extracted and converted into retrieval keywords, which are matched with the label information of the models in the item library to filter out models whose label coincidence degree reaches a preset threshold. For example, the target object is "cup" and its color is red, then the keywords "category: cup; color: red" can be extracted, and models whose labels contain "cup" and "color: red" are retrieved in the item library to obtain candidate models A (label: "cup, red, plastic"), model B (label: "cup, red, ceramic"), and model C (label: "cup, red, glass").

[0059] For multiple candidate models, the adaptation degree of each model is determined through the analysis of the degree of fit between its detailed description information and the task scene requirement. For example, the text analysis tool (such as semantic similarity algorithm) can be called to compare the item description information of each model with the scene (such as "kitchen" "office") and functional requirement (such as "filling hot water" "light and easy to take") implied in the natural language task description, and calculate the scene matching score. One or more models with the highest score are selected; if there are models with the same score, the final result can be determined by further combining other rules (such as model accuracy, commonality), or each model with the same score can be regarded as a target object model, and the simulation task is generated independently based on each target object model.

[0060] For example, the natural language task description is "take a red cup to fill hot water in the kitchen": extract the task scene requirement "kitchen scene" "fill hot water". If the description information of the above candidate models is: model A: "red plastic cup, light, suitable for office, not heat-resistant"; model B: "red ceramic cup, heat-resistant, suitable for kitchen, capacity 300 ml"; model C: "red glass cup, transparent, suitable for living room, easy to break when hot and cold".

[0061] Analyze the scene matching: model B has the highest matching degree (score 90) with "kitchen" and "filling hot water"; model C has the second highest matching degree (score 60, material is not suitable for hot water); model A has the lowest matching degree (score 30, both scene and function are not matched). Model B is determined as the target object model matched with the target object "red cup".

[0062] In this embodiment, through the double-layer screening mechanism of "label preliminary screening + description information scene matching", not only the most suitable target object model can be quickly located to improve the retrieval efficiency, but also the mis-matching caused by relying on a single label can be avoided to ensure that the selected model is consistent with the task in terms of function and applicable scene (such as the kitchen scene preferentially selects the heat-resistant material model), which can more realistically simulate the characteristics of the items in the real scene (such as the rationality of the ceramic cup for filling hot water), and provides a more realistic environment for robot training.

[0063] In one embodiment, as shown in Figure 4 The target scene matched with the associated object and the target work space of the target scene are queried from the preset scene library, including: Step 410, based on the semantic understanding of the natural language task description, the target scene matching the task requirement information is queried from the scene library.

[0064] Step 420, the basic work space matched with the associated object and the target scene is queried from the preset scene library.

[0065] At step 430, based on the semantic understanding of the natural language task description, the position and size information of the basic work space is configured to form the target work space.

[0066] In this embodiment, it can be detected whether the scene is directly mentioned in the natural language. If it is included, the scene is taken as the target scene. If the scene information is not mentioned, the scene suitable for the natural language task description is analyzed through semantic understanding, and the analyzed scene is taken as the target scene. For example, the natural language is “put the clothes on the bed into the wardrobe in the bedroom”, the directly mentioned scene is “bedroom”, and for the natural language “put the clothes on the bed into the wardrobe”, the target scene matched through semantic understanding is “bedroom”.

[0067] Each scene includes one or more basic work spaces, which are predefined general operation area templates (such as “bookshelf operation area” and “refrigerator operation area”) matched with the function of the associated object in the scene. The basic space attribute (such as a plane / steric area, a function label) is included, but one or several parameters such as the related position and size can be fixed values or adjustable values.

[0068] According to the type (such as “wardrobe” and “bookshelf”) and function (such as “storing clothes” and “storing books”) of the associated object, the basic work space (such as “wardrobe front storage area”) matched with the function label of the associated object is searched in the set of basic work spaces associated with the target scene.

[0069] Further, after determining the basic work space, it is also necessary to further consider whether the space size of the basic work space meets the requirements of the corresponding work task. Specifically, after determining the basic work space, the position (such as the position center point parameter) and size information (such as the size parameter) of each associated object can be further configured or modified for the basic work space by combining the operation range of the action in the task (such as “putting clothes” requiring sufficient plane space), the size of the associated object (such as the width of the wardrobe being 1.2 m), and the position of other facilities in the scene (such as no obstacles within 0.5 m beside the wardrobe), such as configuring the position center point coordinates and length x width x height size of each associated object to meet the convenience of the interaction between the target object and the associated object (such as the work space covering the wardrobe door opening range).

[0070] Similar to the selection method of the target object, the finally determined target scene and target work space can also have one or more, and the electronic device can arrange and combine each determined target object, target scene, and target work space, generate a simulation task for each combination, so as to improve the number and diversity of the simulation task generation.

[0071] In one embodiment, the layout constraint information of the target object model in the target working space is formed based on the position relationship, including: determining a first number of interference object models based on the target object and the associated object; forming the layout constraint information of the target object model and the interference object model in the target working space based on the position relationship.

[0072] In this embodiment, one or more interference objects can be appropriately added to the generated simulation task in addition to the target object and the associated object mentioned in the natural language. The interference object is a virtual object of a non-task operation object introduced to simulate the complexity of the real environment. The interference object has certain semantic association with the target object, the associated object or the task scene (such as the interference objects in the "kitchen scene" are mostly tableware and food materials, not tools), and forms a certain challenge to the perception or operation of the robot through the occupation of space, the blocking of part of the path, etc., increases the scene complexity, and the position relationship between the interference object and the associated object and the position relationship between the target object and the associated object can be the same, such as both being above the interference object.

[0073] The first number can be a fixed number preset by the system (such as 2, 3), or a number adaptively calculated according to the natural language task description.

[0074] Specifically, based on the item tag information of the target object, the model associated with the tag can be retrieved from the model library to ensure that the interference object and the task scene are semantically matched. For the retrieved model, a first number of models can be randomly selected as interference object models. The selected models can be the same or different, such as two identical interference object models can be selected. For example, 2 candidate models (such as "banana" and "orange") are randomly selected as interference object models, which belong to the fruit category together with the target object "apple", and conform to the scene semantics.

[0075] For the selected target object model, interference object model and target working space, the position relationship between the target object and the associated object in the target working space is used as the first constraint condition, the position relationship between the interference object and the associated object is generated as the second constraint condition, and the position relationship between the interference object and the target object is generated as the third constraint condition. The combination of the first constraint condition, the second constraint condition and the third constraint condition forms the space constraint information. Among them, the position relationship between the interference object and the associated object can be the same as the position relationship between the target object and the associated object, such as both being above the associated object, and the position relationship between the interference object and the target object is that there is no collision and spatial overlap between the interference object and the target object. In addition, the space constraint information also includes that the target object model and the interference object model cannot exceed the working area of the target working space.

[0076] For the position relationship "apple in the fruit bowl", the target work space is "operation table area (center point [1.2, 0.8, 0], size 1.5m x 0.6m x 0.75m)", and the interference objects are "banana" and "orange". For example, the first constraint condition (core constraint) is that all vertex coordinates of the apple model need to be within the internal space range of the fruit bowl model (the bottom z coordinate of the fruit bowl ≤ the bottom z coordinate of the apple ≤ the top z coordinate of the fruit bowl, and the x and y coordinates are within the range of the fruit bowl plane); the second constraint condition (interference object constraint) is that the banana and orange models need to be placed around the fruit bowl model (the distance from the edge of the fruit bowl is ≥5cm); the third constraint condition (collision constraint) is that the three-dimensional space of the apple model, the banana model and the orange model has no overlap, in addition, the longest side direction of the banana model cannot block the upper area of the apple model (to ensure the smoothness of the robot grabbing path); and the fourth constraint condition (boundary constraint) is that all vertex x coordinates of the apple, the fruit bowl, the banana and the orange need to be within the range of [1.2-0.75, 1.2+0.75] (the length of the operation table is 1.5m), the y coordinates need to be within the range of [0.8-0.3, 0.8+0.3] (the width of the operation table is 0.6m), and the z coordinates need to be within the range of [0, 0.75] (the height of the operation table is 0.75m).

[0077] In this embodiment, the first number of interference object models are introduced to simulate the non-target objects (such as other objects on the kitchen table) existing in the real environment, so that the robot training is closer to the actual scene and the anti-interference ability of the robot is enhanced.

[0078] In one embodiment, as shown in Figure 5 the first number of interference object models are determined based on the target object and the associated object, including: In step 510, the projection area of the target object model on the surface of the associated object is determined based on the target object, the associated object and the association relationship.

[0079] According to the three-dimensional model parameters of the target object and the associated object and the spatial position relationship between the two, the projection area of the target object on the specific surface of the associated object is calculated. The surface can be a surface determined according to the association relationship, such as a horizontal surface. For example, "OnTop" corresponds to the top surface of the associated object, and "Inside" corresponds to the internal bottom surface of the associated object. The size of the target object is set in the target object model, and based on the position, pose and size between the target object model and the associated object model, the corresponding projection area can be calculated.

[0080] In step 520, the constraint area of the interference object is determined based on the projection area and the associated object.

[0081] The constraint area is a threshold (maximum allowable value) for limiting the total projection area of all interference models on the associated object surface, ensuring that the interference models do not occupy too much space. Alternatively, the constraint area can be calculated based on the target object projection area and the available area of the associated object surface (e.g., a certain percentage of the total area of the associated object surface minus the target object projection area), which is a key indicator for screening interference models.

[0082] Alternatively, the difference between the available area of the associated object surface and the target object projection area can be calculated first, and the product of the difference and a preset first coefficient is used as the corresponding constraint area. The coefficient is <1, such as 0.4, 0.5, 0.6, 0.7, etc. For example, the top surface area of the associated object "plate" is 300 cm 2 , the target object projection area is 50 cm 2 , and the constraint area can be set to (300-50) x 60% = 150 cm 2 (the total projection area of all interference models cannot exceed 150 cm 2 ).

[0083] In another embodiment, the constraint area = target object projection area x second coefficient. The second coefficient can be any suitable value, such as any suitable value between 1 and 2 (e.g., the second coefficient is 1.5).

[0084] Step 530, based on the target object and the associated object, determine a plurality of candidate objects matching the target scene.

[0085] Specifically, the label information of the target object (e.g., "mug" label: tableware, cup), the associated object (e.g., "desk" label: furniture, office), and the target scene (e.g., "office" label: office scene) is obtained; from the object library, models with labels intersecting with the above labels (e.g., "office" scene, models related to "office" and "tableware") are screened out, models completely identical to the target object are excluded, and models meeting semantic association are retained as candidate objects.

[0086] For example, the target object "mug" (label: tableware, cup), the associated object "desk" (label: office, furniture), and the target scene "office": from the object library, models with labels containing "office" and "tableware" are retrieved, obtaining "pen model", "notebook model", "file folder model", and "small potted plant model"; the above models are all matched with the office scene, forming a candidate object set.

[0087] Step 540, based on the constraint area, determine a first number of interference models from the plurality of candidate objects.

[0088] In this embodiment, a first number of models are selected from the candidate set to ensure that the total projection area does not exceed the constraint area. Understandably, there can be more than one selection scheme for the feasible candidates, and the electronic device can randomly select one or more selection schemes that meet the conditions, combine each selection scheme with the target object, target scene, and target work space, and generate subsequent simulation tasks based on each selection scheme.

[0089] In this embodiment, by quantitatively calculating the projection area and the constraint area, the total space ratio of the interference object is avoided to be too large, and the core operation space of the target object and the associated object is ensured not to be squeezed, which conforms to the space distribution law of the real scene; by selecting the candidate object based on the target object, the associated object, and the scene, irrelevant models (such as a “stir-frying pot” in an office scene) are avoided to be introduced, and the scene rationality of the simulation task is ensured.

[0090] In one embodiment, the layout constraint information requires that there is no collision and spatial overlap between the target object and the interference object. As shown in FIG. 6A, at least one simulation task meeting the task requirement information is generated under the layout constraint information, including: Figure 6 Step 610, based on the positional relationship, the first pose information of the target object model and the second pose information of each interference object are set in the target work space to form the spatial layout information of the target object and the interference object in the target work space.

[0091] Specifically, the first pose information of the target object model can be generated first, and the second pose information of each interference object model is generated in sequence based on the generated first pose information, and the generation manner of the first pose information and the second pose information meets the positional relationship between the target object, the interference object, and the associated object. For example, the generated first pose information indicates that the target object is on top of the interference object (Ontop).

[0092] Taking the positional relationship “apple on the plate” and the target work space “kitchen operation table (range [x: 0.4-0.6, y: 0.2-0.4, z: 0.7-0.7])” as an example, the first pose information is represented as: apple model position [0.5, 0.3, 0.7] (center of the plate), attitude [0°, 0°, 0°]; the second pose information is represented as: 2 interference objects (banana, orange) in the apple surrounding area ≥5cm in the operation table, banana position [0.55, 0.3, 0.7], attitude [0°, 45°, 0°] (inclined along the length direction); orange position [0.5, 0.35, 0.7], attitude [0°, 0°, 0°]; and the spatial layout information is integrated as the pose information of the three objects.

[0093] ​Step 620, detect whether the spatial layout information meets the layout constraint information. If yes, execute step 630, otherwise, return to execute step 610.

[0094] Step 630, generate the corresponding simulation task.

[0095] In this embodiment, it is verified whether the spatial layout of the target object and the interference object meets the "no collision, no overlap" constraint, and if so, the simulation task is output. Collision detection algorithm (such as Separating Axis Theorem SAT) can be used to check whether the three-dimensional space of the target object model and each interference object model overlaps (i.e. whether there is an intersection); if there is no overlap between all models, it is determined that the constraint is met, the spatial layout information is saved in a structured format (such as JSON), and the simulation task is generated, which is convenient for reuse and sharing.

[0096] If collision is detected, the pose of the target object or the interference object is adjusted again, and detection is performed again until the constraint is met. Specifically, the collision reason (such as the interference object being too close) can be analyzed; based on the analyzed collision reason, the pose is adjusted, which can improve the adjustment efficiency. For example, if the collision reason is that the interference object is too close, the interference object position can be slightly moved (such as offset by 2cm along the x-axis) or the attitude can be rotated (such as changing the banana inclination angle), and the first pose of the target object remains unchanged (priority is given to guarantee the core position relationship). Among them, the iteration number of pose adjustment does not exceed a preset threshold (such as 200 times).

[0097] The steps 610 to 630 described above can be repeatedly executed, thereby generating multiple simulation tasks. For example, it can be executed three times, so that three simulation tasks can be generated, which are represented as: json task1.json task2.json task3.json.

[0098] Each json file (task1.json, task2.json, task3.json) is a simulation task file stored in a structured format, and the file can be represented as: { "cup": {"position": [0.2, 0.8, 0.5], "orientation": [0,0,0], "model":"cup_model_003"}, "table": {...}, "relations": ["OnTop(cup, table)", "Inside(cup, drawer)"] } In one embodiment, generating at least one simulation task meeting the task requirement information under the layout constraint information comprises: randomly setting target object models and interference object models in the target work space based on the position relationship to form spatial layout information of multiple groups of target objects and interference objects; detecting whether each group of spatial layout information meets the layout constraint information; storing spatial layout information meeting the layout constraint information in a structured format to generate a corresponding simulation task.

[0099] In the target work space, pose combinations (combination of the first pose information and the second pose information described above) of multiple groups of target objects and interference objects are randomly generated according to the position relationship framework to form multiple spatial layout information. In the process of random generation, the random range is determined according to the position relationship described above, so that the target objects and the interference objects are both in the reasonable region of the associated objects (for example, the x and y coordinate values of the target objects on the table should be within the range of the table top, and the z coordinate value = the height of the table top; the position of the interference object should be within the work space and maintain a basic safety distance from the target object).

[0100] The first pose information of the target object model and the second pose information of the interference object model are generated within the random range. The layout constraint information can also be detected by using the collision detection algorithm (such as the Separating Axis Theorem SAT) described above, and all layout constraint information layouts that meet the collision detection and position relationship verification at the same time are recorded.

[0101] For the generated simulation task, the simulation environment can be loaded and the simulator can be started to perform simulation training on the robot and collect simulation training result data. For example, the scene can be loaded in psi-lab through a json file to start the simulator; the robot can be trained in the simulator through reinforcement learning, imitation learning, action planning, etc. and the training result data can be collected.

[0102] Based on the simulation training result data, the evaluation indexes of the simulation task difficulty and the task completion rate of the robot are evaluated, and based on the evaluation indexes, the natural language task description, the target object model matching, the selection of the target scene and the target work space, and the optimization of the layout constraint information described above are optimized, so that the simulation training effect of the robot can be further improved.

[0103] The embodiment generates the first pose of the target object model and the second pose of the interference object model at one time to form multiple spatial layout information, detects the layout constraint information for each spatial layout information, and retains the spatial layout information meeting the constraint condition, so as to improve the efficiency of simulation task generation.

[0104] In one embodiment, a computer readable storage medium is provided, and executable instructions are stored on the computer readable storage medium. When the instructions are executed by a processor, the processor executes the steps in the above method embodiments.

[0105] In one embodiment, an electronic device is also provided, including one or more processors; and a memory storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the steps in the above method embodiments.

[0106] In one embodiment, such as Figure 7 The diagram illustrates the structure of an electronic device used to implement an embodiment of this application. The electronic device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0107] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0108] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer-readable medium carrying instructions that, in such embodiments, can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the instructions are executed by central processing unit (CPU) 701, the various method steps described in this application are performed.

[0109] Finally, it should be noted that the above-mentioned embodiments are merely used to illustrate the technical solutions of the present application, rather than limit the present application; even though the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some or all of the technical features thereof can be replaced equivalently; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0110] Furthermore, those skilled in the art could understand that, although some embodiments herein include certain features rather than other features included in other embodiments, the combination of features of different embodiments means to be within the scope of the present application and form different embodiments. For example, all the above embodiments can be used in any combination. The information disclosed in this part of background is merely intended to deepen the understanding of the general background of the present application, and should not be regarded as acknowledging or implying in any form that the information constitutes the prior art known to those skilled in the art.

Claims

1. A method for generating robot simulation tasks, characterized in that, The method includes: Obtain a natural language task description, and extract the task requirement information of the simulation task from the natural language task description. The task requirement information includes the target object, associated objects, and positional relationship between the target object and associated objects that the robot needs to grasp. Retrieve target object models that match the target object from a pre-defined item database; Retrieve the target scene that matches the associated object and the target workspace of the target scene from the preset scene library; Based on the positional relationship, the layout constraint information of the target object model in the target work space is formed, and at least one simulation task that meets the task requirements information is generated under the layout constraint information.

2. The method according to claim 1, characterized in that, The process of forming the layout constraint information of the target object model in the target work space based on the positional relationship includes: A first number of interfering object models are determined based on the target object and the associated object; Based on the positional relationship, layout constraint information for the target object model and the interference object model is generated in the target work space.

3. The method according to claim 2, characterized in that, The step of determining a first number of interfering object models based on the target object and the associated objects includes: The projected area of ​​the target object model on the surface of the associated object is determined based on the target object, associated objects, and associated relationships. The constraint area of ​​the interfering object is determined based on the projected area and the associated object; Based on the target object and the associated object, multiple candidate objects matching the target scene are identified; Based on the constraint area, a first number of interference models are determined from multiple candidates.

4. The method according to claim 2, characterized in that, The layout constraint information requires that there be no collision or spatial overlap between the target object and the interfering object; The step of generating at least one simulation task that conforms to the task requirements under the layout constraint information includes: Based on the positional relationship, the first pose information of the target object model and the second pose information of each interference object are set in the target work space to form the spatial layout information of the target object and interference objects in the target work space. Detect whether the spatial layout information satisfies the layout constraint information; if it does, generate the corresponding simulation task. If the conditions are not met, the process returns to re-execute the setting of the first pose information of the target object model and the second pose information of each interfering object in the target work space based on the positional relationship, until the layout constraint information is met.

5. The method according to claim 2, characterized in that, The layout constraint information requires that there be no collision or spatial overlap between the target object and the associated object; The step of generating at least one simulation task that conforms to the task requirements under the layout constraint information includes: Based on the positional relationship, the target object model and the interference object model are randomly set in the target work space to form multiple sets of spatial layout information of target objects and interference objects; Detect whether each group of spatial layout information satisfies the layout constraint information; The spatial layout information that satisfies the layout constraints is stored in a structured format to generate the corresponding simulation task.

6. The method according to claim 1, characterized in that, The associated objects include initial associated objects and target associated objects; the positional relationships include initial positional relationships and target positional relationships; the workspace includes the size parameters and position center point parameters of each associated object; The step of extracting the task requirement information of the simulation task from the natural language task description includes: The pre-defined large language model is invoked to perform dependency parsing on the natural language task description and extract the predicates. Based on the predicate, the target object, initial associated object, target associated object, initial positional relationship between the target object and the initial associated object, and target positional relationship between the target object and the target associated object are determined from the natural language task description; Based on the target object, initial associated object, target associated object, initial positional relationship, and target positional relationship, a structured language file is formed to reflect the task requirements.

7. The method according to claim 1, characterized in that, The step of retrieving a target object model that matches the target object from a preset item database includes: Retrieve item tag information that matches the target item from a preset item database; When multiple matching item tag information exists, obtain the item description information for each item tag information, and analyze the scene matching between each item description information and the natural language task description; The target object model is determined from multiple item tag information based on the scene matching.

8. The method according to claim 1, characterized in that, The step of retrieving the target scene and the target workspace of the target scene that match the associated object from the preset scene library includes: Based on the semantic understanding of the natural language task description, a target scene matching the task requirement information is retrieved from the scene database; Retrieve a basic workspace from a pre-defined scene library that matches the associated object and the target scene; Based on the semantic understanding of the natural language task description, the location and size information of the basic task space are configured to form the target task space.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores executable instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1 to 8.

10. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to perform the method as described in any one of claims 1 to 8.

Images