Object completion method, apparatus, device, storage medium and program product

Abstract

The application relates to an object completion method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring missing object information of a missing object in a current simulation scene; determining a target reference object type and a target relative spatial relationship matched with the missing object type based on a target matching rule, and acquiring a target reference object instance corresponding to the target reference object type from a current object instance of the current simulation scene; the target matching rule comprises a plurality of object types and relative spatial relationships between different object types; target pose information of the missing object is determined based on reference pose information of the target reference object instance in the current simulation scene and the target relative spatial relationship; and the target pose information is used for completing the missing object in the current simulation scene. The method can complete the missing object in the simulation environment and improve the integrity of the simulation environment.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an object completion method, apparatus, device, storage medium, and program product. Background Technology

[0002] As cloud-native agents demonstrate superior knowledge utilization and reasoning capabilities in knowledge-intensive tasks, their combination of autonomous decision-making, learning, and interaction features reveals enormous application potential in fields such as operations and maintenance, customer service, and industrial automation. To ensure the effectiveness and business stability of these agents in complex tasks, thorough evaluation is crucial. Given the high cost of real-world data collection and annotation, as well as the uncontrollable nature of real-world environments, using simulated environments to support the development and validation of agents has become a current research hotspot.

[0003] After receiving task instructions, an intelligent agent typically inputs these instructions into a program code generation model, which then generates program code to complete the task. The simulation environment needs to provide an execution space containing all necessary objects and interaction steps to verify the correctness and executability of the task program code. However, pre-built simulation scenarios often fail to encompass all the objects required in complex tasks, resulting in low completeness and an inability to reflect the requirements of the expected task, thus affecting the agent's action planning and task execution. Summary of the Invention

[0004] Therefore, it is necessary to provide an object completion method, apparatus, device, storage medium, and program product that can improve the integrity of the simulation environment in response to the above-mentioned technical problems.

[0005] Firstly, this application provides an object completion method, including:

[0006] Obtain missing object information for missing objects in the current simulation scenario; the missing object information includes the missing object type of the missing object.

[0007] Based on the target matching rules, the target reference object type and target relative spatial relationship that match the missing object type are determined, and the target reference object instance corresponding to the target reference object type is obtained from the current object instance of the current simulation scene; the target matching rules include multiple object types and relative spatial relationships between different object types;

[0008] Based on the reference pose information of the target reference object instance in the current simulation scene and the relative spatial relationship of the target, the target pose information of the missing object is determined; the target pose information is used to complete the missing object in the current simulation scene.

[0009] In one of the embodiments, the determining the target pose information of the missing object based on the reference pose information of the reference object instance in the current simulation scene and the target relative spatial relationship comprises:

[0010] obtaining a preset target optimization model, the target optimization model comprising at least one of a spatial cost model and a collision cost model, the spatial cost model being used to calculate a degree of incoordination of the missing object and the target reference object instance in space, and the collision cost model being used to calculate a degree of physical overlap of the missing object and the current object instance;

[0011] inputting the reference pose information into the target optimization model to obtain pose information of the missing object corresponding to a minimum model value of the target optimization model under a constraint condition of satisfying the target relative spatial relationship, so as to obtain the target pose information.

[0012] In one of the embodiments, the inputting the reference pose information into the target optimization model to obtain pose information of the missing object corresponding to a minimum model value of the target optimization model under a constraint condition of satisfying the target relative spatial relationship comprises:

[0013] in a case where the target optimization model comprises the collision cost model, obtaining, based on the reference pose information and adjacent pose information, pose information of the missing object corresponding to a minimum model value of the target optimization model under a constraint condition of satisfying the target relative spatial relationship, so as to obtain the target pose information;

[0014] wherein the adjacent pose information is pose information of an adjacent object instance corresponding to the target reference object instance; and the adjacent pose information is used to combine the pose information of the missing object to determine the model value of the collision cost model.

[0015] In one of the embodiments, the target reference object instance is multiple; and the inputting the reference pose information into the target optimization model to obtain pose information of the missing object corresponding to a minimum model value of the target optimization model under a constraint condition of satisfying the target relative spatial relationship comprises:

[0016] for each of the target reference object instances, obtaining, based on the reference pose information of the target reference object instance, pose information of the missing object corresponding to a minimum model value of the target optimization model under a constraint condition of satisfying the target relative spatial relationship, so as to obtain multiple candidate pose information corresponding to the multiple target reference object instances;

[0017] Compare model values of the target optimization model corresponding to the plurality of candidate pose information respectively, and obtain the target pose information according to a comparison result.

[0018] In one of the embodiments, the missing object information further comprises associated object information of the missing object; the associated object information is used to indicate a type of dependent object expected by the missing object in a task context; and the determining of the target reference object type and the target relative spatial relationship matched with the missing object type based on the target matching rule comprises:

[0019] Determining a plurality of candidate reference object types matched with the missing object type based on the target matching rule;

[0020] Selecting the target reference object type matched with the associated object information from the plurality of candidate reference object types;

[0021] Determining the target relative spatial relationship corresponding to the target reference object type and the missing object type based on the target matching rule.

[0022] In one of the embodiments, the missing object information further comprises associated spatial information of the missing object; the associated spatial information is used to represent a space region to which the missing object is expected to belong in a task context; and the obtaining of the target reference object instance corresponding to the target reference object type from the current object instances of the current simulation scene comprises:

[0023] Obtaining a plurality of candidate reference object instances corresponding to the target reference object type from the current object instances of the current simulation scene;

[0024] Selecting the target reference object instance matched with the associated spatial information from the plurality of candidate reference object instances.

[0025] In a second aspect, the application further provides an object completion device, comprising:

[0026] An obtaining module is configured to obtain missing object information of a missing object in a current simulation scene; the missing object information comprises a missing object type of the missing object;

[0027] A matching module is configured to determine a target reference object type and a target relative spatial relationship matched with the missing object type based on a target matching rule, and obtain a target reference object instance corresponding to the target reference object type from current object instances of the current simulation scene; the target matching rule comprises a plurality of object types and relative spatial relationships between different object types;

[0028] The completion module is configured to determine target pose information of the missing object based on the reference pose information of the target reference object instance in the current simulation scene and the target relative spatial relationship, and the target pose information is used to complete the missing object in the current simulation scene.

[0029] In a third aspect, the present application also provides a computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the steps of the object completion method provided in the first aspect of the present application when executing the computer program.

[0030] In a fourth aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the object completion method provided in the first aspect of the present application when executed by a processor.

[0031] In a fifth aspect, the present application also provides a computer program product, including a computer program, and the computer program implements the steps of the object completion method provided in the first aspect of the present application when executed by a processor.

[0032] The object completion method, device, equipment, storage medium and program product described above, by obtaining missing object information of a missing object in a current simulation scene, the missing object information including a missing object type of the missing object, determining a target reference object type and a target relative spatial relationship matched with the missing object type based on a target matching rule, and obtaining a target reference object instance corresponding to the target reference object type from current object instances of the current simulation scene, determining target pose information of the missing object based on reference pose information of the target reference object instance in the current simulation scene and the target relative spatial relationship. The present application identifies a missing object required by a task and not contained in a current scene, uses a preset knowledge rule and semantic association to screen specific scene elements and their reasonable spatial relationships most suitable for bearing or associating the missing object, converts these semantic level matches into accurate target pose information, can complete the missing object in a simulation environment, and improves the integrity of the simulation environment. At the same time, the target matching rule makes the expected pose of the missing object comply with the scene semantic logic, can significantly improve the executability and adaptability in a cloud native agent scene, and avoids the situation that Agents fail to execute a task or the task logic is interrupted due to lack of necessary objects or unreasonable placement positions. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other related drawings can be obtained without creative labor on the basis of these drawings.

[0034] Figure 1 An application environment diagram of the object completion method in an embodiment;

[0035] Figure 2 A flowchart of the object completion method in an embodiment;

[0036] Figure 3 A schematic diagram of the cloud native intelligent agent performing a task in an embodiment;

[0037] Figure 4 A flowchart of determining target pose information in an embodiment;

[0038] Figure 5 A flowchart of determining target reference object type in an embodiment;

[0039] Figure 6 A flowchart of determining target reference object instance in an embodiment;

[0040] Figure 7 A flowchart of obtaining target pose information in an embodiment;

[0041] Figure 8 A structural block diagram of the object completion apparatus in an embodiment;

[0042] Figure 9 An internal structure diagram of the computer device in an embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0044] It should be noted that the terms "first", "second", etc. used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" and any variations thereof used in the present application are intended to cover non-exclusive inclusion. The term "multiple" used in the present application refers to two and more than two. The term "and / or" used in the present application refers to one of the options or any combination of multiple options.

[0045] Since there are problems such as great difficulty in data collection, high cost of labeling, and uncontrollable environment in real environment, and the simulation environment can generate a large amount of scene data with fine labeling at low cost, verifying the correctness and executability of cloud native agents (Agents) executing tasks through the simulation environment can lay a solid foundation for subsequent evaluation of cloud native agents. However, in the simulation environment, the pre-constructed simulation scene often cannot contain the objects required in various tasks, so that the objects required by the Agents to execute the tasks are difficult to completely present in the simulation scene, thereby affecting the smooth completion of the tasks by the Agents.

[0046] Based on this, in order to solve the above technical problems, the embodiment of the present application provides an object completion method, which can be applied to an application environment as shown in the figure. Figure 1 As shown in the figure, the terminal 102 and the server 104 can communicate through a network, such as a wired or wireless network. The data storage system can store data required by the server 104 to process, for example, it can store pre-recorded target matching rules including multiple object types and relative spatial relationships between different object types. The data storage system can be integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can be, but is not limited to, various desktop computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The terminal 102 can be deployed with a simulator, which can create and maintain a virtual and interactive environment that can simulate physical laws (such as gravity, collision, friction) in the real world and allow virtual entities (such as Agents, robots, objects) to perceive, move and interact in it. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The server 104 can be a background server of the above-mentioned simulator, used to provide background services for the simulator.

[0047] The object completion method provided by the embodiments of the present application can be executed by a computer device, which refers to an electronic device with data calculation, processing and storage capabilities. Figure 1 For example, the application environment shown in the figure can be used to execute the object completion method by the terminal 102 alone (e.g., a simulator installed and running in the terminal 102), by the server 104 alone, or by the terminal 102 and the server 104 in cooperation, and the present application does not limit this.

[0048] In an exemplary embodiment, as shown in the figure, an object completion method is provided, which is applied to the terminal 102 in the figure as an example and includes the following steps 202 to 206. Figure 2 Figure 1 In the figure, the terminal 102 in the figure is taken as an example to illustrate the object completion method, which includes the following steps 202 to 206.

[0049] In step 202, missing object information of a missing object in a current simulation scene is obtained, and the missing object information includes a missing object type of the missing object.

[0050] The missing object refers to an object mentioned in a task description but actually not existing in an initial state of the current simulation scene when an agent executes a preset task or instruction in a simulation environment. For example, if the task instruction received by the agent is “take the juice on the table to the TV in front of it”, and the current simulation scene data does not contain a model or instance of “juice”, the “juice” can be determined as a missing object.

[0051] The missing object information can refer to necessary data for describing and identifying the missing object, which at least includes the missing object type of the missing object, such as “juice”, “remote control” and “book”, etc. The missing object type can refer to a general classification or semantic category of the object.

[0052] For example, the terminal can obtain the required object from the task instruction received by the agent and / or the program code generated by the agent, and then compare the required object with the existing objects in the current simulation scene to determine the missing object and the missing object information according to the comparison result. Alternatively, the missing object information can also be obtained based on the feedback of the agent after executing the program in the simulation scene. When the simulator executes each interactive operation according to the program code generated by the agent, if the program execution fails, for example, the action attempted to be executed by the agent fails to be successful in the simulator (e.g., the object cannot be grasped, the path is blocked, etc.), the terminal can obtain the missing object and the missing object information causing the task failure according to the breakpoint or error report of the program.

[0053] For example, please refer to Figure 3 ​For example, taking the action-type cloud native Agent as an example, the Agent receives a task instruction of an action type, the task instruction including an action name (watching TV) and an action description (taking the juice on the dining table to the coffee table in the living room, and relaxing by watching TV on the sofa in the living room). The Agent generates a program instruction according to the received task instruction, and sends the program instruction to the simulator, so that the simulator performs an interactive operation according to the program instruction.

[0054] Specifically, the Agent needs complete and explicit instructions when performing a task, and the program thereof must include all necessary steps. For example, when performing a common-sense task such as “watching TV”, in addition to the core action of “turning on the TV”, there may also be implicit steps such as “picking up the remote control” or “sitting on the sofa” and corresponding objects. In order to cover these common-sense steps, the Agent will explicitly list all the objects and interactions involved when generating the program, and if the necessary objects are missing in the simulated scene, the agent will not be able to successfully complete the task according to the programmed description.

[0055] The terminal of the embodiment of the present application can extract all objects from the action name, the action description, and the program instruction generated by the Agent, and identify the missing objects by comparing the complete list of objects required by the task with the list of objects already existing in the current simulated scene.

[0056] In step 204, a target reference object type and a target relative spatial relationship that match the missing object type are determined based on a target matching rule, and a target reference object instance corresponding to the target reference object type is obtained from a current object instance of the current simulated scene.

[0057] The target matching rule includes a plurality of object types and relative spatial relationships between different object types. The target matching rule can be a plurality of object types and reasonable or typical relative spatial relationships between different object types that are pre-stored by the terminal and conform to the semantics of the scene. The relative spatial relationship can be used to represent the association of the geometric poses of different objects represented by different object types in space.

[0058] For example, the target matching rule can be a semantic mapping table, which can include an object type of “remote control”, an object type of “coffee table”, and a relative spatial relationship of “remote control placed above coffee table”. For another example, an object type of “floor lamp”, an object type of “sofa”, and a relative spatial relationship of “floor lamp placed on the side of the sofa”.

[0059] The current object instance refers to an object entity that already exists in the current simulated scene.

[0060] Exemplarily, for the spatial relationship indicating the placement or attachment in the relative spatial relationship, the terminal can pre-record the object type (such as a desktop, a floor, or a wall) that the object type (such as a television, a sofa, a remote controller, or a coffee machine) can be placed or attached to, i.e., the object type provides a support surface for the object, to form a target matching rule, to ensure that the object conforms to the scene semantics. After determining the missing object information, the terminal determines the target reference object type and the target relative spatial relationship that match the missing object type based on the target matching rule. After determining the target reference object type, the terminal further searches and identifies the target reference object instance corresponding to the target reference object type from the current object instances in the current simulated scene.

[0061] In step 206, the target pose information of the missing object is determined based on the reference pose information of the target reference object instance in the current simulated scene and the target relative spatial relationship.

[0062] The reference pose information refers to the spatial state of the target reference object instance that has been obtained in the simulated scene, which can include reference position information such as three-dimensional coordinates, and can also include reference attitude information such as a quaternion or a rotation matrix representing three-dimensional rotation information. The target pose information refers to the expected spatial state of the missing object to be completed in the current simulated scene, which is used to complete the missing object in the current simulated scene.

[0063] Exemplarily, the terminal uses the known pose of the target reference object instance in combination with the determined target relative spatial relationship, such as “placed above”, “beside”, “attached to the side”, etc., to identify the relative position and direction that the missing object should occupy in space, to ensure that the missing object can be placed correctly and reasonably relative to the reference object instance according to the expected relative spatial relationship that is reasonable and conforms to the scene semantics, so as to form a coordinated and consistent scene layout.

[0064] In the object completion method, the terminal obtains missing object information of a missing object in the current simulation scene, the missing object information including a missing object type of the missing object, determines a target reference object type and a target relative spatial relationship matched with the missing object type based on a target matching rule, and obtains a target reference object instance corresponding to the target reference object type from current object instances of the current simulation scene, and determines target pose information of the missing object based on reference pose information of the target reference object instance in the current simulation scene and the target relative spatial relationship. The embodiments of the application identify a missing object required by a task and not contained in a current scene, filter specific scene elements and reasonable spatial relationships most suitable for bearing or associating the missing object by using preset knowledge rules and semantic association, convert the semantic matching into accurate target pose information, complete the missing object in the simulation environment, and improve the integrity of the simulation environment. Meanwhile, the target matching rule makes the expected pose of the missing object comply with the scene semantic logic, significantly improves the executability and adaptability in the cloud native agent scene, and avoids the situation that the agent fails to execute the task or the task logic is interrupted due to lack of necessary objects or unreasonable placement positions.

[0065] In actual application, when determining the target pose information of the missing object, the terminal can find a placement scheme of the missing object in the current simulation scene from the perspective of geometric coordination or physical avoidance of conflict with existing elements in the scene, so as to achieve a real and natural effect, in addition to ensuring that the target pose information and the reference pose information comply with the target relative spatial relationship.

[0066] In one exemplary embodiment, as shown in FIG. 4, step 206 includes steps 402 to 404. Wherein: Figure 4

[0067] Step 402, obtaining a preset target optimization model, the target optimization model including at least one of a spatial cost model and a collision cost model.

[0068] The spatial cost model is used to calculate the degree of spatial incoordination of the missing object and the target reference object instance. The collision cost model is used to indicate the degree of physical overlap of the missing object and the current object instance.

[0069] The target optimization model of the embodiments of the application can be a quantitative tool for determining the placement quality of the missing object at a given pose, and the core target is to find a best placement scheme for the missing object that meets spatial coordination and conforms to physical reality.

[0070] ​Exemplarily, the spatial cost model is used to calculate a degree of inconsistency in space between the missing object and the target reference object instance. The degree of inconsistency can include a degree of deviation in position, a degree of misalignment in direction, or a degree of exceeding a reasonable boundary of the reference object instance of the missing object relative to the reference object instance. For example, the model can quantify the deviation in position, the misalignment in direction, or whether the missing object exceeds the reasonable placement boundary of the reference object instance. The purpose is to ensure that the missing object and the reference object maintain visual consistency and geometric reasonableness, so that the completed scene looks natural.

[0071] Exemplarily, the collision cost model is used to calculate a degree of physical overlap between the missing object and other existing object instances in the current simulation scene. The model detects whether there is intersection or penetration between the missing object and the geometry of other scene elements, and gives corresponding penalties according to the degree of overlap. The core purpose is to avoid unreasonable mutual penetration between objects at the physical level, so as to ensure the physical authenticity of the simulation scene. The target optimization model and the various cost models inside it can be constructed according to pre-set rules, or trained from a large amount of scene data by a learning method.

[0072] At step 404, the reference pose information is input into the target optimization model to obtain the pose information of the missing object corresponding to the minimum value of the target optimization model under the constraint condition of satisfying the target relative spatial relationship, so as to obtain the target pose information.

[0073] The target optimization model can include a constraint condition, which can be determined by the target relative spatial relationship and used to limit the effective range of the pose of the missing object. For example, if the target relative spatial relationship is "placed above", the constraint condition can limit the distance between the bottom of the missing object and the upper surface of the reference object instance to be within a certain range; if the target relative spatial relationship is "next to", the constraint condition can limit the missing object and the reference object instance to maintain a reasonable non-contact distance.

[0074] Exemplarily, the terminal calls the pre-constructed target optimization model, inputs the reference pose information and the target relative spatial relationship into the target optimization model, calculates the optional or candidate pose range of the missing object under the constraint condition of satisfying the target relative spatial relationship through the target optimization model, and finds the pose information of the missing object corresponding to the minimum value of the target optimization model in the optional or candidate pose range, so as to obtain the target pose information. The minimum value of the target optimization model can be the minimum value of the spatial cost model, the minimum value of the collision cost model, or the sum of the minimum values of the spatial cost model and the collision cost model.

[0075] It can be understood that, in a possible implementation, in the case where the target optimization model comprises a spatial cost model, the terminal inputs the reference pose information into the spatial cost model to obtain the pose information of the missing object corresponding to the minimum model value of the target optimization model under the constraint condition of meeting the target relative spatial relationship.

[0076] It can be understood that, in another possible implementation, in the case where the target optimization model comprises a collision cost model, the pose information of the missing object corresponding to the minimum model value of the target optimization model under the constraint condition of meeting the target relative spatial relationship is obtained based on the reference pose information and the adjacent pose information to obtain the target pose information.

[0077] The adjacent pose information is the pose information of an adjacent object instance corresponding to the target reference object instance, and is used in combination with the pose information of the missing object to determine the model value of the collision cost model.

[0078] For example, the terminal inputs the reference pose information and the adjacent pose information into the collision cost model to obtain the pose information of the missing object corresponding to the minimum model value of the target optimization model under the constraint condition of meeting the target relative spatial relationship to obtain the target pose information.

[0079] In this embodiment, by obtaining a pre-set target optimization model comprising at least one of a spatial cost model and a collision cost model, the reference pose information is input into the target optimization model to obtain the pose information of the missing object corresponding to the minimum model value of the target optimization model under the constraint condition of meeting the target relative spatial relationship to obtain the target pose information, which can ensure that the missing object is physically and geometrically arranged in a coordinated and reasonable manner and does not conflict with the existing elements in the scene, thereby achieving a real and natural effect.

[0080] In actual applications, there can be multiple object types matched with one object type in the target matching rule, and therefore, the matched object types can be filtered when the target reference object type is determined.

[0081] In an exemplary embodiment, as shown in FIG. 5, the process of determining the target reference object type and the target relative spatial relationship in step 204 comprises steps 502 to 506. Figure 5

[0082] Step 502: determining multiple candidate reference object types matched with the missing object type based on the target matching rule.

[0083] ​In the target matching rule of the embodiment of the present application, one object type can be matched with multiple other object types, that is, in actual application, one missing object type can be matched with multiple candidate reference object types in the target matching rule.

[0084] For example, the terminal acquires a missing object type "fruit juice", queries the preset target matching rule, and acquires all candidate reference object types that are semantically reasonably matched with the missing object type. For "fruit juice", the rule can return a relative spatial relationship of placing fruit juice on it, and the candidate reference object types can be "dining table type", "coffee table type", "television cabinet type", and the like.

[0085] Step 504: selecting a target reference object type that is adapted to the associated object information from the multiple candidate reference object types.

[0086] The missing object information further includes associated object information of the missing object, and the associated object information is used to indicate an expected dependent object type of the missing object in the task context.

[0087] For example, the terminal acquires the associated object information of the missing object from the task instruction received from the agent, for example, "dining table" in "take the fruit juice of the dining table to the coffee table in the living room" as the associated object information of the missing object. The terminal determines "dining table type" as the target reference object type from the candidate reference object types "dining table type", "coffee table type", "television cabinet type".

[0088] Step 506: determining a target relative spatial relationship corresponding to the target reference object type and the missing object type based on the target matching rule.

[0089] In the embodiment, the candidate reference object types are screened through the associated object information of the missing object, which can improve the rationality of the placement position of the missing object and significantly improve the executability and adaptability in the multi-agent scene.

[0090] In actual application, there can be multiple reference object instances corresponding to the target reference object type in the current object instances of the current simulation scene, and therefore, the multiple reference object instances can be screened when the target reference object instance is determined.

[0091] In one exemplary embodiment, as shown in Figure 6 The process of determining the target reference object instance in step 204 includes steps 602 to 604. In which:

[0092] Step 602: acquiring multiple candidate reference object instances corresponding to the target reference object type from the current object instances of the current simulation scene.

[0093] For example, the terminal identifies the "television" as the missing object according to the task instruction "relax by watching TV on the sofa in the living room" and the current simulation scene, and the target reference object type can be "TV cabinet", "mobile support", or "wall surface". The terminal obtains multiple candidate reference object instances from the current object instances of the current simulation scene, including the TV cabinet in the living room, the TV cabinet in the room, the wall surface in the living room, the wall surface in the room, and the mobile support in the room.

[0094] Step 604: selecting a target reference object instance that matches the associated space information from the multiple candidate reference object instances.

[0095] The missing object information further includes associated space information of the missing object, and the associated space information is used to represent the space region to which the missing object is expected to belong in the task context.

[0096] For example, the terminal obtains the associated space information of the missing object "television" as "living room" according to the task instruction "relax by watching TV on the sofa in the living room", and selects "TV cabinet in the living room" and "wall surface in the living room" as the target reference object instance based on the associated space information.

[0097] In this embodiment, the candidate reference object instances are filtered through the associated space information of the missing object, which can improve the rationality of the placement position of the missing object and significantly improve the executability and adaptability in the multi-agent scene.

[0098] In actual application, when there are multiple target reference object instances, the optimal placement scheme of the missing object can be determined for each target reference object instance, and the target pose information can be determined from the multiple optimal placement schemes of the multiple target reference object instances.

[0099] In an exemplary embodiment, as shown in Figure 7 Step 404 includes steps 702 to 704. Wherein:

[0100] Step 702: for each target reference object instance, based on the reference pose information of the target reference object instance, obtaining the pose information of the missing object corresponding to the minimum model value of the target optimization model under the constraint condition of satisfying the target relative spatial relationship, to obtain multiple candidate pose information corresponding to the multiple target reference object instances.

[0101] Step 704: comparing the model values of the target optimization model corresponding to the multiple candidate pose information, and obtaining the target pose information according to the comparison result.

[0102] Exemplarily, the terminal obtains, through target optimization of the target optimization model, for each target reference object instance, a placement scheme of the missing object corresponding to each target reference object instance that is optimal, i.e., pose information at which the model value of the target optimization model is the smallest, and takes the candidate pose information as candidate pose information. The terminal compares the candidate pose information corresponding to different target reference object instances, compares the model values of the target optimization models corresponding to each of the candidate pose information, and takes the candidate pose information with the smallest model value as the target pose information.

[0103] In this embodiment, when there are multiple target reference object instances, the optimal scheme corresponding to each target reference object instance can be determined first, and then the optimal schemes of different target reference object instances are compared to finally determine the target pose information of the missing object, so as to determine the optimal placement scheme based on physical constraints.

[0104] In one exemplary embodiment, an object completion method is provided, and the method comprises:

[0105] Step A1, the terminal obtains task instructions received by the Agents and program codes generated by the Agents, and extracts all objects from the task instructions and the program codes.

[0106] Step A2, the terminal compares the extracted objects with the existing objects in the current simulation scene to identify missing objects.

[0107] Step A3, the terminal determines the target reference object type and the target relative spatial relationship corresponding to the missing object based on a target matching rule, and selects a target reference object instance.

[0108] Step A4, the terminal inputs the reference pose information into the target optimization model for each target reference object instance to obtain, under the constraint condition of meeting the target relative spatial relationship, pose information of the missing object corresponding to the minimum model value of the target optimization model, so as to obtain at least one pose information corresponding to at least one target reference object instance.

[0109] For example, it is assumed that the missing object P, the target reference object instance S, and the target relative spatial relationship represent that the missing object P should be placed on the target reference object instance S, and the target reference object instance S usually provides a plane or surface, and the distance from any point in the scene to the surface can be represented as:

[0110] ;

[0111] Wherein, n is the unit normal vector of the support surface, x0 is a point on the support object plane, and x represents the position of any point in the scene.

[0112] The constraint condition satisfying the target relative spatial relationship can include that a distance of the missing object P to a surface of the target reference object instance S is within a preset distance range, where the distance of the missing object P to the surface of the target reference object instance S can be represented as:

[0113] ;

[0114] Where T+R·b represents the reference point position of the missing object P (which has a reference point b in the local coordinate system) in the scene coordinate; T is a translation vector, and R is a rotation matrix.

[0115] The target optimization model includes a spatial cost model and a collision cost model. The spatial cost model can include at least one of a first target function, a second target function, and a third target function. The first target function is used to calculate the deviation of the missing object from the reference object instance in position. The second target function is used to calculate the misalignment in direction. The third target function is used to calculate the exceeding of the reasonable boundary corresponding to the reference object instance. The collision cost model can include a fourth target function. The fourth target function is used to calculate the physical overlap of the missing object with other existing object instances in the current simulation scene.

[0116] In order to simultaneously consider the alignment of placement, the rationality of position, and the avoidance of collision (such as preventing the object from overlapping with other objects in the scene), taking the spatial cost model including the first target function, the second target function, and the third target function as an example, the target optimization model can include the following total cost function:

[0117] ;

[0118] Wherein, The total cost function value of the target optimization model is used as the model value of the target optimization model.

[0119] The first target function is a fitting alignment cost. When the first target function is approximately equal to 0, it represents that the missing object P is fitted on the target reference object instance S, that is, the bottom of the missing object P is almost coplanar with the support surface of the target reference object instance S.

[0120] The second target function is a direction alignment. It is used to match the main axis of the missing object P with the normal vector or tangent direction of the support surface of the target reference object instance S (such as the object should be placed on the desktop in parallel).

[0121] The fourth target function is a collision penalty, which penalizes the penetration with other scene objects.

[0122] As a third objective function, it represents a boundary constraint for placing the missing object P inside the support area (e.g. a table top, floor or wall surface), which can be achieved by detecting whether the projected position falls within a predefined polygonal area.

[0123] The terminal can find the optimal T and R according to the optimization or search strategy of the above-mentioned target optimization model under the constraint condition of meeting the target relative spatial relationship, that is, the pose information of the missing object corresponding to the minimum model value.

[0124] Step A5, the terminal determines the target pose information of the missing object according to the model value corresponding to at least one pose information.

[0125] For example, the terminal compares the model values corresponding to multiple pose information, and determines the pose information with the minimum model value as the target pose information.

[0126] Step A6, the simulator places the missing object in a reasonable position in the scene according to the target pose information to update the current simulation scene.

[0127] If multiple missing objects are identified in step A2 or the following step A7, the above steps A3 to A6 are sequentially performed for each missing object until the arrangement of multiple missing objects is completed.

[0128] Step A7, the simulator performs all necessary interactive operations in the program instructions, and if the program execution is interrupted and the number of iterations does not exceed the preset number of iterations, the missing object is checked.

[0129] If a missing object is found, the number of iterations is incremented by one, and steps A3 to A7 are repeated, so that the Agents execute the program instructions containing all necessary objects in the updated scene until the program execution is completed or the number of iterations exceeds the preset number of iterations, thereby effectively verifying the feasibility and correctness of the task code in the complete scene.

[0130] In this embodiment, by detecting and completing the missing objects in the simulation environment, the integrity of the scene can be guaranteed at the physical and semantic levels, providing a real and controllable interactive environment for task execution and evaluation. The object completion strategy based on physical collision detection and geometric constraints, combined with a series of optimization objective functions (such as alignment, collision detection, boundary constraint), can significantly improve the executability and adaptability in multi-agent scenes, avoiding the failure of Agents to execute tasks or the interruption of task logic due to the lack of necessary objects or unreasonable placement positions.

[0131] It should be understood that although each step in the flowchart involved in the embodiments described above is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.

[0132] Based on the same inventive concept, the embodiments of the present application also provide an object completion device for implementing the object completion method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more object completion device embodiments provided below can refer to the limitations of the object completion method described above, which will not be repeated here.

[0133] In one exemplary embodiment, as shown in Figure 8 An object completion device is provided, comprising: an acquisition module 802, a matching module 804 and a completion module 806, wherein:

[0134] The acquisition module 802 is configured to acquire missing object information of a missing object in a current simulation scene; the missing object information comprises a missing object type of the missing object.

[0135] The matching module 804 is configured to determine a target reference object type and a target relative spatial relationship matched with the missing object type based on a target matching rule, and acquire a target reference object instance corresponding to the target reference object type from a current object instance of the current simulation scene; the target matching rule comprises a plurality of object types and relative spatial relationships between different object types.

[0136] The completion module 806 is configured to determine target pose information of the missing object based on reference pose information of the target reference object instance in the current simulation scene and the target relative spatial relationship; the target pose information is used to complete the missing object in the current simulation scene.

[0137] In one embodiment, the completion module 806 comprises:

[0138] The acquisition unit is configured to acquire a preset target optimization model, the target optimization model comprising at least one of a spatial cost model and a collision cost model, the spatial cost model being configured to calculate a degree of inconsistency in space between the missing object and the target reference object instance, and the collision cost model being configured to calculate a degree of physical overlap between the missing object and the current object instance.

[0139] The determination unit is configured to input the reference pose information into the target optimization model to obtain pose information of the missing object corresponding to a minimum model value of the target optimization model under a constraint condition of satisfying the target relative spatial relationship, so as to obtain the target pose information.

[0140] In one of the embodiments, when performing the obtaining of the target pose information, the determination unit is configured to, in a case where the target optimization model comprises the collision cost model, acquire, based on the reference pose information and adjacent pose information, pose information of the missing object corresponding to a minimum model value of the target optimization model under a constraint condition of satisfying the target relative spatial relationship, so as to obtain the target pose information; wherein the adjacent pose information is pose information of an adjacent object instance corresponding to the target reference object instance; and the adjacent pose information is used to determine the model value of the collision cost model in combination with the pose information of the missing object.

[0141] In one of the embodiments, the target reference object instance is multiple; and when performing the obtaining of the target pose information, the determination unit is configured to, for each of the target reference object instances, acquire, based on the reference pose information of the target reference object instance, pose information of the missing object corresponding to a minimum model value of the target optimization model under a constraint condition of satisfying the target relative spatial relationship, so as to obtain multiple candidate pose information corresponding to the multiple target reference object instances; compare model values of the target optimization model corresponding to the multiple candidate pose information respectively, and obtain the target pose information according to a comparison result.

[0142] In one of the embodiments, the missing object information further comprises associated object information of the missing object; the associated object information is used to indicate a type of dependent object expected by the missing object in a task context; and the matching module 804, when performing the determining of the target reference object type and the target relative spatial relationship matched with the missing object type based on the target matching rule, is configured to: determine, based on the target matching rule, multiple candidate reference object types matched with the missing object type; select, from the multiple candidate reference object types, the target reference object type adapted to the associated object information; and determine, based on the target matching rule, the target relative spatial relationship corresponding to the target reference object type and the missing object type.

[0143] In one of the embodiments, the missing object information further comprises associated spatial information of the missing object; the associated spatial information is used to represent a space region to which the missing object is expected to belong in a task context; the matching module 804, when performing the operation of obtaining the target reference object instance corresponding to the target reference object type from the current object instances of the current simulation scene, is configured to: obtain a plurality of candidate reference object instances corresponding to the target reference object type from the current object instances of the current simulation scene; and select the target reference object instance that is adapted to the associated spatial information from the plurality of candidate reference object instances.

[0144] The modules in the object completion apparatus described above can be implemented in whole or in part by software, hardware, and combinations thereof. The modules described above can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform the operations corresponding to the modules.

[0145] In one exemplary embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 9 The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store data related to target matching rules. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement an object completion method.

[0146] Those skilled in the art can understand that Figure 9 The structure shown in the above

[0147] In an example embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the object completion method provided by the embodiments of the present application when executing the computer program.

[0148] In an example embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program implementing the steps of the object completion method provided by the embodiments of the present application when executed by a processor.

[0149] In an example embodiment, a computer program product is provided, comprising a computer program, and the computer program implementing the steps of the object completion method provided by the embodiments of the present application when executed by a processor.

[0150] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0151] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0152] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0153] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. An object completion method, characterized by, The method comprises: obtaining missing object information of a missing object in a current simulation scene; the missing object information comprises a missing object type of the missing object; determining a target reference object type and a target relative spatial relationship matched with the missing object type based on a target matching rule, and obtaining a target reference object instance corresponding to the target reference object type from current object instances of the current simulation scene; the target matching rule comprises a plurality of object types and relative spatial relationships between different object types; determining target pose information of the missing object based on reference pose information of the target reference object instance in the current simulation scene and the target relative spatial relationship; the target pose information is used for completing the missing object in the current simulation scene; wherein, the target pose information of the missing object is determined based on the reference pose information of the reference object instance in the current simulation scene and the target relative spatial relationship, comprising: obtaining a pre-set target optimization model, the target optimization model comprising at least one of a spatial cost model and a collision cost model, the spatial cost model being used for calculating a degree of incoordination of the missing object and the target reference object instance in space, and the collision cost model being used for calculating a degree of physical overlap of the missing object and the current object instance; inputting the reference pose information into the target optimization model to obtain pose information of the missing object corresponding to a minimum model value of the target optimization model under a constraint condition of meeting the target relative spatial relationship, so as to obtain the target pose information.

2. The method of claim 1, wherein, The reference pose information comprises reference position information and reference attitude information; the reference position information comprises three-dimensional coordinates of the target reference object instance in the current simulation scene; and the reference attitude information comprises three-dimensional rotation information of the target reference object instance in the current simulation scene.

3. The method of claim 1, wherein, The reference pose information is inputted into the target optimization model to obtain pose information of the missing object corresponding to a minimum model value of the target optimization model under a constraint condition of meeting the target relative spatial relationship, comprising: in a case where the target optimization model comprises the collision cost model, obtaining, based on the reference pose information and adjacent pose information, pose information of the missing object corresponding to a minimum model value of the target optimization model under a constraint condition of meeting the target relative spatial relationship, so as to obtain the target pose information; wherein, the adjacent pose information is pose information of an adjacent object instance corresponding to the target reference object instance; and the adjacent pose information is used for determining the model value of the collision cost model in combination with the pose information of the missing object.

4. The method of claim 1, wherein, The target reference object instance has a plurality of target reference object instances; and the reference pose information is inputted into the target optimization model to obtain pose information of the missing object corresponding to a minimum model value of the target optimization model under a constraint condition of meeting the target relative spatial relationship, comprising: For each of the target reference object instance, based on the reference pose information of the target reference object instance, obtain the pose information of the missing object corresponding to the minimum model value of the target optimization model under the constraint condition of meeting the target relative spatial relationship, to obtain a plurality of candidate pose information corresponding to a plurality of target reference object instances; Compare the model values of the target optimization model corresponding to a plurality of candidate pose information respectively, and obtain the target pose information according to the comparison result.

5. The method according to any one of claims 1 to 4, characterized in that, The missing object information further includes associated object information of the missing object; the associated object information is used to indicate the expected dependent object type of the missing object in the task context; The target matching rule based on the target matching rule determines the target reference object type and the target relative spatial relationship matched with the missing object type, comprising: Based on the target matching rule, determine a plurality of candidate reference object types matched with the missing object type; Select the target reference object type from a plurality of candidate reference object types that matches the associated object information; Based on the target matching rule, determine the target relative spatial relationship corresponding to the target reference object type and the missing object type.

6. The method according to any one of claims 1 to 4, characterized in that, The missing object information further includes associated spatial information of the missing object; the associated spatial information is used to represent the expected space region to which the missing object belongs in the task context; the target reference object instance corresponding to the target reference object type is obtained from the current object instance of the current simulation scene, comprising: Obtain a plurality of candidate reference object instances corresponding to the target reference object type from the current object instance of the current simulation scene; Select the target reference object instance from a plurality of candidate reference object instances that matches the associated spatial information.

7. An object completion apparatus characterized by comprising: The device comprises: An acquisition module is configured to acquire missing object information of a missing object in a current simulation scene; the missing object information includes a missing object type of the missing object; A matching module is configured to determine a target reference object type and a target relative spatial relationship matched with the missing object type based on a target matching rule, and to obtain a target reference object instance corresponding to the target reference object type from a current object instance of the current simulation scene; the target matching rule includes a plurality of object types and relative spatial relationships between different object types; A completion module is configured to determine target pose information of the missing object based on reference pose information of the target reference object instance in the current simulation scene and the target relative spatial relationship; the target pose information is used to complete the missing object in the current simulation scene; The target matching rule based on the target matching rule determines the target reference object type and the target relative spatial relationship matched with the missing object type, comprising: Based on the target matching rule, determine a plurality of candidate reference object types matched with the missing object type; Select the target reference object type from a plurality of candidate reference object types that matches the associated object information; Based on the target matching rule, determine the target relative spatial relationship corresponding to the target reference object type and the missing object type. obtain a preset target optimization model, the target optimization model comprising at least one of a space cost model and a collision cost model, the space cost model being used to calculate a degree of incoordination of the missing object and the target reference object instance in space, and the collision cost model being used to calculate a degree of physical overlap of the missing object and the current object instance; input the reference pose information into the target optimization model to obtain pose information of the missing object corresponding to a minimum value of the target optimization model under a constraint condition of satisfying the target relative spatial relationship, so as to obtain the target pose information.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor implements the steps of the method of any one of claims 1 to 6 when executing the computer program.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6. The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

Images