Object replacement method and electronic equipment

Abstract

The invention provides an object replacement method and electronic equipment, and the method comprises the steps: obtaining an original operation scene and an object model of a new object; determining an original contact point track sequence according to the actuator model, the pose data of the actuator, the action data of the actuator and the object information of the original object; determining geometric transformation parameters of the newly added object; determining visual parameters of the newly added object according to original image data corresponding to the original operation scene; and replacing the original object in the original operation scene with the newly-added object based on the geometric transformation parameter and the visual parameter of the newly-added object. According to the method and the device, the original object in the original operation scene can be automatically replaced by the newly-added object, and the physical rationality and the appearance authenticity of interaction with the real environment during object replacement can also be ensured, so that the replaced newly-added object can easily adapt to the motion mode of the actuator in the original operation scene; therefore, more real simulation operation demonstration data can be generated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data enhancement, in particular to an object replacement method and an electronic device. BACKGROUND

[0002] With the development of robot embodiment intelligence, how to efficiently generate diversified operation scene data with physical rationality and visual authenticity has become a key challenge to promote the generalization ability of robots. Among them, three-dimensional Gaussian splatting (3D Gaussian Splatting, 3DGS for short) gradually becomes an important technical path for constructing realistic virtual environments due to its explicit representation characteristics, high-quality photo-level rendering capability and lightweight modeling process. Specifically, a real scene is reconstructed into a Gaussian point cloud, and an object replacement is realized by combining simulation assets, thereby generating a new interactive scene.

[0003] The object replacement method in the prior art usually relies on manual alignment to embed the mesh model of the target replacement object into the original scene and manually adjust the appearance consistency, and finally complete the replacement and rendering of the grasping object (such as a cup or a box).

[0004] However, the above prior art mainly faces rigid objects without motion constraints, and it is difficult to apply to hinge objects with joint motion characteristics. When the object is a hinge object, problems such as contact point misplacement, motion trajectory conflict, action failure or penetration phenomenon are prone to occur. SUMMARY

[0005] The present application aims to solve the problems of the prior art, such as the easy occurrence of contact point misplacement, motion trajectory conflict, action failure or penetration phenomenon, by providing an object replacement method and an electronic device.

[0006] To achieve the above-mentioned purpose, the technical solutions adopted by the embodiments of the present application are as follows: In a first aspect, an embodiment of the present application provides an object replacement method, which comprises: obtaining an original operation scene and an object model of a new object, the original operation scene comprising object information of an original object, an actuator model, pose data of the actuator and action data of the actuator, and the object model of the new object comprising object information of the new object; determining an original contact point trajectory sequence of the actuator and the original object according to the actuator model, the pose data of the actuator, the action data of the actuator and the object information of the original object; determining geometric transformation parameters of the new object according to the original contact point trajectory sequence, the geometric transformation parameters comprising a scaling ratio, initial motion parameters and position offset information; determine the visual parameter of the new object according to original image data corresponding to the original operation scene; replace the original object in the original operation scene with the new object based on the geometric transformation parameter and the visual parameter of the new object.

[0007] In a second aspect, another embodiment of the present application provides an object replacement device, and the device comprises: An acquisition module is configured to acquire an original operation scene and an object model of a new object, wherein the original operation scene comprises object information of an original object, an executor model, pose data of an executor, and action data of the executor, and the object model of the new object comprises object information of the new object. An original contact point determination module is configured to determine a sequence of original contact point trajectories of the executor and the original object according to the executor model, the pose data of the executor, the action data of the executor, and the object information of the original object. A geometric transformation determination module is configured to determine a geometric transformation parameter of the new object according to the sequence of original contact point trajectories, wherein the geometric transformation parameter comprises a scaling ratio, initial motion parameter, and position offset information. A visual determination module is configured to determine a visual parameter of the new object according to original image data corresponding to the original operation scene. A replacement module is configured to replace the original object in the original operation scene with the new object based on the geometric transformation parameter and the visual parameter of the new object.

[0008] In a third aspect, another embodiment of the present application provides an electronic device, which comprises a processor, a storage medium, and a bus. The storage medium stores machine readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium through the bus. The processor executes the machine readable instructions to perform the steps of any method in the first aspect.

[0009] In a fourth aspect, another embodiment of the present application provides a computer readable storage medium, which stores a computer program. When the computer program is run by a processor, the steps of any method in the first aspect are performed.

[0010] The beneficial effects of the present application are: by obtaining the original operation scene and the object model of the new object, and determining the original contact point trajectory sequence of the executor and the original object according to the executor model, the pose data of the executor and the object information of the original object, the geometric transformation parameters of the new object can be determined according to the original contact point trajectory sequence, and the visual parameters of the new object can be determined according to the original image data corresponding to the original operation scene, so that the original object in the original operation scene is replaced by the new object based on the geometric transformation parameters and the visual parameters of the new object, the original object in the original operation scene can be automatically replaced by the new object, and at the same time, the physical rationality and the appearance authenticity in the interaction with the real environment during the object replacement are guaranteed, so that the replaced new object can easily adapt to the motion mode of the executor in the original operation scene, thereby generating more realistic simulation operation demonstration data.

[0011] Moreover, the geometric transformation parameters of the new object are determined through the original contact point trajectory sequence, so as to ensure that the new object can reproduce reasonable contact behavior under the same executor action, solve the problems of mold penetration, derailment and invalid operation, realize "dynamic matching", and on this basis, the visual parameters of the new object are determined through the original image data corresponding to the original operation scene, realize high-fidelity visual fusion, achieve "realistic vision", and significantly improve the physical authenticity and visual fidelity of the virtual operation scene generation, which is especially suitable for robot operation skill learning and simulation training of hinge objects. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0013] Figure 1 A flowchart of an object replacement method provided by an embodiment of the present application; Figure 2 A schematic diagram of an object replacement method provided by an embodiment of the present application; Figure 3 A flowchart of determining the original contact point trajectory sequence of the executor and the original object in the object replacement method provided by an embodiment of the present application; Figure 4 A flowchart of determining the geometric transformation parameters of the new object in the object replacement method provided by an embodiment of the present application; Figure 5 A flowchart of determining the contact point information of the new object in the object replacement method provided by an embodiment of the present application; Figure 6 A flowchart of a process of obtaining the geometric transformation parameter of the new object in the object replacement method provided by the embodiments of the present application; Figure 7 A flowchart of replacing the original object in the original operation scene with the new object in the object replacement method provided by the embodiments of the present application; Figure 8 Another schematic diagram of the object replacement method provided by the embodiments of the present application; Figure 9 Another schematic diagram of the object replacement method provided by the embodiments of the present application; Figure 10 A schematic diagram of an object replacement device provided by the embodiments of the present application; Figure 11 A schematic diagram of an electronic device structure provided by the embodiments of the present application. DETAILED DESCRIPTION

[0014] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of description and illustration, and are not used to limit the protection scope of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportions. The flowcharts in the present application show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or removed from the flowcharts by those skilled in the art under the guidance of the content of the present application.

[0015] In addition, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0016] It should be noted that the term “comprising” will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0017] The object replacement method in the prior art usually relies on manual alignment, embeds a mesh model of a target replacement object into an original scene, and manually adjusts appearance consistency, finally completing replacement and rendering of a grasping object (such as a cup or a box).

[0018] However, the above prior art mainly faces rigid objects without motion constraints, and is difficult to be applied to hinge objects with joint motion characteristics. When the object is a hinge object, problems such as contact point misplacement, motion trajectory conflict, action failure, or mode penetration may exist.

[0019] Meanwhile, the physical rationality of the original operation behavior cannot be maintained after object replacement, such as contact point misplacement and motion trajectory conflict, which is unreasonable. That is, the existing method cannot guarantee "moving correctly" and is difficult to achieve "seeing truly" when processing a hinge object operation scene, which seriously limits its application in robot fine operation data generation.

[0020] Based on the above problems, the object replacement method provided in the embodiments of the present application is proposed. The object replacement method is used to obtain an original operation scene and an object model of a new object, and determine an original contact point trajectory sequence of an actuator and an original object according to an actuator model, pose data of the actuator, and object information of the original object. The object replacement method is used to determine geometric transformation parameters of the new object according to the original contact point trajectory sequence, and determine visual parameters of the new object according to original image data corresponding to the original operation scene. The object replacement method is used to replace the original object in the original operation scene with the new object based on the geometric transformation parameters and the visual parameters of the new object. The object replacement method can automatically replace the original object in the original operation scene with the new object, and can also guarantee the physical rationality and appearance authenticity of interaction with a real environment during object replacement. The new object after replacement can easily adapt to a motion mode of the actuator in the original operation scene, so that more realistic simulation operation demonstration data can be generated.

[0021] It can be understood that the object replacement method provided in the embodiments of the present application can be applied to any scene in which an object in an original operation scene needs to be replaced. Specifically, the object replacement method provided in the embodiments of the present application can replace an object in a three-dimensional Gaussian splatting (3D Gaussian Splatting) scene for a robot hinge object operation task.

[0022] Exemplarily, the original operation scene can be a 3D Gaussian scene, and the object in the original operation scene can be an object including a hinge joint or a sliding joint.

[0023] The object replacement method provided in the embodiments of the present application is described in detail below in combination with multiple embodiments.

[0024] Figure 1A flowchart of an object replacement method provided by an embodiment of the present application is shown in FIG. 1. The method can be executed by any electronic device with processing capability. The method includes the following steps. Figure 1 S101, obtaining an original operation scene and an object model of a new object.

[0025] Optionally, the original operation scene and the object model of the new object are obtained.

[0026] The original operation scene includes object information of an original object, an executor model, pose data of the executor, and action data of the executor. The object model of the new object includes object information of the new object. Specifically, the original operation scene can be a real Gaussian scene in which the original object is operated by the executor in an environment.

[0027] The original object is an object including a hinge joint or a sliding joint in the original operation scene. The original object includes a movable part and an immovable part. The new object is an object including a hinge joint or a sliding joint.

[0028] Specifically, the object information of the original object includes the geometric morphology and the surface color of the original object. The geometric morphology can be presented in the form of a point cloud. According to different types of point clouds, the geometric morphology can include Gaussian point clouds of the movable part and Gaussian point clouds of the immovable part.

[0029] Specifically, the executor model refers to the geometric morphology of a robot arm and an executor at the end of the robot arm. The pose data of the executor includes the pose of the robot arm and the executor at the end of the robot arm at multiple time steps. The scene information includes the global light distribution in the original operation scene. The action data of the executor refers to data related to the action state of the executor at each time step. For example, taking the executor as a gripper, the action data can include closing at a certain time step and opening at a certain time step.

[0030] Optionally, the original operation scene further includes layout distribution information of the original object and other objects, and scene information.

[0031] Specifically, the object information of the new object includes geometric mesh data of the new object, initial state parameters, material and texture information, and kinematic structure information. The geometric mesh data includes face index and surface normal vector. The initial state parameters include default opening and closing degree. The material and texture information includes a basic color map. The kinematic structure information includes each joint in the new object, joint types of each joint, joint axis direction, and motion degrees of freedom. The joint types include hinge joints and sliding joints.

[0032] For example,​Figure 2 FIG. 1 shows a schematic diagram of an object replacement method according to an embodiment of the present application. Figure 2 As shown in FIG. 1, in the original operation scene, the box object is the original object, and the original operation scene can indicate a dynamic process in which an end effector of a robot arm opens a side cover of the box object.

[0033] S102, determining a sequence of original contact points between the end effector and the original object according to the end effector model, the pose data of the end effector, the action data of the end effector, and the object information of the original object.

[0034] Optionally, after obtaining the original operation scene, the sequence of original contact points between the end effector and the original object can be determined according to the end effector model, the pose data of the end effector, the action data of the end effector, and the object information of the original object.

[0035] For example, each time step at which the action data of the end effector is closed can be found, and at each time step, the contact point between the end effector and the original object can be calculated according to the end effector model, the pose data of the end effector, and the object information of the original object, thereby obtaining the sequence of original contact points.

[0036] The sequence of original contact points between the end effector and the original object refers to a sequence of three-dimensional coordinate points representing the actual interaction path between the end effector of the robot arm and the original object in the real operation process.

[0037] S103, determining the geometric transformation parameters of the new object according to the sequence of original contact points.

[0038] Optionally, after obtaining the sequence of original contact points, the geometric transformation parameters of the new object can be determined according to the sequence of original contact points.

[0039] In one example, the sequence of original contact points can be taken as a target contact point trajectory sequence between the new object and the end effector, and the geometric transformation parameters of the new object can be calculated according to the target contact point trajectory sequence.

[0040] In another example, the object information of the original object, the sequence of original contact points, and the object information of the new object can be input into a pre-trained geometric transformation parameter prediction model, and the geometric transformation parameters of the new object can be inferred from the geometric transformation parameter prediction model. The geometric transformation parameter prediction model can be implemented based on a Point-to-Graph Matching Network, or can also be implemented based on a conditional generative adversarial network (cGAN).

[0041] The geometric transformation parameters include a scaling ratio, initial motion parameters, and position offset information. Specifically, the scaling ratio refers to the change multiple of the overall or partial size of the new object, the initial motion parameters refer to the orientation adjustment of the new object relative to the original coordinate system, and the position offset information refers to the translation amount of the new object in space. For a revolute joint, the initial motion parameters are the rotation angle of the movable part, and for a sliding joint, the initial motion parameters are the translation distance of the movable part.

[0042] S104, determining visual parameters of the new object according to original image data corresponding to the original operation scene.

[0043] Optionally, original image data corresponding to the original operation scene can be obtained, and the visual parameters of the new object can be determined according to the original image data.

[0044] The original image data corresponding to the original operation scene refers to the image data used when the original operation scene is constructed. The original image data includes one or more real indoor operation scene photos taken.

[0045] The visual parameters refer to surface material parameters of the new object, including material properties, pixel colors, textures, and mapping information.

[0046] For example, the original image data can be input into a pre-trained visual parameter prediction model to obtain the visual parameters of the new object.

[0047] S105, replacing the original object in the original operation scene with the new object based on the geometric transformation parameters and the visual parameters of the new object.

[0048] Optionally, after obtaining the geometric transformation parameters and the visual parameters of the new object, the new object can be geometrically transformed according to the geometric transformation parameters, and the visual effect of the new object can be adjusted according to the visual parameters, and then the original object in the original operation scene can be replaced with the new object.

[0049] In the embodiment, by obtaining the original operation scene and the object model of the new object, and determining the original contact point trajectory sequence of the executor and the original object according to the executor model, the pose data of the executor and the object information of the original object, the geometric transformation parameters of the new object can be determined according to the original contact point trajectory sequence, and the visual parameters of the new object can be determined according to the original image data corresponding to the original operation scene, so that the original object in the original operation scene is replaced by the new object based on the geometric transformation parameters and the visual parameters of the new object. The original object in the original operation scene can be automatically replaced by the new object, and meanwhile, the physical rationality and the appearance authenticity in the interaction with the real environment during the object replacement can be ensured, so that the new object after replacement can easily adapt to the motion mode of the executor in the original operation scene, thereby generating more real simulation operation demonstration data.

[0050] Moreover, the geometric transformation parameters of the new object are determined through the original contact point trajectory sequence, so as to ensure that the new object can reproduce reasonable contact behavior under the same executor action, solve the problems such as mold penetration, derailment and invalid operation, realize the “dynamic pair”, and on this basis, the visual parameters of the new object are determined through the original image data corresponding to the original operation scene, realize high-fidelity visual fusion, achieve “real view”, and significantly improve the physical authenticity and visual fidelity of the virtual operation scene generation, which is especially suitable for the operation skill learning and simulation training of robots on hinge objects.

[0051] In a possible implementation manner, Figure 3 A flowchart for determining the original contact point trajectory sequence of the executor and the original object in the object replacement method provided by the embodiments of the present application is shown in Figure 3 The determination of the original contact point trajectory sequence of the executor and the original object in S102 according to the executor model, the pose data of the executor, the motion data of the executor and the object information of the original object includes: S301, determining a plurality of candidate contact points of the executor and the original object according to the executor model, the pose data of the executor and the object information of the original object.

[0052] The plurality of candidate contact points of the executor and the original object can include one or more candidate contact points of the executor and the original object at a plurality of time steps.

[0053] Optionally, at each time step, a surface point set of the executor model can be determined according to the executor model and the pose data of the executor, and a spatial nearest point between the surface point set of the executor model and a surface where a Gaussian point cloud of a movable part in the object information of the original object is located can be determined, so as to obtain the plurality of candidate contact points of the executor and the original object.

[0054] In an example, taking one time step as an example, the executor model can be adjusted according to the pose data at the time step to obtain a surface point set of the executor model, and a spatial nearest point between the surface point set of the executor model and a surface formed by the Gaussian point cloud of the movable part of the original object is detected through a collision detection algorithm, and the spatial nearest point is taken as a candidate contact point at the time step. The collision detection algorithm can be a distance field-based collision detection algorithm.

[0055] In another example, taking one time step as an example, the executor model can be adjusted according to the pose data at the time step to obtain a surface point set of the executor model, and a surface point in the surface point set of the executor model closest to the original object is taken as a candidate contact point at the time step.

[0056] In another example, taking one time step as an example, the minimum Euclidean distance between each surface point in the surface point set of the executor model and the Gaussian point cloud of the movable part of the original object can also be calculated, and if the minimum Euclidean distance is less than a preset distance threshold, the corresponding surface point is taken as a candidate contact point at the time step.

[0057] S302, determining an original contact point trajectory sequence of the executor and the original object according to each candidate contact point and the action data of the executor.

[0058] Optionally, after obtaining each candidate contact point, each candidate contact point can be screened according to the action data of the executor to obtain the original contact point of the executor and the original object at the plurality of time steps, so as to filter and smooth the original contact point of the executor and the original object at the plurality of time steps to obtain the original contact point trajectory sequence.

[0059] In an example, taking one candidate contact point as an example, the time step corresponding to the candidate contact point can be determined, and the action state of the executor at the time step is determined from the action data of the executor, and if the action state of the executor is closed, the candidate contact point can be taken as the original contact point at the time step.

[0060] In another example, taking one candidate contact point as an example, the time step corresponding to the candidate contact point can be determined, and the action state of the executor at the time step is determined from the action data of the executor, and at the same time, the relative action speed of the executor at the time step is determined, and if the action state of the executor is closed and the relative action speed of the executor tends to 0, the candidate contact point can be taken as the original contact point at the time step.

[0061] Exemplarily, after obtaining the plurality of original contact points, the plurality of original contact points can be filtered and trajectory smoothed by a Kalman filtering algorithm or a sliding window to obtain the original contact point trajectory sequence.

[0062] By executing the executor model, the pose data of the executor, and the object information of the original object, the plurality of candidate contact points of the executor and the original object are determined, and according to each candidate contact point and the action data of the executor, the original contact point trajectory sequence of the executor and the original object is determined, which can improve the accuracy of contact point identification, thereby providing a high-quality supervision signal for subsequent object replacement.

[0063] In a possible implementation, the determination of the geometric transformation parameter of the new object according to the original contact point trajectory sequence in S103 includes: The geometric transformation parameter of the new object is determined according to the original operation scene and the object model of the new object with the original contact point trajectory sequence as a supervision signal.

[0064] Optionally, the geometric transformation parameter of the new object can be obtained by an optimization algorithm according to the original operation scene and the object model of the new object with the original contact point trajectory as a supervision signal.

[0065] Exemplarily, the contact trajectory model of the new object and the executor in the original operation scene can be constructed by the original operation scene and the physical model of the new object with the original contact point trajectory sequence as a supervision signal, and the geometric transformation parameter of the new object can be obtained by an optimization algorithm to the contact trajectory model, so that the new object can generate a new contact point trajectory consistent with the original contact point trajectory sequence under the same motion. The solving target of the geometric transformation parameter is to minimize the error between the predicted contact trajectory of the new object and the executor in the original operation scene and the original contact point trajectory sequence.

[0066] The geometric transformation parameter of the new object is determined according to the original operation scene and the object model of the new object with the original contact point trajectory sequence as a supervision signal, which ensures the physical rationality after replacement, realizes the object replacement of "behavior migration", improves the generalization ability of the robot, and at the same time, does not need manual adjustment of the position and angle of the new object, which is suitable for large-scale simulation data generation.

[0067] In a possible implementation, Figure 4 A flowchart for determining the geometric transformation parameter of the new object in the object replacement method provided by the embodiments of the present application is shown in FIG. 8. Figure 4 The geometric transformation parameter of the new object is determined according to the original operation scene and the object model of the new object with the original contact point trajectory sequence as a supervision signal, which includes: S401, determine contact point information of the new object according to object information of the original object and an object model of the new object.

[0068] Optionally, the contact point information of the new object can be obtained through structure alignment and cross-instance functional semantic migration according to the object information of the original object and the object model of the new object.

[0069] For example, the object model of the new object can be normalized into a target space for structure alignment and cross-instance functional semantic migration, so that the contact point information of the new object can be obtained through denormalization after migration. The target space can be a preset unit space or a space obtained through the object model of the new object or the object information of the original object.

[0070] The contact point information of the new object refers to a set of positions on the new object that should be in contact with the actuator under the current geometric configuration.

[0071] S402, in the first stage, obtain initial geometric transformation parameters of the new object according to the contact point information and taking the original contact point trajectory sequence as a supervision signal.

[0072] It can be understood that after obtaining the contact point information, the initial geometric transformation parameters of the new object can be obtained by taking the original contact point trajectory sequence as a supervision signal, combining the contact point information, the original operation scene and the physical model of the new object, and solving inversely through an optimization algorithm. In the process of solving the initial geometric transformation parameters of the new object, the initial geometric transformation parameters of the new object can be obtained through multi-stage solving.

[0073] Optionally, in the first stage, the initial geometric transformation parameters of the new object can be obtained according to the contact point information and taking the original contact point trajectory sequence as a supervision signal. The initial geometric transformation parameters can be geometric transformation parameters matched with the original contact point trajectory sequence.

[0074] For example, in the first stage, the initial geometric transformation parameters of the new object can be obtained by taking the original contact point trajectory sequence as a supervision signal, assuming that the contact point moves at a constant speed along the surface of the movable part and that the actuator and the new object are in ideal contact without sliding, constructing a contact trajectory model of the new object and the actuator in the original operation scene according to the contact point information, the original operation scene and the physical model of the new object, and inversely solving the contact trajectory model through an optimization algorithm.

[0075] S403, in the second stage, obtain the geometric transformation parameters of the new object according to the initial geometric transformation parameters and taking the original contact point trajectory sequence as a supervision signal.

[0076] It can be understood that after obtaining the initial geometric change parameter in the first stage, since the first stage does not consider the existence of slipping, fine adjustment, speed change in actual operation, the contact point trajectory may not be a smooth arc, and even jump, etc. Therefore, the newly added object may not be reasonable in physical interaction although the trajectory looks correct. Therefore, the second stage optimization can be performed on the basis of the first stage to obtain the geometric transformation parameter of the newly added object.

[0077] Optionally, in the second stage, the intersection of the movable surface of the newly added object and the actuator can be determined according to the initial geometric transformation parameter, and the original contact point trajectory sequence is used as a supervision signal to determine the geometric transformation parameter of the newly added object.

[0078] Through the processing of the first stage, the initial geometric transformation parameter of the newly added object is obtained, so that the newly added object can imitate the motion of the original object to achieve the same contact point path indicated by the original contact point trajectory sequence of the original object. Thus, the initial geometric transformation parameter of the newly added object is fitted from the historical data of the original object. On this basis, through the processing of the second stage, the interaction result of the newly added object and the actuator can be actively predicted on the basis of the same contact point path, so that the geometric logic of the contact between the newly added object and the actuator is established, thereby improving the physical rationality of the interaction result of the newly added object and the actuator and preventing pseudo-fitting and the like.

[0079] In a possible implementation manner, Figure 5 A flowchart for determining contact point information of a newly added object in an object replacement method provided by an embodiment of the present application is shown in FIG. 5. Figure 5 The determination of the contact point information of the newly added object in S401 according to the object information of the original object and the object model of the newly added object includes the following steps. S501, according to the object information of the original object and the object model of the newly added object, normalizing the point cloud of each movable part of the original object and the movable surface point cloud of the newly added object to a preset unit cube space.

[0080] Optionally, the point cloud of each movable part in the object information of the original object can be normalized to the preset unit cube space according to the object information of the original object, and the movable surface point cloud of the newly added object can be normalized to the preset unit cube space according to the object model of the newly added object, so as to establish a scale difference-free alignment reference and eliminate the scale and pose difference between the original object and the newly added object.

[0081] Exemplarily, the bounding center of the original object can be calculated for each point cloud of each movable part in the object information of the original object, the bounding center is translated to the origin, the maximum circumscribed sphere radius is calculated, and the maximum circumscribed sphere radius is scaled to a unit range, so as to normalize each point cloud of each movable part in the object information of the original object to a preset unit cube space.

[0082] Exemplarily, the bounding center of the new object can be calculated for each movable surface point cloud of the new object, the bounding center is translated to the origin, the maximum circumscribed sphere radius is calculated, and the maximum circumscribed sphere radius is scaled to a unit range, so as to normalize each movable surface point cloud of the new object to a preset unit cube space.

[0083] The unit cube space refers to a cube with a side length of 1.

[0084] S502, in the unit cube space, the new contact point trajectory sequence is determined according to the original contact point trajectory sequence.

[0085] Optionally, in the unit cube space, each original contact point in the original contact point trajectory sequence is traversed, and for the current original contact point traversed, the current original contact point is projected to the surface of the new object to obtain a new contact point, and after the traversal is completed, the new contact point trajectory sequence is obtained.

[0086] Exemplarily, for the current original contact point traversed, the nearest point cloud to the current original contact point is searched in the point cloud of the new object by a nearest neighbor search algorithm, as the new contact point.

[0087] The new contact point trajectory sequence is a set of new contact points in the normalized space.

[0088] S503, the contact point information of the new object is determined according to the new contact point trajectory sequence.

[0089] Optionally, after the new contact point trajectory sequence is obtained, the new contact point trajectory sequence is processed by inverse normalization to obtain the contact point information of the new object.

[0090] By normalizing each point cloud of each movable part of the original object and each movable surface point cloud of the new object to a preset unit cube space, and in the unit cube space, the new contact point trajectory sequence is determined according to the original contact point trajectory sequence, so that the new contact point trajectory sequence is processed by inverse normalization to obtain the contact point information of the new object, the scale difference interference can be eliminated, the cross-size object matching can be realized, the accuracy and stability of the contact point mapping can be improved, and the contact propagation under non-rigid deformation is also supported.

[0091] In one possible implementation, in step S402 above, the original contact point trajectory sequence is used as a monitoring signal, and the initial geometric transformation parameters of the newly added object are obtained based on the contact point information, including: The original contact point trajectory sequence and contact point information are input into the first optimization function, and the initial geometric transformation parameters of the newly added object are obtained by solving the first optimization function.

[0092] Optionally, the original contact point trajectory sequence and contact point information can be input into the first optimization function, and the first optimization function can be solved using a preset optimization algorithm to obtain the initial geometric transformation parameters of the newly added object. The optimization algorithm can be the L-BFGS algorithm.

[0093] The first optimization function characterizes the error between the contact point trajectory of the newly added object under ideal motion and the original contact point trajectory sequence. The first optimization function includes the geometric transformation parameters to be solved.

[0094] Specifically, the first optimization function can be as follows:

[0095] in, This is the original contact point trajectory sequence. For contact point information, Let be the geometric transformation parameters to be solved. , For scaling ratio, These are the initial motion parameters. This is the location offset information. For time steps, To initiate the contact time step, For the contact end time step, This indicates the contact point between the newly added object and the actuator after geometric transformation; that is, it indicates the position after the geometric transformation parameters are applied. After that, with It moves at a speed.

[0096] By inputting the original contact point trajectory sequence and contact point information into the first optimization function, and solving the first optimization function using the L-BFGS algorithm to obtain the initial geometric transformation parameters of the newly added object, the contact path between the newly added object and the actuator can be made as close as possible to the actual interaction trajectory in the original operation scene when performing the same action. This not only allows for visual embedding into the scene but also maintains consistency in physical interaction logic, avoiding problems such as clipping, derailment, or invalid operations. Furthermore, compared to other global optimization methods (such as genetic algorithms), the L-BFGS algorithm is faster and more accurate in local fine-grained search, achieving good computational efficiency while maintaining accuracy.

[0097] In one possible implementation, Figure 6 This is a flowchart illustrating the process of obtaining the geometric transformation parameters of a newly added object in the object replacement method provided in this application embodiment, with reference to... Figure 6 As shown, in S403 above, the original contact point trajectory sequence is used as the supervision signal, and the geometric transformation parameters of the newly added object are obtained based on the initial geometric transformation parameters, including: S601. Determine multiple transformation planes for the newly added object based on the initial geometric transformation parameters.

[0098] Optionally, initial geometric transformation parameters can be applied to the new object, and at each time step, the transformation plane of the new object at each time step can be determined.

[0099] The transformation plane refers to the local surface plane of a newly added object at each time step after the initial geometric transformation parameters are applied.

[0100] S602. Determine the geometric transformation parameters of the new object based on the original contact point trajectory sequence and the transformation planes of the new object.

[0101] Optionally, after obtaining the transformation plane, the actuator's pose data can be discretized into a series of line segments to obtain each discrete line segment. It can then be determined whether each transformation plane intersects with each discrete line segment. If an intersection exists, the error between the intersection and the corresponding point in the original contact point trajectory sequence can be determined. If the error is greater than a preset error threshold, the initial geometric transformation parameters can be adjusted based on the error. If the error is less than or equal to the preset error threshold, the initial geometric transformation parameters can be determined to be physically reasonable.

[0102] For example, the original contact point trajectory sequence, initial geometric transformation parameters, and actuator pose data can be input into a second optimization function, and the geometric transformation parameters of the newly added object can be obtained by solving the second optimization function using a preset optimization algorithm. The optimization algorithm can be the L-BFGS algorithm.

[0103] Specifically, the second optimization function can be as follows:

[0104] in, This is the original contact point trajectory sequence. For time steps, Let be the geometric transformation parameters to be solved. , For scaling ratio, For time steps The motion parameters under, For position offset information, specifically, motion parameters At each time step All have been optimized, scaling ratio With position offset information Maintain consistency over global time. This represents the point cloud of the movable portion of a newly added object after the initial geometric transformation parameters have been changed. Indicates at time step At that time, the intersection of the transformation plane of the newly added object and the pose data of the actuator, This is the pose data for the actuator.

[0105] By using initial geometric transformation parameters, multiple transformation planes of the newly added object are determined. Furthermore, by using the original contact point trajectory sequence and the transformation planes of the newly added object, its geometric transformation parameters are further determined. This allows for the explicit modeling of the kinematic constraints of the movable parts of the newly added object by abstracting the surfaces into several functional planes. It also enables the analysis of the spatial geometric relationships and contact logic between the actuator and these planes. This ensures that the optimization process no longer relies solely on coordinate alignment but is based on realistic mechanical motion mechanisms, improving the physical plausibility of the results. Additionally, it enhances the understanding and adaptation capabilities for complex joint behavior.

[0106] In one possible implementation, step S104 above determines the visual parameters of the newly added object based on the original image data corresponding to the original operation scenario, including: The original image data is input into a pre-trained illumination prediction model, which predicts the ambient illumination information of the environment in which the new object is located. Based on the ambient illumination information, the visual parameters of the new object are determined.

[0107] Optionally, the original image data is input into a pre-trained illumination prediction model, which then predicts the ambient illumination information of the environment in which the new object is located.

[0108] The illumination prediction model can be based on a diffusion model, such as the DiffusionLight model. Ambient lighting information refers to the panoramic ambient light in the original operating scene, such as a simulated HDR chrome sphere image or a two-dimensional image.

[0109] Optionally, after obtaining the ambient lighting information, the surface of the newly added object can be sampled to obtain the current surface lighting of the newly added object. Then, the ambient lighting information can be used to perform delighting or lighting enhancement processing to determine the pixel color, material properties, texture, and texture information of each pixel of the newly added object.

[0110] By inputting the raw image data into a pre-trained lighting prediction model, the lighting prediction model predicts the ambient lighting information of the environment in which the new object is located. Based on the ambient lighting information, the visual parameters of the new object are determined, which can achieve high-fidelity appearance consistency. This allows the material rendering of the new object to correctly respond to the direction of the main light source in the scene, and also match the light and shadow transitions and projection rules of the surrounding objects. At the same time, it can maintain a consistent lighting interaction effect under different viewing angles.

[0111] In one possible implementation, Figure 7 This application provides a flowchart illustrating the process of replacing an original object in a given operation scenario with a newly added object in the object replacement method provided in this embodiment. (Refer to...) Figure 7 As shown, in S105 above, based on the geometric transformation parameters and visual parameters of the newly added object, the original object in the original operation scene is replaced with the newly added object, including: S701. Delete the original object from the original operation scenario to obtain the intermediate operation scenario.

[0112] Optionally, the original object can be deleted from the original operation scenario to obtain an intermediate operation scenario.

[0113] For example, Figure 8 This is another schematic diagram of the object replacement method provided in the embodiments of this application, referred to... Figure 8 As shown, the intermediate operation scenario can be represented by the blurred and black areas in the image.

[0114] S702. Identify the empty areas in the intermediate operation scene and repair them to obtain the target operation scene.

[0115] Optionally, the hole region can be calculated based on the pose difference between the new object and the original object.

[0116] Optionally, after obtaining the void region, multiple complete point clouds can be searched around the void region, and the complete point clouds can be copied and migrated to the void region to repair the void region and obtain the target operation scene.

[0117] The empty area represents the region where the newly added object differs from the original object in the original operation scenario.

[0118] For example, geometric transformation parameters can be applied to the newly added object to construct the original object point cloud model and the transformed newly added object point cloud model. The overlap of the original object point cloud model and the transformed newly added object point cloud model can be analyzed in three-dimensional space to identify the void regions.

[0119] For example, multiple continuous point clouds with vertically upward normal vectors, uniform density, and continuous and consistent color and texture can be found near the hole region as multiple complete point clouds, and each complete point cloud can be translated as a whole to the hole region to obtain the target operation scene.

[0120] For example, slight random jittering can be added to the repaired hollow areas in the target operation scene to simulate subtle changes in real materials and prevent the repetitive texture feeling of "copy and paste".

[0121] For example, in the target operation scenario, point cloud weight gradient fusion can also be performed on the boundary of the repaired hole area to allow the color and transparency between the original boundary point and the newly inserted point to transition smoothly and avoid abrupt changes.

[0122] By identifying and repairing the empty areas in the intermediate operation scene, the target operation scene can be obtained. This can eliminate visual artifacts, improve scene integrity, and provide a reasonable embedding environment for new objects, thereby enhancing physical credibility.

[0123] S703. In the target operation scene, add a new object based on the geometric transformation parameters and visual parameters of the new object.

[0124] Optionally, in the target operation scene, the new object is added according to the geometric transformation parameters and the parameters indicated by the visual parameters.

[0125] For example, Figure 9 This is another schematic diagram of the object replacement method provided in the embodiments of this application, referred to... Figure 9 As shown, in the target operation scene, after adding a new object according to the geometric transformation parameters and the parameters indicated by the visual parameters, the following can be obtained: Figure 9 The operation scenario shown.

[0126] Based on the same inventive concept, this application also provides an object replacement device corresponding to the object replacement method. Since the principle of the device in this application is similar to the object replacement method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0127] Reference Figure 10 As shown, Figure 10 This is a schematic diagram of an object replacement device provided in an embodiment of the present application. The device includes: an acquisition module 1001, an original contact point determination module 1002, a geometric transformation determination module 1003, a vision determination module 1004, and a replacement module 1005. The acquisition module 1001 is used to acquire the original operation scene and the object model of the newly added object. The original operation scene includes the object information of the original object, the executor model, the pose data of the executor and the motion data of the executor. The object model of the newly added object includes: the object information of the newly added object. The original contact point determination module 1002 is used to determine the original contact point trajectory sequence between the actuator and the original object based on the actuator model, the actuator pose data, the actuator motion data, and the object information of the original object. The geometric transformation determination module 1003 is used to determine the geometric transformation parameters of the newly added object based on the original contact point trajectory sequence. The geometric transformation parameters include: scaling ratio, initial motion parameters, and position offset information. The vision determination module 1004 is used to determine the visual parameters of the newly added object based on the original image data corresponding to the original operation scene. Replacement module 1005 is used to replace the original object in the original operation scene with the newly added object based on the geometric transformation parameters and visual parameters of the newly added object.

[0128] In one possible implementation, the original contact point determination module 1002 is specifically used for: Based on the actuator model, the actuator pose data, and the object information of the original object, multiple candidate contact points between the actuator and the original object are determined. Based on each candidate contact point and the actuator's motion data, the original contact point trajectory sequence between the actuator and the original object is determined.

[0129] In one possible implementation, the geometric transformation determination module 1003 is specifically used for: Using the original contact point trajectory sequence as a supervision signal, the geometric transformation parameters of the new object are determined based on the original operation scenario and the object model of the new object.

[0130] In one possible implementation, the geometric transformation determination module 1003 is specifically used for: Based on the object information of the original object and the object model of the new object, determine the contact point information of the new object; In the first stage, the original contact point trajectory sequence is used as the supervision signal, and the initial geometric transformation parameters of the new object are obtained based on the contact point information. In the second stage, the original contact point trajectory sequence is used as the supervision signal, and the geometric transformation parameters of the newly added object are obtained according to the initial geometric transformation parameters.

[0131] In one possible implementation, the geometric transformation determination module 1003 is specifically used for: Based on the object information of the original object and the object model of the new object, the point clouds of the movable parts of the original object and the point clouds of the movable surfaces of the new object are normalized to the preset unit cube space. In the unit cube space, the new contact point trajectory sequence is determined based on the original contact point trajectory sequence; Based on the newly added contact point trajectory sequence, determine the contact point information of the newly added object.

[0132] In one possible implementation, the geometric transformation determination module 1003 is specifically used for: The original contact point trajectory sequence and contact point information are input into the first optimization function, and the initial geometric transformation parameters of the new object are obtained by solving the first optimization function. The first optimization function represents the error between the contact point trajectory of the new object in the ideal motion state and the original contact point trajectory sequence. The first optimization function includes the geometric transformation parameters to be solved.

[0133] In one possible implementation, the geometric transformation determination module 1003 is specifically used for: Based on the initial geometric transformation parameters, determine multiple transformation planes for the newly added object; Based on the original contact point trajectory sequence and the transformation planes of the newly added object, determine the geometric transformation parameters of the newly added object.

[0134] In one possible implementation, the vision determination module 1004 is specifically used for: The original image data is input into the pre-trained illumination prediction model, which then predicts the ambient illumination information of the environment where the new object is located. Based on ambient lighting information, determine the visual parameters of the newly added object.

[0135] In one possible implementation, module 1005 is replaced, specifically for: From the original operation scenario, delete the original object to obtain the intermediate operation scenario; Identify the empty areas in the intermediate operation scene and repair them to obtain the target operation scene. The empty areas are the areas where the new object and the original object differ in the original operation scene. In the target operation scenario, a new object is added based on the geometric transformation parameters and visual parameters of the new object.

[0136] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0137] This application also provides an electronic device, such as... Figure 11 As shown,Figure 11 The schematic diagram of the electronic device structure provided in this application embodiment includes: a processor 1101 and a memory 1102, and optionally, a bus 1103. The memory 1102 stores machine-readable instructions executable by the processor 1101. When the electronic device is running, the processor 1101 and the memory 1102 communicate via the bus 1103, and the processor 1101 executes the machine-readable instructions to perform the steps of the above-described object replacement method.

[0138] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the object replacement method described above.

[0139] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0140] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0141] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An object replacement method, characterized in that, include: Obtain the original operation scene and the object model of the newly added object. The original operation scene includes the object information of the original object, the executor model, the pose data of the executor, and the motion data of the executor. The object model of the newly added object includes: the object information of the newly added object. Based on the actuator model, the actuator pose data, the actuator motion data, and the object information of the original object, determine the original contact point trajectory sequence between the actuator and the original object; Based on the original contact point trajectory sequence, the geometric transformation parameters of the newly added object are determined. The geometric transformation parameters include: scaling ratio, initial motion parameters, and position offset information. The visual parameters of the newly added object are determined based on the original image data corresponding to the original operation scenario. Based on the geometric transformation parameters and visual parameters of the newly added object, the original object in the original operation scene is replaced with the newly added object.

2. The object replacement method according to claim 1, characterized in that, The step of determining the original contact point trajectory sequence between the actuator and the original object based on the actuator model, the actuator pose data, the actuator motion data, and the object information of the original object includes: Based on the actuator model, the actuator pose data, and the object information of the original object, multiple candidate contact points between the actuator and the original object are determined; Based on each candidate contact point and the actuator's motion data, the original contact point trajectory sequence between the actuator and the original object is determined.

3. The object replacement method according to claim 1, characterized in that, The step of determining the geometric transformation parameters of the newly added object based on the original contact point trajectory sequence includes: Using the original contact point trajectory sequence as a supervision signal, the geometric transformation parameters of the newly added object are determined based on the original operation scenario and the object model of the newly added object.

4. The object replacement method according to claim 3, characterized in that, The step of using the original contact point trajectory sequence as a supervision signal to determine the geometric transformation parameters of the newly added object based on the original operation scenario and the object model of the newly added object includes: Based on the object information of the original object and the object model of the newly added object, determine the contact point information of the newly added object; In the first stage, the original contact point trajectory sequence is used as a supervision signal, and the initial geometric transformation parameters of the newly added object are obtained based on the contact point information. In the second stage, using the original contact point trajectory sequence as a supervision signal, the geometric transformation parameters of the newly added object are obtained according to the initial geometric transformation parameters.

5. The object replacement method according to claim 4, characterized in that, The step of determining the contact point information of the newly added object based on the object information of the original object and the object model of the newly added object includes: Based on the object information of the original object and the object model of the newly added object, the point clouds of the movable parts of the original object and the point clouds of the movable surfaces of the newly added object are normalized to a preset unit cube space. In the unit cube space, a new contact point trajectory sequence is determined based on the original contact point trajectory sequence; Based on the newly added contact point trajectory sequence, the contact point information of the newly added object is determined.

6. The object replacement method according to claim 4, characterized in that, The step of using the original contact point trajectory sequence as a monitoring signal and obtaining the initial geometric transformation parameters of the newly added object based on the contact point information includes: The original contact point trajectory sequence and the contact point information are input into the first optimization function, and the initial geometric transformation parameters of the newly added object are obtained by solving the first optimization function. The first optimization function characterizes the error between the contact point trajectory of the newly added object in the ideal motion state and the original contact point trajectory sequence. The first optimization function includes: the geometric transformation parameters to be solved.

7. The object replacement method according to claim 4, characterized in that, The step of using the original contact point trajectory sequence as a monitoring signal and obtaining the geometric transformation parameters of the newly added object based on the initial geometric transformation parameters includes: Based on the initial geometric transformation parameters, determine multiple transformation planes for the newly added object; Based on the original contact point trajectory sequence and the transformation planes of the newly added object, the geometric transformation parameters of the newly added object are determined.

8. The object replacement method according to claim 1, characterized in that, The step of determining the visual parameters of the newly added object based on the original image data corresponding to the original operation scenario includes: The original image data is input into a pre-trained illumination prediction model, which then predicts the ambient illumination information of the environment where the new object is located. Based on the ambient lighting information, determine the visual parameters of the newly added object.

9. The object replacement method according to claim 1, characterized in that, The process of replacing the original object in the original operation scene with the newly added object based on the geometric transformation parameters and visual parameters of the newly added object includes: From the original operation scenario, the original object is deleted to obtain the intermediate operation scenario; Identify the empty areas in the intermediate operation scene and repair them to obtain the target operation scene. The empty areas are the areas where the newly added object and the original object differ in the original operation scene. In the target operation scenario, the new object is added based on its geometric transformation parameters and visual parameters.

10. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when the electronic device is running, are executed by the processor to perform the steps of the object replacement method as described in any one of claims 1 to 9.

Images