Method and apparatus for three-dimensional object-based interaction

Abstract

The application relates to the technical field of three-dimensional modeling, and provides an interaction method and device based on a three-dimensional object. The method comprises the following steps: disassembling a three-dimensional model to be interacted into parts, and acquiring a target point cloud set of each part; generating a candidate assembled three-dimensional object according to the target point cloud set of each part and a geometric constraint relationship of each part; optimizing the candidate assembled three-dimensional object according to an evaluation score of the candidate assembled three-dimensional object, so as to obtain a target assembled three-dimensional object; and collecting interaction data of the target assembled three-dimensional object and an interaction subject in a simulation environment; wherein the geometric constraint relationship represents an assembly constraint condition between the parts; and the evaluation score is determined based on an assembly pose deviation value and a joint point matching deviation value of the candidate assembled three-dimensional object. The application realizes efficient three-dimensional object generation and interaction data collection.

Description

Technical Field

[0001] This application relates to the field of 3D modeling technology, and in particular to an interactive method and apparatus based on 3D objects. Background Technology

[0002] Existing 3D object generation technologies mainly rely on traditional CAD modeling or deep learning-based generation.

[0003] However, existing 3D object generation technologies still face many challenges. On the one hand, traditional CAD modeling relies on manual design, resulting in low production efficiency; while deep learning-based generative models can generate objects, they often ignore the geometric constraints between parts, leading to unreasonable or unstable generated objects. On the other hand, in terms of scene interaction, existing methods typically separate object generation and interaction, with data acquisition relying on manual annotation, resulting in low data acquisition efficiency and an inability to achieve end-to-end optimization. Furthermore, existing 3D object generation technologies can only handle static objects and cannot dynamically combine parts (hinge objects). In the field of object combination, although graph neural network-based assembly methods consider topological relationships, they lack evaluation mechanisms and are prone to producing unreasonable combinations.

[0004] Therefore, it is of great significance to provide an end-to-end solution to achieve efficient 3D object generation and interactive data acquisition. Summary of the Invention

[0005] This application provides an interactive method and apparatus based on three-dimensional objects to solve the problems of ignoring geometric constraints between parts and the inability to dynamically combine parts in the prior art, thereby achieving efficient three-dimensional object generation and interactive data acquisition.

[0006] In a first aspect, this application provides an interaction method based on three-dimensional objects, the method comprising: The interactive 3D model is decomposed into its components, and the target point cloud set of each component is obtained. Based on the target point cloud set of each part and the geometric constraint relationship of each part, a candidate assembled 3D object is generated; Based on the evaluation scores of the candidate assembled 3D objects, the candidate assembled 3D objects are optimized to obtain the target assembled 3D object; Collect interaction data between the target assembled 3D object and the interactive subject in a simulation environment; The geometric constraint relationship represents the assembly constraint conditions between the parts; The evaluation score is determined based on the assembly pose deviation value and joint matching deviation value of the candidate assembled 3D object.

[0007] Optionally, according to the interaction method based on three-dimensional objects of this application, optimizing the candidate assembled three-dimensional objects based on their evaluation scores to obtain the target assembled three-dimensional object includes: Repeat the iterative steps until the preset number of iterations is reached, and use the candidate 3D object generated in the last iteration as the target 3D object to be assembled. The iterative steps include: Based on the target point cloud set of each part, the geometric constraint relationship of each part, and the assembly correction information of the previous iteration, a candidate 3D object for assembly in this iteration is generated. Obtain the assembly pose deviation value and joint matching deviation value of the candidate assembled 3D objects in this round of iteration, and determine the evaluation score of the candidate assembled 3D objects in this round of iteration based on the assembly pose deviation value and joint matching deviation value. Based on the evaluation scores of the candidate 3D objects assembled in this round of iteration, the assembly correction information for this round of iteration is determined; The assembly correction information is used to adjust the pose of each part in the candidate 3D object to be assembled in this round of iteration; The assembly correction information from the previous iteration used in the first iteration is the preset assembly correction information.

[0008] Optionally, according to the interaction method based on 3D objects of this application, obtaining the assembly pose deviation value and joint matching deviation value of the candidate 3D objects to be assembled in the current iteration includes: Based on the positional difference between the current point cloud coordinates and the preset point cloud coordinates of the candidate 3D object to be assembled in this iteration, the positional deviation of this iteration is determined; based on the angle difference between the current rotation quaternion and the preset rotation quaternion of the candidate 3D object to be assembled in this iteration, the angle deviation of this iteration is determined; based on the positional deviation and the angle deviation of this iteration, the assembly pose deviation value of the candidate 3D object to be assembled in this iteration is determined. Based on the positional difference between the current joint position and the preset joint position of the candidate 3D object to be assembled in this round of iteration, the joint matching deviation value of the candidate 3D object to be assembled in this round of iteration is determined.

[0009] Optionally, according to the interactive method based on three-dimensional objects of this application, before generating candidate assembled three-dimensional objects, the method further includes: Obtain text commands; the text commands include the name of the 3D model to be interacted with and the names of each part; The text instructions are input into the classifier, and the parts for generating candidate assembled 3D objects are determined based on the primary and secondary categories output by the classifier. The primary category is used to characterize the object type; The secondary categories are used to characterize the part type.

[0010] Optionally, according to the interaction method based on three-dimensional objects of this application, obtaining the target point cloud set of each part includes: Point cloud sampling is performed on each of the aforementioned parts to generate a first point cloud set; Edge sampling is performed on each of the components to generate a second point cloud set; Based on the first point cloud set and the second point cloud set, determine the candidate point cloud set for each part; Gaussian noise is added to the candidate point cloud set of each part, and the target point cloud set of each part is determined based on the candidate point cloud set after adding Gaussian noise.

[0011] Optionally, according to the interaction method based on three-dimensional objects of this application, the geometric constraint relationships of each part include at least one of the following: The components satisfy symmetry constraints. The contact surface matching constraints are satisfied between the various parts; The components satisfy stability constraints.

[0012] Secondly, this application also provides an interactive device based on a three-dimensional object, the device comprising: The disassembly module is used to disassemble the 3D model to be interacted with into its parts and obtain the target point cloud set of each part. The generation module is used to generate candidate assembled 3D objects based on the target point cloud set of each part and the geometric constraint relationship of each part; The determination module is used to optimize the candidate assembled 3D objects based on their evaluation scores to obtain the target assembled 3D object. The interaction module is used to collect interaction data between the target assembled 3D object and the interactive subject in a simulation environment; The geometric constraint relationship represents the assembly constraint conditions between the parts; The evaluation score is determined based on the assembly pose deviation value and joint matching deviation value of the candidate assembled 3D object.

[0013] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the interaction method based on three-dimensional objects as described in the first aspect above.

[0014] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the interaction method based on a three-dimensional object as described in the first aspect above.

[0015] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the interaction method based on three-dimensional objects as described in the first aspect above.

[0016] The interactive method and apparatus based on 3D objects provided in this application generate candidate assembled 3D objects by using the target point cloud set of each part and the geometric constraints of each part. Optimization is then performed based on evaluation scores of assembly pose deviation and joint matching deviation, enabling the method and apparatus to implement an evaluation mechanism. This significantly reduces unreasonable and invalid 3D object assembly and improves the realism and stability of the synthesized 3D objects. By collecting interaction data between the target assembled 3D objects and the interactive subject in a simulation environment, the method and apparatus improve the efficiency of 3D object generation and interaction data acquisition, realizing an end-to-end automated process. This provides rich, realistic, and highly generalizable training data reserves for downstream large-scale visual motion models, effectively supporting the expansion of intelligent application fields such as robotics. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the interaction method based on three-dimensional objects provided in this application.

[0019] Figure 2 This is a flowchart illustrating the process of generating and assembling three-dimensional objects based on text commands, as provided in this application.

[0020] Figure 3 This is an example diagram of the target point cloud set of each part provided in this application.

[0021] Figure 4 This is a schematic diagram of the structure of the interactive device based on a three-dimensional object provided in this application.

[0022] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0024] Figure 1 This is a flowchart illustrating the interaction method based on 3D objects provided in this application, as shown below. Figure 1 As shown, the method may include: Step 110: Decompose the 3D model to be interacted with into its parts and obtain the target point cloud set of each part; Step 120: Generate candidate assembled 3D objects based on the target point cloud set of each part and the geometric constraint relationship of each part; Step 130: Optimize the candidate assembled 3D objects based on their evaluation scores to obtain the target assembled 3D object; Step 140: Collect interaction data between the target assembled 3D object and the interactive subject in the simulation environment; Among them, geometric constraints characterize the assembly constraints between parts; The evaluation score is determined based on the assembly pose deviation and joint matching deviation of the candidate assembled 3D object.

[0025] It should be noted that the execution entity of the aforementioned interactive method based on three-dimensional objects can be a computer device with capabilities for three-dimensional model processing, point cloud computing, and simulation operation, such as a server, graphics workstation, cloud computing platform, dedicated computing device, or terminal loaded with three-dimensional simulation interactive system software. This application does not impose specific limitations in this regard.

[0026] Specifically, the interactive 3D model refers to a predefined digital 3D model that needs to be virtually assembled and subsequently used for physical interaction simulation, such as a virtual refrigerator, microwave oven, or cabinet with drawers. This application does not specifically limit this.

[0027] A part refers to the basic unit obtained by disassembling a 3D model to be interacted with based on its physical structure or functional attributes. For example, a refrigerator can be disassembled into parts such as the main frame, the upper door, and the lower door. This application does not specifically limit this.

[0028] A target point cloud set refers to a set of point clouds that represents the geometric shape, edge features, or spatial contour of a part's three-dimensional surface.

[0029] Candidate assembled 3D objects refer to 3D object models with certain topological relationships that are initially or intermediately generated based on the target point cloud set and geometric constraints during the assembly and optimization process.

[0030] The target assembly of a three-dimensional object refers to a three-dimensional object whose parts are reasonably assembled and meet physical and geometric constraints after evaluation and optimization.

[0031] An interactive subject refers to a subject that physically interacts with a target assembled three-dimensional object in a simulated scene, such as a virtual robotic arm, intelligent robot, robot dog, or virtual digital human. This application does not specifically limit this.

[0032] A simulation environment is a virtual 3D platform used to simulate the physical laws of the real world, such as the Isaac simulation platform, the Gazebo open-source simulator, or the MuJoCo physics simulation engine. This application does not specifically limit this.

[0033] Interactive data refers to visual and motion data recorded during the interaction between the interactive subject and the target assembling a 3D object. For example, interactive data may include camera recording data, the motion trajectory of a robotic arm, posture data, and force feedback. This application does not specifically limit this.

[0034] Assembly pose deviation refers to the degree of difference between the current spatial state of the assembled parts in a virtual three-dimensional space and the theoretically expected spatial state. For example, assembly pose deviation may include spatial distance offset in three-dimensional translation, orientation misalignment in rotation angle, or overall attitude error based on a multi-dimensional coordinate system. This application does not specifically limit this.

[0035] Joint point matching deviation refers to the alignment error in the spatial relative position of joint points used to achieve topological connections or motion constraints between adjacent parts with relative motion relationships. For example, joint points can refer to physical connection points such as hinge points, rotation center axes, and sliding guide rails. Furthermore, joint point matching deviation can be used to evaluate the accuracy of dynamic components at their joints to ensure that the final generated 3D object can perform reasonable relative motion according to expected physical laws when subjected to force. Dynamic components can include parts such as refrigerator door hinges and drawer slides. This application does not specifically limit this.

[0036] Specifically, in step 110, when the user inputs a construction instruction, the preset interactive 3D model is read and then disassembled into its components. For example, if the interactive 3D model is a refrigerator 3D model, it can be disassembled into components such as the refrigerator's main frame, left door, and right door.

[0037] Subsequently, spatial features are extracted from each part to obtain the target point cloud set of each part, and the target point cloud set of each part is used as the input basis for generating the 3D object.

[0038] In step 120, based on the target point cloud set of each part obtained in step 110, and combined with the pre-defined geometric constraints between the parts, these parts are reassembled in a virtual 3D space. The geometric constraints characterize the assembly constraints between the parts. Examples include the contact surface matching constraint between the left door and the main frame, and the symmetry constraint between the double doors. Based on the geometric constraints of each part, a preliminary assembly of a candidate 3D object with topological connections is generated.

[0039] Optionally, the candidate assembled 3D object can be implemented based on a 3D object generation module. This 3D object generation module includes a generation model, which can be a deep neural network based on MLP (Multilayer Perceptron), a GNN (Graph Neural Network), a Transformer network based on self-attention and cross-attention mechanisms, or a policy generation network based on reinforcement learning or a multi-agent framework. This application does not impose specific limitations on this.

[0040] In step 130, since the candidate assembled 3D objects generated in the initial assembly may have spatial overlap or unreasonable connections, it is necessary to verify the candidate assembled 3D objects. Specifically, an evaluation score is calculated for the candidate assembled 3D objects, which is determined based on the assembly pose deviation value and joint matching deviation value of the candidate assembled 3D objects. Based on this evaluation score, the poses of each part in the candidate assembled 3D objects are adjusted and optimized to obtain the target assembled 3D object.

[0041] In step 140, the generated target assembled 3D object is placed in a simulation environment, where a robotic arm can serve as the interactive subject. The robotic arm dynamically interacts with the target assembled 3D object in the simulation environment, performing actions such as opening a refrigerator door or closing a refrigerator drawer. During the interaction, video recording can be used to collect interaction data between the target assembled 3D object and the interactive subject in the simulation environment. This data is then used to collect simulation data for subsequent visual motion models, improving the model's generalization ability.

[0042] The interactive method based on 3D objects provided in this application generates candidate assembled 3D objects by using the target point cloud set of each part and the geometric constraints of each part. It then optimizes the method based on evaluation scores of assembly pose deviation and joint matching deviation, thus implementing an evaluation mechanism. This significantly reduces unreasonable and invalid 3D object assembly and improves the realism and stability of the synthesized 3D objects. By collecting interaction data between the target assembled 3D objects and the interactive subject in a simulation environment, the efficiency of 3D object generation and interaction data acquisition is improved. This enables the method to achieve an end-to-end automated process, providing rich, realistic, and highly generalizable training data reserves for downstream large-scale visual motion models, effectively supporting the expansion of intelligent application fields such as robotics.

[0043] In one embodiment, optimizing candidate assembled 3D objects based on their evaluation scores to obtain a target assembled 3D object may include: Repeat the iterative steps until the preset number of times is reached, and use the candidate assembled 3D object generated in the last round of iteration as the target assembled 3D object. The iterative steps include: Based on the target point cloud set of each part, the geometric constraint relationship of each part, and the assembly correction information of the previous iteration, a candidate 3D object for assembly in this iteration is generated. Obtain the assembly pose deviation value and joint matching deviation value of the candidate assembled 3D objects in this round of iteration, and determine the evaluation score of the candidate assembled 3D objects in this round of iteration based on the assembly pose deviation value and joint matching deviation value. Based on the evaluation scores of the candidate 3D objects assembled in this round of iteration, the assembly correction information for this round of iteration is determined; Among them, the assembly correction information is used to adjust the pose of each part in the candidate 3D object to be assembled in this round of iteration; The assembly correction information from the previous iteration used in the first iteration is the preset assembly correction information.

[0044] Specifically, the previous iteration assembly correction information refers to the pose adjustment gradient or feedback instruction calculated by the evaluation module based on the previous assembly error in the previous iteration, which is used to guide the current iteration on how to avoid the assembly deviation of the previous iteration.

[0045] Preset assembly correction information refers to the initial feedback parameters assigned during the initialization phase, which can be a zero matrix or a small bias value initialized randomly.

[0046] Specifically, the iterative steps of this application are implemented based on a combination module and an evaluation and verification module. First, the combination module determines the preset assembly correction information based on the zero matrix or a randomly initialized small bias value, and uses the preset assembly correction information as the assembly correction information used in the first round of iteration.

[0047] Subsequently, in each iteration step, the assembly module obtains the target point cloud set of each part to be processed, the geometric constraint relationship of each part, and the assembly correction information of the previous iteration. Based on the above information, the assembly module performs feature fusion and pose inference to construct the part assembly code for this iteration, thereby generating the candidate 3D object for this iteration based on the part assembly code for this iteration. This process can be expressed by formula (1): Formula (1) in, Indicates the iteration round, Indicates the first The part assembly code generated by the round of iteration, This represents a generative model that performs the actions of assembling 3D objects. This represents the set of target point clouds for each component. This indicates the geometric constraints between the parts. Indicates the first Round-by-round iterative assembly of correction information.

[0048] Simultaneously, the evaluation and verification module receives candidate assembled 3D objects for this iteration and performs quality evaluation on them. The evaluation and verification module obtains the assembly pose deviation value and joint matching deviation value of the candidate assembled 3D objects for this iteration, and determines the evaluation score of the candidate assembled 3D objects for this iteration based on the assembly pose deviation value and joint matching deviation value.

[0049] Subsequently, based on the evaluation scores of the candidate 3D objects for this iteration, the optimization gradient or pose adjustment vector is calculated to determine the assembly correction information for this iteration. This assembly correction information is used to adjust the poses of each part in the candidate 3D objects for this iteration in the next iteration, guiding the generation module to stitch together in a direction with smaller deviations and better conformity to physical laws.

[0050] Finally, determine whether the current iteration count has reached the preset iteration count. If the preset iteration count has not been reached, use the assembly correction information from this iteration as the "previous iteration assembly correction information" for the next iteration, and continue repeating the above iteration steps; if the preset iteration count has been reached, terminate the iteration, and use the candidate assembled 3D object generated in the last iteration as the target assembled 3D object.

[0051] The interactive method based on 3D objects provided in this application generates candidate 3D objects for the current iteration by repeatedly executing iterative steps and based on the target point cloud set of each part, the geometric constraint relationship of each part, and the assembly correction information of the previous iteration. The method quantifies and evaluates the assembly pose deviation value and joint matching deviation value, and then generates new correction information to optimize the candidate 3D objects for the current iteration by using the evaluation score. This method can continuously optimize the assembly state of 3D objects under complex geometric constraints, avoiding unreasonable phenomena such as model clipping and joint misalignment that are easy to occur in single static assembly. Thus, it generates 3D objects that conform to the physical functional constraints of the real world, providing high-quality and high-fidelity dynamic 3D digital resources for subsequent simulation interaction.

[0052] In one embodiment, obtaining the assembly pose deviation value and joint matching deviation value of the candidate assembled 3D object in the current iteration may include: Based on the positional difference between the current point cloud coordinates and the preset point cloud coordinates of the candidate assembled 3D objects in this iteration, the positional deviation of this iteration is determined; based on the angle difference between the current rotation quaternion and the preset rotation quaternion of the candidate assembled 3D objects in this iteration, the angle deviation of this iteration is determined; based on the positional deviation and the angle deviation of this iteration, the assembly pose deviation value of the candidate assembled 3D objects in this iteration is determined. Based on the positional difference between the current joint position and the preset joint position of the candidate 3D object to be assembled in this iteration, the joint matching deviation value of the candidate 3D object to be assembled in this iteration is determined.

[0053] Specifically, the current point cloud coordinates refer to the three-dimensional coordinates of the predicted sampling points on the surface of each part in the current virtual three-dimensional space of the candidate assembled three-dimensional object in this iteration.

[0054] Preset point cloud coordinates refer to the ideal three-dimensional coordinate values ​​of sampling points on the surface of each part, which are pre-set according to physical laws or the structure of real-world objects.

[0055] The current rotation quaternion is used to represent the four-dimensional vector of the current rotational orientation of each part in the candidate assembled 3D object in this iteration. Rotation quaternions can effectively avoid gimbal lock problems in 3D rotation.

[0056] The preset rotational quaternion refers to the four-dimensional vector of rotational posture that each part should have in an ideal assembly state.

[0057] The current joint position refers to the current spatial coordinates of the parts with relative motion relationships at the connection point in the candidate assembled 3D object in this iteration.

[0058] The preset joint position refers to the ideal spatial coordinates of the connection points of each part in the ideal assembly state.

[0059] Specifically, the evaluation and verification module will evaluate the quality of the candidate assembled 3D objects in this iteration from multiple dimensions such as positional accuracy and overall rationality. The overall approach to the evaluation score can be expressed by the following formula (2): Formula (2) in, Indicates the first Round of iterative evaluation scores, This represents the evaluation function used to perform quality assessment on candidate assembled 3D objects.

[0060] Optionally, It can be a regression scoring network built on MLP, a Transformer scoring model based on cross-attention mechanism, or a pre-built loss function for quantifying pose deviation, etc. This application does not make specific limitations in this regard.

[0061] In order to obtain an evaluation score The specific quantitative values ​​are used as the basis for determining the termination of the iteration in the verification phase. The evaluation and verification module further decomposes and calculates the assembly pose deviation value and joint matching deviation value of each part of the candidate assembled 3D object in this round of iteration.

[0062] The evaluation and verification module obtains the current point cloud coordinates of the candidate assembled 3D objects in this iteration and obtains the ideal preset point cloud coordinates, calculates the positional difference between the two, and thus determines the positional deviation of this iteration. Subsequently, it obtains the current rotation quaternion of the candidate assembled 3D objects in this iteration and obtains the ideal preset rotation quaternion, calculates the angle difference between the two, and thus determines the angle deviation of this iteration. Subsequently, the evaluation and verification module combines the positional deviation and the angle deviation of this iteration to determine the assembly pose deviation value of the candidate assembled 3D objects in this iteration. The assembly pose deviation value can be expressed by the following formula (3): Formula (3) in, This indicates the assembly pose deviation value. Indicates the preset point cloud coordinates. Indicates the current point cloud coordinates. This indicates the preset rotation quaternion. This represents the current rotation quaternion, and arccos represents the arccosine function. Indicates the first One part.

[0063] Simultaneously, the evaluation and verification module obtains the current joint position and the preset joint position of the candidate assembled 3D object in this iteration, and determines the joint matching deviation value in this iteration based on the positional difference between the two. The joint matching deviation value can be expressed by the following formula (4): Formula (4) in, This represents the joint matching deviation value. Indicates the current position of the joint. This indicates the preset joint position.

[0064] Finally, the calculated assembly pose deviation value and joint matching deviation value are added together to obtain the specific evaluation score. The evaluation score can be expressed by the following formula (5): Formula (5) The interactive method based on 3D objects provided in this application determines the positional deviation by the positional difference between the current point cloud coordinates and the preset point cloud coordinates, and determines the angular deviation by the angular difference between the current rotation quaternion and the preset rotation quaternion. Combining the positional deviation and angular deviation, the assembly pose deviation value is obtained. Furthermore, the joint matching deviation value is determined by the positional difference between the current joint position and the preset joint position. This method can decompose complex quality assessment into multiple fine-grained dimensions such as spatial translation, spatial rotation, and physical connection hubs, accurately assessing the rationality of the assembly of each part in the 3D model. It avoids the visual clipping or assembly invalidity problems that are easily caused by a single evaluation index, thereby greatly improving the physical realism and geometric accuracy of the assembled 3D objects.

[0065] In one embodiment, before generating the candidate assembled 3D object, the following may also be included: Obtain text commands; text commands include the name of the 3D model to be interacted with and the names of each part; Input the text commands into the classifier, and determine the parts to generate candidate assembled 3D objects based on the primary and secondary categories output by the classifier. The primary category is used to characterize the object type; Secondary categories are used to characterize part types.

[0066] Specifically, text commands refer to text information input by a user or external system to guide the generation of three-dimensional objects and their parts. For example, text commands can be natural language strings such as "refrigerator upper door," "refrigerator lower door," and "refrigerator main frame" typed by the user through an interactive interface; or they can be structured text generated by an intelligent voice assistant through speech recognition. This application does not impose specific limitations on this.

[0067] A classifier refers to an algorithmic model deployed in the input mapping layer of the 3D object generation module, used to perform semantic parsing and hierarchical feature extraction on the input text. For example, a classifier can be a text classification neural network based on deep learning; it can also be a semantic matching engine based on a pre-defined rule dictionary and knowledge graph. This application does not impose specific limitations on this.

[0068] The primary category is used to represent the overall category of the three-dimensional model at a macroscopic level. For example, the primary category can be a macroscopically defined object class such as a refrigerator, microwave oven, or storage cabinet. This application does not impose specific limitations on this.

[0069] Secondary categories are used to characterize specific micro-component categories belonging to primary categories. For example, for the primary category "refrigerator," the corresponding secondary categories could be specific parts such as the upper door, lower door, main frame, and drawers. This application does not impose specific limitations on this.

[0070] Specifically, it retrieves the text commands input by the user. These text commands contain the name of the 3D model to be interacted with, as well as the names of each part. For example, the user inputs a sequence of text commands containing "refrigerator upper door, refrigerator lower door, refrigerator main frame".

[0071] The acquired text commands are then input into the classifier in the input mapping layer. This classifier performs semantic parsing and hierarchical deconstruction on the received text commands, and outputs the corresponding primary and secondary categories. For example, when the input is "refrigerator upper door," the primary category representing the object type is "refrigerator," and the secondary category representing the part type is "door."

[0072] Finally, based on the primary and secondary categories output by the classifier, and combined with the geometric constraints of each category, the parts for generating candidate assembled 3D objects are determined.

[0073] Figure 2 This is a flowchart illustrating the process of generating and assembling 3D objects based on text commands, as provided in this application. Figure 2 As shown, the process retrieves a text command containing the names of the 3D model to be interacted with and its parts. This text command is then input into the 3D object generation module, which uses an internal classifier to parse the command and generate corresponding individual parts, such as part 1, part 2, and so on, up to part n. Next, the individual parts are assembled according to geometric constraints to obtain the target assembled 3D object. Finally, the target assembled 3D object is placed in a simulation environment to interact with the interactive subject, thus completing the end-to-end process from text input to dynamic interaction.

[0074] The interactive method based on 3D objects provided in this application obtains text commands containing the name of the 3D model to be interacted with and the name of the parts, inputs the text commands into a classifier for parsing, and then uses the primary and secondary categories output by the classifier to determine the parts to generate candidate assembled 3D objects. This method establishes a hierarchical mapping relationship from semantic concepts to specific 3D parts, avoiding the problems of chaotic part generation or feature mismatch caused by the lack of semantic guidance in traditional generation methods, thereby greatly improving the flexibility and semantic accuracy of 3D object generation.

[0075] In one embodiment, obtaining the target point cloud set for each part may include: Point cloud sampling is performed on each part to generate the first point cloud set; Edge sampling is performed on each part to generate a second set of point clouds; Based on the first and second point cloud sets, determine the candidate point cloud sets for each part; Gaussian noise is added to the candidate point cloud set of each part, and the target point cloud set of each part is determined based on the candidate point cloud set after adding Gaussian noise.

[0076] The first point cloud set refers to the set of data points obtained by globally sampling the surface of a 3D part. For example, a density distribution-based sampling algorithm can be used to ensure that the generated point cloud can uniformly cover the entire surface of the 3D part; the density distribution-based sampling algorithm can be the FPS (Farthest Point Sampling) algorithm or the Uniform Random Sampling algorithm. This application does not make specific limitations in this regard.

[0077] The second point cloud refers to the set of data points obtained by sampling the contour areas of a 3D part, such as boundaries, corners, and regions with abrupt changes in curvature. The second point cloud is used to capture the sharp features and structural details of the part.

[0078] Gaussian noise refers to random error data whose probability density function follows a Gaussian distribution. For example, introducing Gaussian noise into a point cloud can simulate the physical measurement errors generated by visual sensors such as depth cameras and LiDAR when scanning objects in the real world.

[0079] Specifically, the individual parts of the 3D model to be interacted with are obtained, and the surface of each part is sampled as a whole to generate a first set of point clouds representing the basic shape of the parts.

[0080] Subsequently, to prevent the contours of the parts from becoming blurred during subsequent assembly, edge sampling is performed on each part. For example, edge regions are obtained by identifying curvature variation areas on the surface of each part, and point clouds are extracted from the edge regions to generate a second set of point clouds representing the edges of the parts.

[0081] Next, the generated first point cloud set and the second point cloud set are spliced ​​and merged, and the coordinate data of the two are brought together in the same coordinate system. Thus, based on the first point cloud set and the second point cloud set, the candidate point cloud set for each part is determined.

[0082] Finally, Gaussian noise is added to the candidate point cloud sets of each part, and the target point cloud set of each part is determined based on the candidate point cloud sets after adding Gaussian noise. The process of adding Gaussian noise can be expressed by the following formula (6): Formula (6) in, Indicates the first The target point cloud set of each component. Indicates the first A candidate point cloud set for each part. The variance of Gaussian noise is represented. This represents a Gaussian noise function that follows a normal distribution.

[0083] Figure 3 This is an example diagram of the target point cloud set of each part provided in this application, such as... Figure 3 As shown, the collected overall point cloud (corresponding to the first point cloud set mentioned above) and edge point cloud (corresponding to the second point cloud set mentioned above) are input into the classifier for feature extraction and preliminary encoding; simultaneously, externally input text instructions are acquired as conditional guidance information. Subsequently, the encoded point cloud features are input into the backbone of the deep neural network, which contains 24 repeating structural layers. Each layer contains a self-attention module and a parallel cross-attention module. The self-attention module is used to mine the spatial correlation features within the point cloud, while the text instructions are directly input into the cross-attention module of each layer through long connections, interacting with the point cloud features, thereby using text semantics to guide the generation and detail optimization of each part.

[0084] The interactive method based on 3D objects provided in this application performs point cloud sampling and edge sampling on each part separately, and determines the candidate point cloud set for each part based on the generated first point cloud set and second point cloud set. This method can ensure that the point cloud uniformly covers the object surface while fully preserving the contour and detailed features of the parts, greatly enriching the sampling information of the point cloud. Furthermore, by adding Gaussian noise to the candidate point cloud set to determine the target point cloud set, this method avoids the problem of overfitting the model on ideal noise-free data, thereby significantly improving the robustness and generalization ability of the generated model and the subsequent large-scale visual action model when dealing with complex interactive scenarios.

[0085] In one embodiment, the geometric constraints of the parts may include at least one of the following: The components satisfy symmetry constraints. All parts satisfy the contact surface matching constraint; The stability constraints are satisfied between the components.

[0086] Specifically, symmetry constraints refer to the geometric mapping relationship that parts with mirror or rotational symmetry design properties must satisfy around a specific plane or axis of symmetry during spatial assembly. For example, such parts may include double doors, four table legs, etc. This application does not specifically limit this.

[0087] Contact surface matching constraints refer to the requirement that, in the real physical world, the surfaces of two parts that need to be closely fitted and connected must be tightly adjacent during virtual assembly, without any unreasonable floating gaps or mutual penetration. For example, contact surface matching constraints can be applied to the joints between doors and door frames, or the contact surfaces between drawers and tracks. This application does not specifically limit this application.

[0088] Stability constraints refer to the requirements that an assembled 3D object must maintain static stability during placement and interaction, taking into account factors such as gravity, support forces, and mechanical equilibrium in a physical simulation environment. For example, a stability constraint can require the center of gravity of the assembled object to fall within the projection range of its bottom support surface.

[0089] Specifically, the geometric constraints of each part include at least one of the following: Optionally, symmetry constraints are satisfied between the parts. For example, when generating a 3D object of a "double-door refrigerator," the left and right doors are symmetrical components. Therefore, during assembly, the left and right doors are constrained to maintain strict symmetry with respect to the central cross-section of the refrigerator's main frame, thus ensuring that the generated model conforms to conventional industrial design specifications.

[0090] Optionally, the contact surface matching constraints between the parts are satisfied. For example, when assembling the refrigerator door and the refrigerator body frame, the distance between the boundary point clouds where the two parts intersect is calculated to constrain the contact surfaces of the two parts to fit together, thereby avoiding physically impossible gaps in the generated model and preventing clipping errors caused by coordinate overlap.

[0091] Optionally, stability constraints are satisfied between the parts. To ensure that the generated 3D object can be successfully placed in the simulation environment and interact with the interactive subject, the mass distribution of each part is calculated during assembly. For example, by constraining the center of gravity of the assembled 3D object to be within the supporting polygon of the base, it is ensured that the 3D object will not tilt or collapse abnormally when subjected to gravity or slight touch from the robotic arm.

[0092] The interactive method based on 3D objects provided in this application ensures the structural regularity of assembled 3D objects through symmetry constraints, ensures the tightness of part splicing and prevents spatial penetration through contact surface matching constraints, and ensures the force balance of assembled 3D objects in the physical environment through stability constraints. This method can strictly guide the scientific assembly of complex objects under multi-dimensional geometric constraints, avoiding the problems of model appearance distortion and instability in simulation environment that are easily caused by unconstrained random splicing. This greatly improves the physical realism of assembled 3D objects and provides high-quality data support for subsequent collection of interactive data.

[0093] The interactive device based on three-dimensional objects provided in this application will be described below. The interactive device based on three-dimensional objects described below can be referred to and correspond to the interactive method based on three-dimensional objects described above, and can achieve the same technical effect. It will not be repeated here.

[0094] Figure 4 This is a structural schematic diagram of the interactive device based on three-dimensional objects provided in this application, such as... Figure 4 As shown, the device may include: The disassembly module 410 is used to disassemble the 3D model to be interacted with into its parts and obtain the target point cloud set of each part. The generation module 420 is used to generate candidate assembled 3D objects based on the target point cloud set of each part and the geometric constraint relationship of each part. The determining module 430 is used to optimize the candidate assembled 3D object based on the evaluation score of the candidate assembled 3D object to obtain the target assembled 3D object; The interaction module 440 is used to collect the interaction data between the target assembled 3D object and the interactive subject in a simulation environment; The geometric constraint relationship represents the assembly constraint conditions between the parts; The evaluation score is determined based on the assembly pose deviation value and joint matching deviation value of the candidate assembled 3D object.

[0095] In one embodiment, the determining module 430 is specifically used for: Repeat the iterative steps until the preset number of iterations is reached, and use the candidate 3D object generated in the last iteration as the target 3D object to be assembled. The iterative steps include: Based on the target point cloud set of each part, the geometric constraint relationship of each part, and the assembly correction information of the previous iteration, a candidate 3D object for assembly in this iteration is generated. Obtain the assembly pose deviation value and joint matching deviation value of the candidate assembled 3D objects in this round of iteration, and determine the evaluation score of the candidate assembled 3D objects in this round of iteration based on the assembly pose deviation value and joint matching deviation value. Based on the evaluation scores of the candidate 3D objects assembled in this round of iteration, the assembly correction information for this round of iteration is determined; The assembly correction information is used to adjust the pose of each part in the candidate 3D object to be assembled in this round of iteration; The assembly correction information from the previous iteration used in the first iteration is the preset assembly correction information.

[0096] In one embodiment, the determining module 430 is specifically used for: Based on the positional difference between the current point cloud coordinates and the preset point cloud coordinates of the candidate 3D object to be assembled in this iteration, the positional deviation of this iteration is determined; based on the angle difference between the current rotation quaternion and the preset rotation quaternion of the candidate 3D object to be assembled in this iteration, the angle deviation of this iteration is determined; based on the positional deviation and the angle deviation of this iteration, the assembly pose deviation value of the candidate 3D object to be assembled in this iteration is determined. Based on the positional difference between the current joint position and the preset joint position of the candidate 3D object to be assembled in this round of iteration, the joint matching deviation value of the candidate 3D object to be assembled in this round of iteration is determined.

[0097] In one embodiment, before generating the candidate assembled 3D object, the generation module 420 is further configured to: Obtain text commands; the text commands include the name of the 3D model to be interacted with and the names of each part; The text instructions are input into the classifier, and the parts for generating candidate assembled 3D objects are determined based on the primary and secondary categories output by the classifier. The primary category is used to characterize the object type; The secondary categories are used to characterize the part type.

[0098] In one embodiment, the disassembly module 410 is specifically used for: Point cloud sampling is performed on each of the aforementioned parts to generate a first point cloud set; Edge sampling is performed on each of the components to generate a second point cloud set; Based on the first point cloud set and the second point cloud set, determine the candidate point cloud set for each part; Gaussian noise is added to the candidate point cloud set of each part, and the target point cloud set of each part is determined based on the candidate point cloud set after adding Gaussian noise.

[0099] In one embodiment, the geometric constraints of the parts include at least one of the following: The components satisfy symmetry constraints. The contact surface matching constraints are satisfied between the various parts; The components satisfy stability constraints.

[0100] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute the interaction method based on a three-dimensional object as described in any of the above embodiments, for example including: The interactive 3D model is decomposed into its components, and the target point cloud set of each component is obtained. Based on the target point cloud set of each part and the geometric constraint relationship of each part, a candidate assembled 3D object is generated; Based on the evaluation scores of the candidate assembled 3D objects, the candidate assembled 3D objects are optimized to obtain the target assembled 3D object; Collect interaction data between the target assembled 3D object and the interactive subject in a simulation environment; The geometric constraint relationship represents the assembly constraint conditions between the parts; The evaluation score is determined based on the assembly pose deviation value and joint matching deviation value of the candidate assembled 3D object.

[0101] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] On the other hand, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the interactive methods based on three-dimensional objects provided by the methods described above, the method comprising: The interactive 3D model is decomposed into its components, and the target point cloud set of each component is obtained. Based on the target point cloud set of each part and the geometric constraint relationship of each part, a candidate assembled 3D object is generated; Based on the evaluation scores of the candidate assembled 3D objects, the candidate assembled 3D objects are optimized to obtain the target assembled 3D object; Collect interaction data between the target assembled 3D object and the interactive subject in a simulation environment; The geometric constraint relationship represents the assembly constraint conditions between the parts; The evaluation score is determined based on the assembly pose deviation value and joint matching deviation value of the candidate assembled 3D object.

[0103] Furthermore, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the interactive methods based on three-dimensional objects provided by the above methods, the method including: The interactive 3D model is decomposed into its components, and the target point cloud set of each component is obtained. Based on the target point cloud set of each part and the geometric constraint relationship of each part, a candidate assembled 3D object is generated; Based on the evaluation scores of the candidate assembled 3D objects, the candidate assembled 3D objects are optimized to obtain the target assembled 3D object; Collect interaction data between the target assembled 3D object and the interactive subject in a simulation environment; The geometric constraint relationship represents the assembly constraint conditions between the parts; The evaluation score is determined based on the assembly pose deviation value and joint matching deviation value of the candidate assembled 3D object.

[0104] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An interaction method based on three-dimensional objects, characterized in that, The method includes: The interactive 3D model is decomposed into its components, and the target point cloud set of each component is obtained. Based on the target point cloud set of each part and the geometric constraint relationship of each part, a candidate assembled 3D object is generated; Based on the evaluation scores of the candidate assembled 3D objects, the candidate assembled 3D objects are optimized to obtain the target assembled 3D object; Collect interaction data between the target assembled 3D object and the interactive subject in a simulation environment; The geometric constraint relationship represents the assembly constraint conditions between the parts; The evaluation score is determined based on the assembly pose deviation value and joint matching deviation value of the candidate assembled 3D object; The step of optimizing the candidate assembled 3D objects based on their evaluation scores to obtain the target assembled 3D object includes: Repeat the iterative steps until the preset number of iterations is reached, and use the candidate 3D object generated in the last iteration as the target 3D object to be assembled. The iterative steps include: Based on the target point cloud set of each part, the geometric constraint relationship of each part, and the assembly correction information of the previous iteration, a candidate 3D object for assembly in this iteration is generated. Obtain the assembly pose deviation value and joint matching deviation value of the candidate assembled 3D objects in this round of iteration, and determine the evaluation score of the candidate assembled 3D objects in this round of iteration based on the assembly pose deviation value and joint matching deviation value. Based on the evaluation scores of the candidate 3D objects assembled in this round of iteration, the assembly correction information for this round of iteration is determined; The assembly correction information is used to adjust the pose of each part in the candidate 3D object to be assembled in this round of iteration; The assembly correction information from the previous iteration used in the first iteration is the preset assembly correction information; The process of obtaining the assembly pose deviation value and joint matching deviation value of the candidate assembled 3D objects in this iteration includes: Based on the positional difference between the current point cloud coordinates and the preset point cloud coordinates of the candidate 3D object to be assembled in this iteration, the positional deviation of this iteration is determined; based on the angle difference between the current rotation quaternion and the preset rotation quaternion of the candidate 3D object to be assembled in this iteration, the angle deviation of this iteration is determined; based on the positional deviation and the angle deviation of this iteration, the assembly pose deviation value of the candidate 3D object to be assembled in this iteration is determined. Based on the positional difference between the current joint position and the preset joint position of the candidate 3D object to be assembled in this round of iteration, the joint matching deviation value of the candidate 3D object to be assembled in this round of iteration is determined.

2. The interaction method based on three-dimensional objects according to claim 1, characterized in that, Before generating the candidate assembled 3D object, the process also includes: Obtain text commands; the text commands include the name of the 3D model to be interacted with and the names of each part; The text instructions are input into the classifier, and the parts for generating candidate assembled 3D objects are determined based on the primary and secondary categories output by the classifier. The primary category is used to characterize the object type; The secondary categories are used to characterize the part type.

3. The interaction method based on three-dimensional objects according to claim 1, characterized in that, The acquisition of the target point cloud set for each part includes: Point cloud sampling is performed on each of the aforementioned parts to generate a first point cloud set; Edge sampling is performed on each of the components to generate a second point cloud set; Based on the first point cloud set and the second point cloud set, determine the candidate point cloud set for each part; Gaussian noise is added to the candidate point cloud set of each part, and the target point cloud set of each part is determined based on the candidate point cloud set after adding Gaussian noise.

4. The interaction method based on three-dimensional objects according to claim 1, characterized in that, The geometric constraints of the parts include at least one of the following: The components satisfy symmetry constraints. The contact surface matching constraints are satisfied between the various parts; The components satisfy stability constraints.

5. An interactive device based on a three-dimensional object, characterized in that, The device includes: The disassembly module is used to disassemble the 3D model to be interacted with into its parts and obtain the target point cloud set of each part. The generation module is used to generate candidate assembled 3D objects based on the target point cloud set of each part and the geometric constraint relationship of each part; The determination module is used to optimize the candidate assembled 3D objects based on their evaluation scores to obtain the target assembled 3D object. The interaction module is used to collect interaction data between the target assembled 3D object and the interactive subject in a simulation environment; The geometric constraint relationship represents the assembly constraint conditions between the parts; The evaluation score is determined based on the assembly pose deviation value and joint matching deviation value of the candidate assembled 3D object; The determining module is specifically used for: Repeat the iterative steps until the preset number of iterations is reached, and use the candidate 3D object generated in the last iteration as the target 3D object to be assembled. The iterative steps include: Based on the target point cloud set of each part, the geometric constraint relationship of each part, and the assembly correction information of the previous iteration, a candidate 3D object for assembly in this iteration is generated. Obtain the assembly pose deviation value and joint matching deviation value of the candidate assembled 3D objects in this round of iteration, and determine the evaluation score of the candidate assembled 3D objects in this round of iteration based on the assembly pose deviation value and joint matching deviation value. Based on the evaluation scores of the candidate 3D objects assembled in this round of iteration, the assembly correction information for this round of iteration is determined; The assembly correction information is used to adjust the pose of each part in the candidate 3D object to be assembled in this round of iteration; The assembly correction information from the previous iteration used in the first iteration is the preset assembly correction information; The process of obtaining the assembly pose deviation value and joint matching deviation value of the candidate assembled 3D objects in this iteration includes: Based on the positional difference between the current point cloud coordinates and the preset point cloud coordinates of the candidate 3D object to be assembled in this iteration, the positional deviation of this iteration is determined; based on the angle difference between the current rotation quaternion and the preset rotation quaternion of the candidate 3D object to be assembled in this iteration, the angle deviation of this iteration is determined; based on the positional deviation and the angle deviation of this iteration, the assembly pose deviation value of the candidate 3D object to be assembled in this iteration is determined. Based on the positional difference between the current joint position and the preset joint position of the candidate 3D object to be assembled in this round of iteration, the joint matching deviation value of the candidate 3D object to be assembled in this round of iteration is determined.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the interactive method based on three-dimensional objects as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the interactive method based on three-dimensional objects as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the interactive method based on three-dimensional objects as described in any one of claims 1 to 4.

Images