Three-dimensional reconstruction method and device for hinged object, electronic equipment and storage medium

Abstract

The invention provides a hinged object three-dimensional reconstruction method and device, electronic equipment and a storage medium, and relates to the technical field of three-dimensional reconstruction, and the method can obtain target images of at least two object states of a to-be-reconstructed target hinged object under a single view angle; extracting original three-dimensional Gaussian representation and target joint parameter data of the target hinged object from the target image; and according to the original three-dimensional Gaussian representation and the target joint parameter data, performing three-dimensional reconstruction of a specified object state on the target hinged object to obtain a target three-dimensional reconstruction image. In this way, the joint parameter data are applied to three-dimensional reconstruction of the target hinged object, the data fusion technology in the multi-object state is combined, the appearance, geometry and dynamic information of the hinged object can be effectively captured from a small number of images of a single view angle, efficient three-dimensional reconstruction is carried out based on Gaussian spattering, and the three-dimensional reconstruction efficiency is improved. The demand for the data volume and the calculation cost are reduced, the data utilization efficiency is improved, and the accuracy and efficiency of three-dimensional reconstruction are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and in particular to a method, device, electronic equipment and storage medium for three-dimensional reconstruction of an articulated object. Background Art

[0002] In the field of 3D modeling and computer vision, 3D reconstruction of articulated objects has always been a challenge. Articulated objects, such as furniture and robots, are difficult to accurately capture using traditional 3D reconstruction methods due to their complex geometric structures and dynamic characteristics. Existing methods usually require a large number of multi-view images, which is often unrealistic in practical applications and computationally expensive.

[0003] Furthermore, existing dynamic scene modeling approaches, such as Neural Radiance Fields (NeRF) and its derivatives, typically rely on Multilayer Perceptrons (MLP) to implicitly represent 3D objects, which are computationally very expensive, limiting their applications. Summary of the invention

[0004] The purpose of the present invention is to provide a method, device, electronic device and storage medium for three-dimensional reconstruction of an articulated object, so as to reduce data requirements and computing costs and improve the accuracy and efficiency of three-dimensional reconstruction.

[0005] In a first aspect, the present invention provides a method for three-dimensional reconstruction of an articulated object, comprising:

[0006] Acquire target images of at least two object states of the target articulated object to be reconstructed under a single viewing angle; wherein the articulation states of at least one articulated structure of the target articulated object in different object states are different;

[0007] Extracting an original three-dimensional Gaussian representation and target joint parameter data of a target articulated object from a target image; wherein the original three-dimensional Gaussian representation includes a set of original Gaussian balls corresponding to the target articulated object, and the target joint parameter data includes joint types and joint parameter information of an articulated structure of the target articulated object;

[0008] According to the original three-dimensional Gaussian representation and the target joint parameter data, the target articulated object is three-dimensionally reconstructed in the specified object state to obtain a target three-dimensional reconstructed image.

[0009] Furthermore, the original three-dimensional Gaussian representation and target joint parameter data of the target articulated object are extracted from the target image, including:

[0010] Extract features of the target image to obtain target image features;

[0011] The target image features are subjected to three-dimensional Gaussian representation conversion and joint parameter extraction to obtain the original three-dimensional Gaussian representation and target joint parameter data; wherein each original Gaussian ball in the original three-dimensional Gaussian representation includes a three-dimensional covariance matrix, position information, joint probability information, transparency information and color information, the three-dimensional covariance matrix is ​​used to represent the three-dimensional scale information and direction information of the corresponding Gaussian ball in an object state, and the joint probability information includes a set of probability values ​​for indicating that the corresponding Gaussian ball belongs to each articulated structure and non-articulated structure; the target joint parameter data includes a translation vector of a translation joint, and / or a rotation axis, a rotation center and a rotation angle of a rotation joint.

[0012] Furthermore, the target image includes an image sequence consisting of two-dimensional images in different object states; and feature extraction is performed on the target image to obtain target image features, including:

[0013] The image features of each two-dimensional image are extracted and compressed through the first group of two-dimensional convolutional networks to obtain the initial image features corresponding to each two-dimensional image;

[0014] The temporal dynamic information of the initial image features corresponding to each two-dimensional image is extracted through a set of two-dimensional convolutional long short-term memory neural networks to obtain the target image features.

[0015] Furthermore, the target image features are converted into three-dimensional Gaussian representation and joint parameters are extracted to obtain the original three-dimensional Gaussian representation and target joint parameter data, including:

[0016] The target image features are converted into a three-dimensional Gaussian representation through a second set of two-dimensional convolutional networks to obtain an original three-dimensional Gaussian representation;

[0017] The joint types and joint parameters in the target image features are extracted through a multi-layer perceptron to obtain the target joint parameter data.

[0018] Further, according to the original three-dimensional Gaussian representation and the target joint parameter data, a three-dimensional reconstruction of the target articulated object is performed at a specified object state to obtain a target three-dimensional reconstructed image, including:

[0019] According to the target joint parameter data, the original three-dimensional Gaussian representation is adjusted to the specified object state to obtain the target three-dimensional Gaussian representation;

[0020] The target three-dimensional Gaussian representation is rendered at a specified viewing angle to obtain a target three-dimensional reconstructed image.

[0021] Furthermore, the original Gaussian sphere includes a three-dimensional covariance matrix, position information, joint probability information, transparency information, and color information; according to the target joint parameter data, the original three-dimensional Gaussian representation is adjusted to the specified object state to obtain the target three-dimensional Gaussian representation, including:

[0022] According to the target joint parameter data, a set of SE3 transformation matrices are determined to transform the object state corresponding to the original three-dimensional Gaussian representation to the specified object state, and the SE3 transformation matrix corresponds to the articulated structure one by one;

[0023] According to the joint probability information of each original Gaussian ball, a set of SE3 transformation matrices are transformed on the three-dimensional covariance matrix, position information, transparency information and color information of each original Gaussian ball to obtain a set of target Gaussian balls corresponding to each original Gaussian ball;

[0024] Multiple groups of target Gaussian balls corresponding to all original Gaussian balls are determined as target three-dimensional Gaussian representations.

[0025] Further, the target three-dimensional Gaussian representation is rendered at a specified viewing angle to obtain a target three-dimensional reconstructed image, including:

[0026] Project the target three-dimensional Gaussian representation onto the two-dimensional plane corresponding to the specified viewing angle;

[0027] The target Gaussian sphere projected onto the two-dimensional plane is rendered using the following transparency blending strategy: for each pixel to be rendered, the transparency and order of the overlapping Gaussian spheres in the projection direction of the pixel are calculated, and then the overlapping Gaussian spheres are traversed in order, and the predicted color is weighted according to the transparency to obtain the final color of the pixel.

[0028] In a second aspect, the present invention further provides a device for three-dimensional reconstruction of an articulated object, comprising:

[0029] An image acquisition module, used to acquire target images of at least two object states of a target articulated object to be reconstructed under a single viewing angle; wherein the articulation state of at least one articulated structure of the target articulated object under different object states is different;

[0030] A data extraction module is used to extract the original three-dimensional Gaussian representation and target joint parameter data of the target articulated object from the target image; wherein the original three-dimensional Gaussian representation includes a set of original Gaussian balls corresponding to the target articulated object, and the target joint parameter data includes the joint type and joint parameter information of the articulated structure of the target articulated object;

[0031] The three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the specified object state of the target articulated object according to the original three-dimensional Gaussian representation and the target joint parameter data to obtain the target three-dimensional reconstructed image.

[0032] In a third aspect, the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the computer program, the method for three-dimensional reconstruction of an articulated object of the first aspect is implemented.

[0033] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for three-dimensional reconstruction of an articulated object of the first aspect is executed.

[0034] The method, device, electronic device and storage medium for three-dimensional reconstruction of an articulated object provided by the embodiment of the present invention can obtain target images of at least two object states of a target articulated object to be reconstructed under a single perspective; wherein the articulation state of at least one articulated structure of the target articulated object under different object states is different; extract the original three-dimensional Gaussian representation and target joint parameter data of the target articulated object from the target image; wherein the original three-dimensional Gaussian representation includes a group of original Gaussian balls corresponding to the target articulated object, and the target joint parameter data includes the joint type and joint parameter information of the articulated structure of the target articulated object; according to the original three-dimensional Gaussian representation and the target joint parameter data, the target articulated object is three-dimensionally reconstructed in a specified object state to obtain a target three-dimensional reconstructed image. In this way, the joint parameter data is applied to the three-dimensional reconstruction of the target articulated object, combined with the data fusion technology under multiple object states, so that the appearance, geometry and dynamic information of the articulated object can be effectively captured from a small number of images under a single perspective, and efficient three-dimensional reconstruction is performed based on Gaussian splashing, which significantly reduces the demand for data volume and computing cost, while improving data utilization efficiency, and improving the accuracy and efficiency of three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A schematic flow chart of a method for three-dimensional reconstruction of an articulated object provided by an embodiment of the present invention;

[0037] Figure 2 A data flow diagram of a method for three-dimensional reconstruction of an articulated object provided by an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of joint parameters of a translation joint provided by an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of joint parameters of a rotary joint provided by an embodiment of the present invention;

[0040] Figure 5A schematic structural diagram of a device for three-dimensional reconstruction of an articulated object provided by an embodiment of the present invention;

[0041] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] For the 3D reconstruction of articulated objects, traditional 3D reconstruction methods usually require a large number of multi-view images. Among dynamic scene modeling methods, NeRF technology currently faces multiple challenges: it relies entirely on multi-layer perceptrons, resulting in huge computing resource requirements, and the model training and rendering processes are time-consuming. In recent years, Gaussian Splatting (3DGS) technology, as a method of 3D reconstruction, has achieved impressive results by rasterizing a set of Gaussian ellipsoids to approximate the appearance of a 3D scene. However, due to the geometric and dynamic complexity of articulated objects, there are certain difficulties in applying Gaussian Splatting technology to articulated objects.

[0044] Therefore, the main disadvantages of the prior art are:

[0045] 1. High data requirements: A large number of multi-view images are required, which is difficult to achieve in practical applications.

[0046] 2. High computational cost: The representation method that relies on neural radiation fields has a large amount of computation and is not suitable for real-time or near real-time application scenarios.

[0047] 3. Lack of joint representation: For the modeling of articulated objects, existing methods often cannot effectively capture the dynamic and geometric characteristics of their joint parts.

[0048] Furthermore, regarding the above NeRF technology, in order to improve the efficiency of training and rendering, many studies have turned to explicit feature representation methods, such as voxels, grids, point clouds, and hash tables, in order to solve this problem. These methods can significantly reduce the time required to optimize the radiation field, however, they also bring higher video memory consumption and reduce the quality of the final rendered image due to the lower resolution. At the same time, video memory limitations also affect the maximum resolution of the renderable image.

[0049] At the same time, NeRF can only be used to render static objects. When facing three-dimensional articulated objects, it is usually chosen to use multiple MLPs to represent each part of the object separately, and then use predefined axis parameters to model the articulated structure. Since each part of the object is independently stored in multiple MLPs, there is often a lack of effective supervision during training to store each part of the object in the corresponding MLP. During rendering, since multiple MLPs need to be sampled independently, the cost of computing resources increases linearly with the increase in object parts. Therefore, it is difficult for the current mainstream methods to balance the final rendering speed and rendering quality.

[0050] At the same time, NeRF-based methods are difficult to generalize effectively. When faced with new objects, they often require a large number of images from different perspectives for retraining. In general application scenarios, it is difficult to obtain these images.

[0051] Based on this, the embodiments of the present invention provide a method, device, electronic device and storage medium for 3D reconstruction of an articulated object, which can effectively capture the appearance, geometry and dynamic information of the articulated object from a small number of images from a single perspective, and perform efficient 3D reconstruction. The embodiments of the present invention improve the accuracy and data efficiency of 3D reconstruction by explicitly modeling joints during the 3D reconstruction process and capturing geometry, dynamics and appearance information at the same time.

[0052] To facilitate understanding of this embodiment, a method for three-dimensional reconstruction of an articulated object disclosed in an embodiment of the present invention is first introduced in detail.

[0053] The embodiment of the present invention provides a method for three-dimensional reconstruction of an articulated object, which can be executed by an electronic device with data processing capability. Figure 1 The flowchart of a method for three-dimensional reconstruction of an articulated object is shown in FIG. 1 , and the method mainly includes the following steps S110 to S130:

[0054] Step S110 , obtaining target images of at least two object states of the target articulated object to be reconstructed under a single viewing angle; wherein the articulation states of at least one articulated structure of the target articulated object under different object states are different.

[0055] An articulated object refers to two or more rigid bodies connected together by one or more articulated structures such as joints in three-dimensional space. These rigid bodies can rotate or translate relative to each other within a limited range. The articulated object to be reconstructed is referred to as the target articulated object. Different articulated states of the articulated structure refer to: for example, the opening or closing of a cabinet door, the pulling out and pushing back of a drawer, different cabinet door opening angles or different drawer pulling distances are regarded as different articulated states. If the articulated state of one or more articulated structures of the target articulated object changes, then the target articulated object is in different object states before and after the change.

[0056] In specific implementation, the target articulated object can be photographed from a single perspective by a camera installed in a fixed position. The object state of the target articulated object can be changed during the shooting process, and a two-dimensional image can be photographed under different object states, thereby obtaining an image sequence composed of two-dimensional images under different object states, and this image sequence is the target image.

[0057] Step S120, extracting the original three-dimensional Gaussian representation and target joint parameter data of the target articulated object from the target image; wherein the original three-dimensional Gaussian representation includes a set of original Gaussian balls corresponding to the target articulated object, and the target joint parameter data includes the joint type and joint parameter information of the articulated structure of the target articulated object.

[0058] In some possible embodiments, the above step S120 may include the following steps S121 and S122:

[0059] Step S121, extracting features from the target image to obtain target image features.

[0060] The above-mentioned target image may include an image sequence composed of two-dimensional images in different object states; based on this, the image features of each two-dimensional image can be first extracted and compressed through a first group of two-dimensional convolutional networks to obtain the initial image features corresponding to each two-dimensional image; and then a group of two-dimensional convolutional long short-term memory neural networks can be used to extract the temporal dynamic information of the initial image features corresponding to each two-dimensional image to obtain the target image features.

[0061] Step S122, performing three-dimensional Gaussian representation conversion and joint parameter extraction on the target image features to obtain the original three-dimensional Gaussian representation and target joint parameter data.

[0062] Among them, each original Gaussian sphere in the original three-dimensional Gaussian representation includes a three-dimensional covariance matrix, position information, joint probability information, transparency information and color information. The three-dimensional covariance matrix is ​​used to represent the three-dimensional scale information and direction information of the corresponding Gaussian sphere in an object state. The joint probability information includes a set of probability values ​​for indicating that the corresponding Gaussian sphere belongs to each articulated structure and non-articulated structure. The color information can be represented by a spherical harmonic function for representing anisotropic colors; the target joint parameter data includes the translation vector of the translation joint, and / or the rotation axis, rotation center and rotation angle of the rotation joint.

[0063] A set of probability values ​​of an original Gaussian ball may include a probability value belonging to each joint and a probability value belonging to the main body (non-joint). Assuming that the target articulated object has 2 joints, a set of probability values ​​may include a probability of 0.8 belonging to the main body, a probability of 0.1 belonging to joint 1, and a probability of 0.1 belonging to joint 2. The target joint parameter data may include a set of joint parameter data, wherein each joint parameter data corresponds to an articulated structure, and each joint parameter data may include the joint type and joint parameter information of the corresponding articulated structure, wherein the number of articulated structures may be given in the form of hyperparameters.

[0064] In specific implementation, the target image features can be converted into a three-dimensional Gaussian representation through a second set of two-dimensional convolutional networks to obtain the original three-dimensional Gaussian representation; the joint types and joint parameters in the target image features can be extracted through a multi-layer perceptron to obtain the target joint parameter data.

[0065] Step S130 , performing three-dimensional reconstruction of the target articulated object in a specified object state according to the original three-dimensional Gaussian representation and the target joint parameter data, to obtain a target three-dimensional reconstructed image.

[0066] The above-mentioned specified object state may be one of the object states corresponding to the target image, or may not be the object state corresponding to the target image.

[0067] In some possible embodiments, the above step S130 may include the following steps S131 and S132:

[0068] Step S131, adjusting the original three-dimensional Gaussian representation to a specified object state according to the target joint parameter data to obtain a target three-dimensional Gaussian representation.

[0069] The above-mentioned original Gaussian ball may include a three-dimensional covariance matrix, position information, joint probability information, transparency information and color information. Based on this, a set of SE3 transformation matrices for transforming the object state corresponding to the original three-dimensional Gaussian representation to the specified object state can be determined according to the target joint parameter data. The SE3 transformation matrix corresponds to the articulated structure one by one; according to the joint probability information of each original Gaussian ball, the three-dimensional covariance matrix, position information, transparency information and color information of each original Gaussian ball are transformed by a set of SE3 transformation matrices to obtain a set of target Gaussian balls corresponding to each original Gaussian ball; multiple sets of target Gaussian balls corresponding to all original Gaussian balls are determined as the target three-dimensional Gaussian representation. One joint parameter data corresponds to one SE3 transformation matrix. The SE3 transformation matrix can describe the transformation of an object from one object state to another object state. Specifically, the SE3 transformation matrix can describe the transformation of the corresponding articulated structure from one articulated state to another articulated state.

[0070] Step S132, rendering the target three-dimensional Gaussian representation at a specified viewing angle to obtain a target three-dimensional reconstructed image.

[0071] The above-mentioned specified viewing angle may not be the viewing angle corresponding to the target image. Optionally, the target three-dimensional Gaussian representation may be projected onto a two-dimensional plane corresponding to the specified viewing angle; the target Gaussian sphere projected onto the two-dimensional plane is rendered using the following transparency blending strategy: for each pixel to be rendered, the transparency and order of the overlapping Gaussian spheres in the projection direction of the pixel are calculated, and then the overlapping Gaussian spheres are traversed in order, and the final color of the pixel is obtained by weighting the predicted color according to the transparency.

[0072] The method for three-dimensional reconstruction of an articulated object provided by an embodiment of the present invention can obtain target images of at least two object states of a target articulated object to be reconstructed under a single viewing angle; wherein the articulation state of at least one articulated structure of the target articulated object under different object states is different; extract the original three-dimensional Gaussian representation and target joint parameter data of the target articulated object from the target image; wherein the original three-dimensional Gaussian representation includes a group of original Gaussian balls corresponding to the target articulated object, and the target joint parameter data includes the joint type and joint parameter information of the articulated structure of the target articulated object; according to the original three-dimensional Gaussian representation and the target joint parameter data, the target articulated object is three-dimensionally reconstructed in a specified object state to obtain a target three-dimensional reconstructed image. In this way, applying the joint parameter data to the three-dimensional reconstruction of the target articulated object, combined with the data fusion technology under multiple object states, can effectively capture the appearance, geometry and dynamic information of the articulated object from a small number of images under a single viewing angle, and perform efficient three-dimensional reconstruction based on Gaussian splashing, which significantly reduces the demand for data volume and computing cost, while improving data utilization efficiency, and improving the accuracy and efficiency of three-dimensional reconstruction.

[0073] For easier understanding, see Figure 2 An embodiment of the present invention discloses a system for implementing a method for three-dimensional reconstruction of an articulated object based on Gaussian splashing, the system comprising an image encoder 200, a timing information processor 210, a Gaussian decoder 220, a joint predictor 230, a Gaussian editor 240, and a Gaussian renderer 250.

[0074] The image encoder 200 is used to process an input image sequence, extract its features and output them to the timing information processor 210, and different image frames share the same image encoder weights.

[0075] The role of the temporal information processor 210 is to capture and process the temporal dynamic information in the image sequence and output it to the Gaussian decoder 220 and the joint predictor 230. This can be achieved by integrating a convolutional long short-term memory network, which can process spatiotemporal data to understand the movement and changes of objects in the image sequence over time. The output of the temporal information processor 210 will provide key temporal information for subsequent joint prediction and Gaussian editing.

[0076] The Gaussian decoder 220 is responsible for converting the feature information processed by the temporal information processor 210 into a three-dimensional Gaussian representation in 3D space. This process involves converting the image information into a 3D representation, which can be achieved by predicting the parameters of each Gaussian ball. These parameters may include the position, scale, orientation, and color of the Gaussian ball.

[0077] The joint predictor 230 is one of the core components of the system, which is responsible for predicting the type and parameters of the joint. This can include the translation vector of the translation joint and the rotation axis, rotation center and rotation angle of the rotation joint. The prediction results of the joint predictor 230 will directly affect the editing method of the three-dimensional Gaussian representation in the Gaussian editor 240.

[0078] The Gaussian editor 240 receives the joint parameters from the joint predictor 230 and the 3D Gaussian representation from the Gaussian decoder 220, and adjusts the position and orientation of the Gaussian representation according to these parameters. This process involves applying corresponding geometric transformations to each Gaussian ball to simulate the movement of the joint. The Gaussian editor 240 also ensures that these transformed Gaussian balls can accurately reflect the actual movement state of the object by using the joint parameters of the joint predictor 230.

[0079] The Gaussian renderer 250 is the last component of the entire system, which is responsible for rendering the edited Gaussian spheres into the final image. This process includes calculating the contribution of each Gaussian sphere to the final image and compositing them based on their color, opacity and other properties.

[0080] The design of the entire system allows the joint prediction and 3D reconstruction processes to promote each other, improving the accuracy and efficiency of reconstruction. In this way, the dynamic behavior of articulated objects can be accurately reconstructed and simulated under limited viewing angles. The innovation of this method is that it can simultaneously process the appearance, geometry and dynamic information of the object, which is difficult to achieve with traditional 3D reconstruction methods. In addition, by introducing the concept of joints, the present invention can also provide richer and more accurate 3D models for fields such as robotics, virtual reality and augmented reality.

[0081] For ease of understanding, the above-mentioned method for 3D reconstruction of articulated objects is described in detail below. The method comprises the following steps:

[0082] Step 0, first obtain images of multiple states of the target to be reconstructed under one or more viewing angles, extract the two-dimensional image features of the target to be reconstructed in the image of each state, and compress the features of the image of each state into a feature map (i.e., initial image features).

[0083] The different states of the target (i.e., the above-mentioned target articulated object) refer to different states of the articulated object, such as the opening or closing of a cabinet door, the pulling out and pushing back of a drawer. Different cabinet door opening angles or different drawer pulling out distances are regarded as different articulated states.

[0084] Two-dimensional image features can be meaningful information extracted from an image, which can describe certain attributes or relationships of objects in the image. Two-dimensional image features can include one or more of edges, corners, spots, and texture patterns in the image; feature maps can be used to convert two-dimensional image features in an image into a two-dimensional numerical matrix.

[0085] It can be understood that the embodiment of the present invention achieves dimensionality reduction by extracting the two-dimensional image features of the target to be reconstructed in the image of each state and compressing these features into a feature map, thereby significantly improving the computational efficiency and storage efficiency, and ensuring that the extracted feature vector has high recognition accuracy and robustness.

[0086] The feature extraction in step 0 above can be achieved through a set of two-dimensional convolutional networks.

[0087] Step 1: Generate a set of new feature maps by extracting features from feature maps of images in different states.

[0088] It is understandable that by comprehensively analyzing the feature maps of the image in different states, it is possible to effectively supplement the position information that is difficult to observe in a single state. For example, when the cabinet door is in an open state, its image can provide relevant information inside the cabinet for the image in the closed state, which is not available in a single state.

[0089] In addition, the embodiments of the present invention can more accurately obtain the articulation information of the articulated object by comparing and analyzing the feature graphs in different states. For example, by comparing the images of the cabinet door in the open and closed states, the relevant information of the door panel, the cabinet body and the rotating shaft can be revealed. This comparative analysis method helps to deeply understand the structure and motion characteristics of the articulated object.

[0090] The above step 1 can be implemented by a set of two-dimensional convolutional long short-term memory neural networks.

[0091] Step 2: Obtain a set of Gaussian spheres representing the target to be reconstructed by processing the new feature map obtained in step 1.

[0092] It can be understood that the new feature map obtained in step 1 already contains information in different states, so the Gaussian sphere obtained in step 2 also contains information about the object in different states, such as position, scale, direction, color and other information in different states.

[0093] The Gaussian ball predicted in this step includes a set of probabilities, which are used to indicate the probability that the current Gaussian ball belongs to a specific part of the articulated object, including the probability of belonging to each joint and the probability of belonging to the body (non-joint).

[0094] Each Gaussian sphere is represented by a three-dimensional covariance matrix Σ, which is expressed as the combination of a scaling matrix S and a rotation matrix Q, where S is represented by a three-dimensional vector s and Q is represented by a rotation quaternion q: Σ = QSS T Q T .

[0095] The above step 2 can be implemented by a set of two-dimensional convolutional networks.

[0096] Step 3, by processing the new feature map obtained in step 1, obtain multiple sets of joint parameters for representing the articulated structure of the target to be reconstructed.

[0097] like Figure 3 As shown, for a translation joint, such as a drawer, the direction of movement of the drawer can be predicted. p and the moving distance δ p Among them, axis p It can be obtained by the components u in the three coordinate axis directions of the coordinate system where the drawer is located p 、v p 、w p Composition, δ p It may include distance values ​​at different times t0, t1, t2, etc.

[0098] like Figure 4 As shown, for a rotation joint, such as a cabinet door, the normal vector axis of the rotation plane of the cabinet door can be predicted r , rotation center O r and the rotation angle δ r Among them, axis r It can be obtained from the components u in the three coordinate axis directions of the coordinate system where the cabinet door is located r 、v r 、w r Composition, δ r It can include angle values ​​at different times t0, t1, t2, etc. r The components x in the three coordinate axis directions of the three-dimensional coordinate system of the target to be reconstructed r ,y r 、z r constitute.

[0099] The above step 3 can be implemented by a multi-layer perceptron, which can be a feed-forward artificial neural network consisting of multiple layers of neurons, including an input layer, a hidden layer, and an output layer.

[0100] Step 4: Convert the joint parameters obtained in step 3 into a SE3 transformation matrix, and then use the obtained SE3 transformation matrix to process the set of Gaussian spheres obtained in step 2.

[0101] Among them, the SE3 transformation matrix is ​​a representation of a special Euclidean group, which is used to describe rigid body transformations in three-dimensional space, including rotations and translations. It can be expressed as a 4×4 homogeneous matrix in the following form:

[0102]

[0103] Here, R is a 3×3 rotation matrix, representing the SO(3) group, t is a 3×1 translation vector, and 0 is a 1×3 zero vector. The SO(3) group refers to a special orthogonal group in three-dimensional space, which consists of all possible rotations that keep the length of the vector unchanged and preserve the right-hand rule (i.e., do not introduce mirror transformations).

[0104] Current translation direction vector The translation distance δ, then the translation transformation matrix R can be written as:

[0105]

[0106] Normal vector of the existing rotation plane The rotation center O = (x0, y0, z0), the rotation angle δ, then the rotation transformation matrix R can be written as:

[0107] R=T -1 R r T;

[0108] in,

[0109] R r =I+(sin(δ))K+(1-cos(δ))K 2 ;

[0110]

[0111] When using the SE3 transformation matrix to process the Gaussian sphere obtained in step 2, the SE3 transformation can be performed on Q that constitutes Σ (i.e., the three-dimensional covariance matrix), xyz used to represent the position of the Gaussian sphere, and the spherical harmonic function used to represent the anisotropic color through a set of probability values ​​(i.e., joint probability information).

[0112] Through SE3 transformation, multiple groups of target Gaussian spheres under specified states can be obtained. The number of groups of target Gaussian spheres is the same as the number of probability values ​​(for example, 3 probability values ​​correspond to 3 groups of target Gaussian spheres). Among them, an original Gaussian sphere can be transformed into multiple target Gaussian spheres through a group of probability values. Different target Gaussian spheres may be different in transparency, position and spherical harmonic function.

[0113] Step 5: Render each target Gaussian sphere obtained in step 4 into an image at a specified viewing angle.

[0114] The target Gaussian ball projected onto the two-dimensional plane corresponding to the specified viewing angle can be rendered using a transparency α blending strategy: for each pixel to be rendered, the transparency α and their order corresponding to the overlapping Gaussian balls in the projection direction of the point are calculated, and then the overlapping Gaussian balls are traversed in order, and the final color of the pixel is obtained by weighting the predicted color c according to the transparency. In the embodiment of the present invention, multiple groups of target Gaussian balls obtained by SE3 transformation are considered during rendering.

[0115] The network models used in the above method (such as the first set of two-dimensional convolutional networks, a set of two-dimensional convolutional long short-term memory neural networks, the second set of two-dimensional convolutional networks and multi-layer perceptrons, etc.) are trained. During training, the labels of sample images include information such as object state, viewpoint, and joint representation. It should be noted that the specified viewpoint during training can be different from the specified viewpoint in subsequent applications.

[0116] Compared with the prior art, the embodiments of the present invention have achieved significant improvements and advantages in many aspects. First, the embodiments of the present invention improve the accuracy of joint prediction by integrating joint parameters into the 3DGS (3D Geographic Scene) rendering process in a differentiable manner, so that the network model can more accurately capture the dynamic and geometric characteristics of articulated objects. Secondly, combined with multimodal data fusion technology (i.e., data fusion technology under multi-object state), the embodiments of the present invention effectively capture the appearance, geometry and dynamic information of articulated objects from a small number of images from a single perspective, significantly reducing the demand for data volume and improving data utilization efficiency. In terms of computational cost, the embodiments of the present invention achieve a fast three-dimensional reconstruction process through an optimized architecture design, reducing the computational cost of the model, making it more suitable for real-time or near real-time application scenarios. In summary, the embodiments of the present invention not only improve the accuracy and real-time performance of three-dimensional reconstruction, but also enhance the generalization ability of the network model and the quality of the data set, providing an efficient and accurate solution for the field of three-dimensional reconstruction of articulated objects, which has important practical application value and broad market prospects.

[0117] Corresponding to the above-mentioned method for 3D reconstruction of an articulated object, an embodiment of the present invention further provides a device for 3D reconstruction of an articulated object. Figure 5The schematic diagram of the structure of a device for 3D reconstruction of an articulated object is shown, the device comprising:

[0118] An image acquisition module 501 is used to acquire target images of at least two object states of a target articulated object to be reconstructed under a single viewing angle; wherein the articulation state of at least one articulated structure of the target articulated object in different object states is different;

[0119] A data extraction module 502 is used to extract the original three-dimensional Gaussian representation and target joint parameter data of the target articulated object from the target image; wherein the original three-dimensional Gaussian representation includes a set of original Gaussian balls corresponding to the target articulated object, and the target joint parameter data includes the joint type and joint parameter information of the articulated structure of the target articulated object;

[0120] The three-dimensional reconstruction module 503 is used to perform three-dimensional reconstruction of the target articulated object in a specified object state according to the original three-dimensional Gaussian representation and the target joint parameter data to obtain a target three-dimensional reconstructed image.

[0121] The three-dimensional reconstruction device of an articulated object provided by an embodiment of the present invention can obtain target images of at least two object states of a target articulated object to be reconstructed under a single perspective; wherein the articulation state of at least one articulated structure of the target articulated object under different object states is different; extract the original three-dimensional Gaussian representation and target joint parameter data of the target articulated object from the target image; wherein the original three-dimensional Gaussian representation includes a group of original Gaussian balls corresponding to the target articulated object, and the target joint parameter data includes the joint type and joint parameter information of the articulated structure of the target articulated object; according to the original three-dimensional Gaussian representation and the target joint parameter data, the target articulated object is three-dimensionally reconstructed in a specified object state to obtain a target three-dimensional reconstructed image. In this way, the joint parameter data is applied to the three-dimensional reconstruction of the target articulated object, combined with the data fusion technology under the multi-object state, so that the appearance, geometry and dynamic information of the articulated object can be effectively captured from a small number of images under a single perspective, and efficient three-dimensional reconstruction is performed based on Gaussian splashing, which significantly reduces the demand for data volume and computing cost, while improving data utilization efficiency, and improving the accuracy and efficiency of three-dimensional reconstruction.

[0122] Furthermore, the above-mentioned data extraction module 502 is specifically used to: perform feature extraction on the target image to obtain target image features; perform three-dimensional Gaussian representation conversion and joint parameter extraction on the target image features to obtain original three-dimensional Gaussian representation and target joint parameter data; wherein each original Gaussian ball in the original three-dimensional Gaussian representation includes a three-dimensional covariance matrix, position information, joint probability information, transparency information and color information, the three-dimensional covariance matrix is ​​used to represent the three-dimensional scale information and direction information of the corresponding Gaussian ball in an object state, and the joint probability information includes a set of probability values ​​for indicating that the corresponding Gaussian ball belongs to each articulated structure and non-articulated structure; the target joint parameter data includes the translation vector of the translation joint, and / or, the rotation axis, rotation center and rotation angle of the rotation joint.

[0123] Furthermore, the target image includes an image sequence consisting of two-dimensional images in different object states; the data extraction module 502 is also used to: extract and compress image features of each two-dimensional image through a first group of two-dimensional convolutional networks to obtain initial image features corresponding to each two-dimensional image; extract temporal dynamic information of the initial image features corresponding to each two-dimensional image through a group of two-dimensional convolutional long short-term memory neural networks to obtain target image features.

[0124] Furthermore, the above-mentioned data extraction module 502 is also used to: convert the target image features into a three-dimensional Gaussian representation through a second group of two-dimensional convolutional networks to obtain the original three-dimensional Gaussian representation; extract the joint type and joint parameters in the target image features through a multi-layer perceptron to obtain the target joint parameter data.

[0125] Furthermore, the above-mentioned 3D reconstruction module 503 is specifically used to: adjust the original 3D Gaussian representation to a specified object state according to the target joint parameter data to obtain a target 3D Gaussian representation; render the target 3D Gaussian representation at a specified viewing angle to obtain a target 3D reconstructed image.

[0126] Furthermore, the above-mentioned original Gaussian sphere includes a three-dimensional covariance matrix, position information, joint probability information, transparency information and color information; the three-dimensional reconstruction module 503 is also used to: determine a set of SE3 transformation matrices that transform the object state corresponding to the original three-dimensional Gaussian representation to the specified object state according to the target joint parameter data, and the SE3 transformation matrix corresponds to the articulated structure one by one; according to the joint probability information of each original Gaussian sphere, the three-dimensional covariance matrix, position information, transparency information and color information of each original Gaussian sphere are converted by a set of SE3 transformation matrices to obtain a set of target Gaussian spheres corresponding to each original Gaussian sphere; and determine multiple groups of target Gaussian spheres corresponding to all original Gaussian spheres as the target three-dimensional Gaussian representation.

[0127] Furthermore, the above-mentioned three-dimensional reconstruction module 503 is also used to: project the target three-dimensional Gaussian representation onto a two-dimensional plane corresponding to the specified viewing angle; the target Gaussian sphere projected onto the two-dimensional plane is rendered by the following transparency blending strategy: for each pixel point to be rendered, calculate the transparency corresponding to the overlapping Gaussian spheres in the projection direction of the pixel point and their order, and then traverse the overlapping Gaussian spheres in order, and obtain the final color of the pixel point by weighted combination of the predicted color according to the transparency.

[0128] The implementation principle and technical effects of the device provided in this embodiment are the same as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0129] like Figure 6 As shown, an electronic device 600 provided by an embodiment of the present invention includes: a processor 601, a memory 602 and a bus, the memory 602 stores a computer program that can be run on the processor 601, when the electronic device 600 is running, the processor 601 and the memory 602 communicate through the bus, and the processor 601 executes the computer program to implement the above-mentioned three-dimensional reconstruction method of articulated objects.

[0130] Specifically, the memory 602 and the processor 601 can be general-purpose memories and processors, which are not specifically limited here.

[0131] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for three-dimensional reconstruction of an articulated object in the above method embodiment is executed. The computer-readable storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk, or an optical disk, and other media that can store program codes.

[0132] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0133] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limiting, and thus other examples of the exemplary embodiments may have different values.

[0134] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and a part of a module, a program segment or a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0135] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for three-dimensional reconstruction of an articulated object, characterized in that: include: Acquire target images of at least two object states of a target articulated object to be reconstructed under a single viewing angle; wherein the articulation states of at least one articulated structure of the target articulated object under different object states are different; Extracting an original three-dimensional Gaussian representation and target joint parameter data of the target articulated object from the target image; wherein the original three-dimensional Gaussian representation includes a set of original Gaussian balls corresponding to the target articulated object, and the target joint parameter data includes joint types and joint parameter information of the articulated structure of the target articulated object; According to the original three-dimensional Gaussian representation and the target joint parameter data, the target articulated object is three-dimensionally reconstructed in a specified object state to obtain a target three-dimensional reconstructed image.

2. The method according to claim 1, characterized in that The step of extracting the original three-dimensional Gaussian representation and target joint parameter data of the target articulated object from the target image comprises: Extracting features of the target image to obtain target image features; The target image features are subjected to three-dimensional Gaussian representation conversion and joint parameter extraction to obtain the original three-dimensional Gaussian representation and the target joint parameter data; wherein each of the original Gaussian balls in the original three-dimensional Gaussian representation includes a three-dimensional covariance matrix, position information, joint probability information, transparency information and color information, the three-dimensional covariance matrix is ​​used to represent the three-dimensional scale information and direction information of the corresponding Gaussian ball in an object state, and the joint probability information includes a set of probability values ​​for indicating that the corresponding Gaussian ball belongs to each articulated structure and non-articulated structure; the target joint parameter data includes a translation vector of a translation joint, and / or a rotation axis, a rotation center and a rotation angle of a rotation joint.

3. The method according to claim 2, characterized in that The target image includes an image sequence composed of two-dimensional images in different object states; extracting features from the target image to obtain target image features includes: Extracting and compressing image features of each of the two-dimensional images through a first group of two-dimensional convolutional networks to obtain initial image features corresponding to each of the two-dimensional images; The target image features are obtained by extracting temporal dynamic information of the initial image features corresponding to each of the two-dimensional images through a group of two-dimensional convolutional long short-term memory neural networks.

4. The method according to claim 2, characterized in that: The step of performing three-dimensional Gaussian representation conversion and joint parameter extraction on the target image features to obtain the original three-dimensional Gaussian representation and the target joint parameter data includes: Converting the target image features into a three-dimensional Gaussian representation through a second set of two-dimensional convolutional networks to obtain the original three-dimensional Gaussian representation; The joint type and joint parameters in the target image features are extracted through a multi-layer perceptron to obtain the target joint parameter data.

5. The method according to claim 1, characterized in that: The method of performing three-dimensional reconstruction of a specified object state on the target articulated object according to the original three-dimensional Gaussian representation and the target joint parameter data to obtain a target three-dimensional reconstructed image includes: According to the target joint parameter data, the original three-dimensional Gaussian representation is adjusted to the specified object state to obtain a target three-dimensional Gaussian representation; The target three-dimensional Gaussian representation is rendered at a specified viewing angle to obtain the target three-dimensional reconstructed image.

6. The method according to claim 5, characterized in that The original Gaussian sphere includes a three-dimensional covariance matrix, position information, joint probability information, transparency information and color information; the original three-dimensional Gaussian representation is adjusted to the specified object state according to the target joint parameter data to obtain the target three-dimensional Gaussian representation, including: According to the target joint parameter data, determining a set of SE3 transformation matrices for transforming the object state corresponding to the original three-dimensional Gaussian representation to the specified object state, wherein the SE3 transformation matrices correspond one-to-one to the articulated structure; According to the joint probability information of each of the original Gaussian balls, a three-dimensional covariance matrix, position information, transparency information and color information of each of the original Gaussian balls are transformed by a set of SE3 transformation matrices to obtain a set of target Gaussian balls corresponding to each of the original Gaussian balls; A plurality of groups of target Gaussian spheres corresponding to all the original Gaussian spheres are determined as the target three-dimensional Gaussian representation.

7. The method according to claim 6, characterized in that The rendering of the target three-dimensional Gaussian representation at a specified viewing angle to obtain the target three-dimensional reconstructed image includes: Projecting the target three-dimensional Gaussian representation onto a two-dimensional plane corresponding to the specified viewing angle; The target Gaussian sphere projected onto the two-dimensional plane is rendered using the following transparency blending strategy: for each pixel to be rendered, the transparency and order of the overlapping Gaussian spheres in the projection direction of the pixel are calculated, and then the overlapping Gaussian spheres are traversed in order, and the final color of the pixel is obtained by weighting the transparency and combining the predicted color.

8. A device for three-dimensional reconstruction of an articulated object, characterized in that: include: An image acquisition module, used to acquire target images of at least two object states of a target articulated object to be reconstructed under a single viewing angle; wherein the articulation states of at least one articulated structure of the target articulated object under different object states are different; A data extraction module, used to extract the original three-dimensional Gaussian representation and target joint parameter data of the target articulated object from the target image; wherein the original three-dimensional Gaussian representation includes a group of original Gaussian balls corresponding to the target articulated object, and the target joint parameter data includes the joint type and joint parameter information of the articulated structure of the target articulated object; The three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the specified object state of the target articulated object according to the original three-dimensional Gaussian representation and the target joint parameter data to obtain a target three-dimensional reconstructed image.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the method for three-dimensional reconstruction of an articulated object according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for three-dimensional reconstruction of an articulated object according to any one of claims 1 to 7 is executed.

Images