Method, device, equipment and storage medium for reconstructing three-dimensional shape of object

Abstract

The present invention provides a method, apparatus, device, and storage medium for reconstructing the three-dimensional shape of an object, involving artificial intelligence image processing. The method comprises: normalizing a two-dimensional image of a single object to obtain a normalized image; inputting the normalized image into an object shape reconstruction network, obtaining corresponding feature information through a feature extractor of the object shape reconstruction network, and inputting the information into a generator of the object shape reconstruction network to obtain a three-dimensional tensor of confidence scores for the object's predicted voxels; and binarizing each element in the three-dimensional confidence tensor based on a preset threshold to obtain voxels representing the object's three-dimensional shape. The present invention can accurately and efficiently recover the three-dimensional structure of an object from a two-dimensional image, improving the efficiency of three-dimensional shape reconstruction and saving manpower and material resources.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence image processing technology, and in particular to a method, device, equipment and storage medium for reconstructing the three-dimensional shape of an object. Background Art

[0002] One of the main goals of 3D reconstruction is to recover the 3D structure of an object from a 2D image. In recent years, with the development of industries such as virtual reality, 3D printing, autonomous driving, smart healthcare, and film and television production, the demand for 3D models has exploded. Traditional manual modeling methods are no longer able to meet this demand. Accurate and efficient 3D reconstruction methods are crucial to solving this problem. Summary of the Invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, an embodiment of the present invention provides a method for reconstructing the three-dimensional shape of an object, aiming to accurately and efficiently restore the three-dimensional structure of an object from a two-dimensional image.

[0004] In a first aspect, an embodiment of the present invention provides a method for reconstructing the three-dimensional shape of an object, comprising: normalizing a two-dimensional image including a single object to obtain a normalized image; inputting the normalized image into an object shape reconstruction network, obtaining corresponding feature information through a feature extractor of the object shape reconstruction network, and inputting the feature information into a generator of the object shape reconstruction network to obtain a three-dimensional confidence tensor of the predicted voxels of the object; and binarizing each element in the three-dimensional confidence tensor based on a preset threshold to obtain a voxel representing the three-dimensional shape of the object.

[0005] The method for reconstructing the three-dimensional shape of an object according to an embodiment of the present invention has at least the following beneficial effects: by normalizing a two-dimensional image including a single object and inputting the image into an object shape reconstruction network, the object shape network is trained, and a three-dimensional system tensor of the confidence of the predicted voxels is obtained through the trained object shape reconstruction network, and the corresponding voxels are obtained after binarization, so that the three-dimensional structure of the object can be accurately and efficiently restored from the two-dimensional image, thereby improving the efficiency of three-dimensional shape reconstruction and saving manpower and material resources.

[0006] According to some embodiments of the present invention, the feature extractor is composed of a plurality of densely connected blocks and a deformable convolution module.

[0007] According to some embodiments of the present invention, the following parameters of each of the densely connected blocks are configured: the number of channels of each convolutional layer participating in the dense connection in the densely connected block, the number of channels of the feature map output by the densely connected block, and the number of convolutional layers participating in the dense connection in the densely connected block.

[0008] According to some embodiments of the present invention, the deformable convolution module includes a main line and a first learning branch, wherein the first learning branch learns the image area where the weighted summation of each convolution is applied to predict an offset vector for each point of the corresponding window in the convolution operation of the main line.

[0009] According to some embodiments of the present invention, the feature information obtained by the feature extractor is subjected to a reshape operation so that the feature information can be converted into voxels of a three-dimensional shape.

[0010] According to some embodiments of the present invention, the binarization processing of each element in the confidence three-dimensional tensor based on a preset threshold includes: if the element is smaller than the preset threshold, setting the value corresponding to the element to a first value, and the first value represents a grid in a blank state in the voxel; if the element is larger than the preset threshold, setting the value corresponding to the element to a second value, and the second value represents a grid in an occupied state in the voxel.

[0011] According to some embodiments of the present invention, the object shape reconstruction network is trained by a first loss function, where the first loss function is a weighted sum of a binary cross entropy loss function and a dice loss.

[0012] In a second aspect, an embodiment of the present invention provides a device for reconstructing the three-dimensional shape of an object, comprising: a normalization module for normalizing a two-dimensional image including a single object to obtain a normalized image; a reconstruction module for inputting the normalized image into an object shape reconstruction network, obtaining corresponding feature information through a feature extractor of the object shape reconstruction network, and inputting the feature information into a generator of the object shape reconstruction network to obtain a three-dimensional tensor of confidence of the predicted voxels of the object; and a binarization module for binarizing each element in the three-dimensional confidence tensor based on a preset threshold to obtain voxels representing the three-dimensional shape of the object.

[0013] The device for reconstructing the three-dimensional shape of an object according to an embodiment of the present invention has at least the following beneficial effects: by normalizing a two-dimensional image including a single object and inputting the image into an object shape reconstruction network, the object shape network is trained, and a three-dimensional system tensor of the confidence of the predicted voxels is obtained through the trained object shape reconstruction network, and the corresponding voxels are obtained after binarization, so that the three-dimensional structure of the object can be accurately and efficiently restored from the two-dimensional image, thereby improving the efficiency of three-dimensional shape reconstruction and saving manpower and material resources.

[0014] In a third aspect, an embodiment of the present invention provides a device comprising a processor and a memory coupled to the processor, wherein the memory stores program instructions executable by the processor, and when the processor executes the program instructions stored in the memory, the method for reconstructing the three-dimensional shape of an object described in the first aspect is implemented.

[0015] In a fourth aspect, an embodiment of the present invention provides a storage medium having program instructions stored therein, wherein the program instructions, when executed by a processor, can implement the method for reconstructing the three-dimensional shape of an object as described in the first aspect.

[0016] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 It is a flowchart of the steps of the method according to an embodiment of the present invention.

[0019] Figure 2 is a schematic diagram of a process for reconstructing a three-dimensional shape of an object in an embodiment of the present invention;

[0020] Figure 3 2 is a schematic diagram of the network architecture of an object shape reconstruction network according to an embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of a network architecture of densely connected blocks in an embodiment of the present invention;

[0022] Figure 5 Schematic diagram of the network architecture of deformable convolution in an embodiment of the present invention;

[0023] Figure 6 is a schematic diagram of the internal modules of the device in an embodiment of the present invention;

[0024] Figure 7 is a schematic diagram of a device in an embodiment of the present invention;

[0025] Figure 8 is a schematic diagram of a storage medium in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. In the subsequent description, suffixes such as "module," "component," or "unit" used to represent elements are used solely to facilitate the description of the present invention and have no specific meaning in themselves. Therefore, "module," "component," or "unit" may be used interchangeably. "First," "second," and the like are used solely to distinguish technical features and should not be construed as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In this subsequent description, the consecutive numbering of method steps is for ease of review and understanding. In conjunction with the overall technical solution of the present invention and the logical relationship between the various steps, adjusting the order of implementation of the steps does not affect the technical effects achieved by the technical solution of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and should not be construed as limiting the present invention.

[0027] Example 1

[0028] Reference Figure 1 This embodiment discloses a method for reconstructing the three-dimensional shape of an object, including steps S100 to S300.

[0029] Step S100 , normalizing a two-dimensional image including a single object to obtain a normalized image.

[0030] Specifically, a method of acquiring a two-dimensional image including a single object may be, for example, photographing the single object, or segmenting a two-dimensional image including multiple objects so that one two-dimensional image only includes a single object.

[0031] Reference Figure 2 , a two-dimensional image (i.e., a 2D image) containing a single object is first normalized, that is, the grayscale value of the image (e.g., 0-255) is uniformly mapped to between 0-1 for processing by the neural network.

[0032] In step S200 , the normalized image is input into the object shape reconstruction network, corresponding feature information is obtained through the feature extractor of the object shape reconstruction network, and input into the generator of the object shape reconstruction network to obtain a three-dimensional tensor of confidence of the predicted voxels of the object.

[0033] Specifically, the object shape reconstruction network is as follows: Figure 2 The feature extractor and generator are included.

[0034] The feature extractor, consisting of multiple densely connected blocks and a deformable convolutional module, is essentially a combination of convolutional layers, batch normalization layers, activation function layers, and pooling layers through a rich set of connections. The feature extractor operates on a 2D image containing a single object, extracting the feature information corresponding to the object in the image through layer-by-layer compression and abstraction.

[0035] The generator, consisting of several deconvolutional layers, batch normalization layers, and activation function layers, gradually analyzes the feature information extracted from the image, ultimately obtaining a three-dimensional tensor representing the confidence of the predicted voxels of the object. Each entry in this three-dimensional confidence tensor has a value between 0 and 1, representing the predicted confidence value of the occupied state of each grid in the voxel representation of the object's 3D shape.

[0036] In step S300 , based on a preset threshold, each element in the confidence three-dimensional tensor is binarized to obtain a voxel (short for volume element) representing the three-dimensional shape of the object.

[0037] Specifically, a preset threshold value d (0≤d≤1) is set. Elements in the confidence three-dimensional tensor with values ​​less than the preset threshold d are set to a first value (e.g., 0), representing a grid with an empty voxel. Elements in the confidence three-dimensional tensor with values ​​less than the preset threshold d are set to a second value (e.g., 1), representing a grid with an occupied voxel. This yields voxels that ultimately represent the object's three-dimensional shape. Using these voxels, the object's three-dimensional shape can be reconstructed.

[0038] An example of a specific network architecture of the object shape reconstruction network in this embodiment is as follows: Figure 3 First, the input two-dimensional image is used to extract the corresponding object-related feature information through the feature extractor. In order to ensure better information representation capabilities, two special network structures are used in this embodiment: densely connected blocks (equivalent to Figure 3 Dense block in ), deformable convolution module.

[0039] Densely connected blocks will establish more complex connections between convolutional layers, so that convolutional layers participating in dense connections will have direct skipping connections with each other. Densely connected blocks can alleviate gradient diffusion during training to obtain a better representation module. At the same time, since a large number of feature maps are reused, the number of parameters in the network model can be effectively reduced. For example Figure 3 The Dense block in the block only needs to configure three parameters: the number of channels k of each convolutional layer participating in the dense connection in this dense connection block, the number of channels c of the final output feature map of this dense connection block, and the number n of convolutional layers participating in the dense connection in this dense connection block. For example, Figure 3 In the Dense block 2D(32,128,6), k is 32, c is 128, and n is 6. Figure 3 Refer to the example diagram of the network architecture of the densely connected block (Dense block (k,c,n)) in Figure 4 .

[0040] The deformable convolution module of this embodiment is compared with the general convolution layer. Figure 5 , using a first branch to learn the image area affected by each convolution weighted summation, thereby breaking the regular windowing rule, so that the network pays more attention to the effective area in the image, while also adaptively reducing the interference of redundant information on the results. For the target task, the network only needs to pay attention to the information of the object part, and the background is equivalent to the noise that interferes with the prediction. Therefore, the use of deformable convolution can effectively enhance the ability of feature expression. Figure 5 As shown, in the deformable convolution module of this embodiment, a branch (i.e., the first learning branch) predicts an offset vector for each point in the window corresponding to each pixel in the convolution operation of the main line to achieve flexibility in the focus area. Figure 5 In the example, a window of 3×3 is used. Since the input is a two-dimensional image, the offset of each point is an offset vector including two directions.

[0041] In this embodiment, the feature extractor performs a reshape operation on the acquired feature information so that the feature information corresponding to the two-dimensional image can be converted into voxels of a three-dimensional shape in the neural network.

[0042] The feature information is fed into the generator. Through deconvolution-based computational blocks, the feature information is gradually decoded and upsampled, ultimately generating a three-dimensional tensor of the predicted voxel's confidence. To ensure that the confidence of each voxel is between 0 and 1, the activation function of the final layer, for example, is a sigmoid function.

[0043] To better train the reconstruction model, this embodiment uses a binary crossentropy loss function and a dice loss function to jointly supervise the training process, focusing on the accuracy of single voxel grid classification and overall morphology, respectively, to comprehensively measure the local and global restoration effects. The total loss function (equivalent to the first loss function) is the weighted sum of the two parts:

[0044] L total =λ1L bce +λ2L dice

[0045] Among them, Lbce Represents the value of binary cross entropy loss, L dice represents the dice loss (the value of dice loss), λ1 and λ2 represent two weights respectively, L total is the total loss value.

[0046] L bce The formula is:

[0047]

[0048] L dice The formula is:

[0049]

[0050] Among them, gt i Represents the state value of the grid point in the ground truth, p i represents the confidence value of the network's predicted grid points, and N represents the side length of the model that generates the voxel.

[0051] This embodiment normalizes a two-dimensional image including a single object and inputs it into an object shape reconstruction network, trains the object shape network, and obtains a three-dimensional system tensor of the confidence of the predicted voxels through the trained object shape reconstruction network. After binarization, the corresponding voxels are obtained. This can accurately and efficiently restore the three-dimensional structure of the object from the two-dimensional image, improve the efficiency of three-dimensional shape reconstruction, and save manpower and material resources.

[0052] Example 2

[0053] Reference Figure 6 This embodiment discloses a device 600 for reconstructing a three-dimensional shape of an object, including: a normalization module 610 , a reconstruction module 620 and a binarization module 630 .

[0054] The normalization module 610 normalizes the two-dimensional image including a single object to obtain a normalized image.

[0055] The reconstruction module 620 inputs the normalized image obtained by the normalization module 610 into the object shape reconstruction network, obtains the corresponding feature information through the feature extractor of the object shape reconstruction network, and inputs it into the generator of the object shape reconstruction network to obtain the three-dimensional confidence tensor of the predicted voxels of the object, and inputs it into the binarization module 630.

[0056] The binarization module 630 is configured to perform binarization processing on each element in the confidence three-dimensional tensor based on a preset threshold value to obtain voxels representing the three-dimensional shape of the object.

[0057] Similar to Example 1, this embodiment normalizes a two-dimensional image including a single object and inputs it into an object shape reconstruction network, trains the object shape network, and obtains a three-dimensional system tensor of confidence of the predicted voxels through the trained object shape reconstruction network. After binarization, the corresponding voxels are obtained, which can accurately and efficiently restore the three-dimensional structure of the object from the two-dimensional image, improve the efficiency of three-dimensional shape reconstruction, and save manpower and material resources.

[0058] Example 3

[0059] Reference Figure 7 This embodiment provides a device including a processor 810 and a memory 820 coupled to the processor 810. The memory 820 stores program instructions executable by the processor 810. When the processor 810 executes the program instructions stored in the memory 820, the method for reconstructing the three-dimensional shape of an object in Example 1 is implemented. The processor 810 may also be referred to as a CPU (Central Processing Unit). The processor 810 may be an integrated circuit chip with signal processing capabilities. The processor 810 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor, or any conventional processor. The memory 820 may include various components (e.g., machine-readable media), including but not limited to random access memory components, read-only components, and any combination thereof. The memory 820 may also include instructions (e.g., software) (e.g., stored on one or more machine-readable media); the instructions implement the method of the embodiment of the present invention.

[0060] It should be understood that, in order to avoid redundancy, the contents not covered in this embodiment may refer to embodiment 1. This embodiment has at least the same beneficial effects as embodiment 1.

[0061] Example 4

[0062] Reference Figure 8 This embodiment provides a storage medium storing program instructions 910. When executed by a processor, program instructions 910 implement the method for reconstructing the three-dimensional shape of an object described in Example 1. It should be understood that, to avoid redundancy, details not covered in this embodiment may be referred to in Example 1. This embodiment has at least the same beneficial effects as Example 1.

[0063] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above and the functional modules / units in the devices may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0064] In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium (storage medium for short), which may include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those of ordinary skill in the art, the term computer-readable medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0065] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention shall fall within the scope of the present invention.

Claims

1. A method for reconstructing the three-dimensional shape of an object, characterized in that: include: Normalizing the two-dimensional image including the single object to obtain a normalized image; Inputting the normalized image into an object shape reconstruction network, obtaining corresponding feature information through a feature extractor of the object shape reconstruction network, and inputting the feature information into a generator of the object shape reconstruction network to obtain a three-dimensional tensor of confidence of the predicted voxels of the object; Based on a preset threshold, binarize each element in the confidence three-dimensional tensor to obtain a voxel representing the three-dimensional shape of the object; The feature extractor consists of multiple densely connected blocks and a deformable convolution module.

2. The method for reconstructing the three-dimensional shape of an object according to claim 1, wherein: It also includes configuring the following parameters of each of the densely connected blocks: the number of channels of each convolutional layer participating in the dense connection in the densely connected block, the number of channels of the feature map output by the densely connected block, and the number of convolutional layers participating in the dense connection in the densely connected block.

3. The method for reconstructing the three-dimensional shape of an object according to claim 1, wherein: The deformable convolution module includes a main line and a first learning branch, wherein the first learning branch learns the image area where the weighted sum of each convolution is applied to predict an offset vector for each point of the corresponding window in the convolution operation of the main line.

4. The method for reconstructing the three-dimensional shape of an object according to claim 1, wherein: The feature information obtained by the feature extractor is subjected to a reshape operation so that the feature information can be converted into voxels of a three-dimensional shape.

5. The method for reconstructing the three-dimensional shape of an object according to claim 1, wherein: The binarization processing of each element in the confidence three-dimensional tensor based on a preset threshold comprises: If the element is smaller than the preset threshold, setting the value corresponding to the element to a first value, where the first value represents a blank grid in the voxel; If the element is greater than the preset threshold, the value corresponding to the element is set to a second value, where the second value represents an occupied grid in the voxel.

6. The method for reconstructing the three-dimensional shape of an object according to claim 1, wherein: The object shape reconstruction network is trained using a first loss function, where the first loss function is a weighted sum of a binary cross entropy loss function and a dice loss.

7. A device for reconstructing the three-dimensional shape of an object, characterized in that: A method for reconstructing the three-dimensional shape of an object according to any one of claims 1 to 6, comprising: a normalization module, configured to normalize a two-dimensional image including a single object to obtain a normalized image; a reconstruction module, configured to input the normalized image into an object shape reconstruction network, obtain corresponding feature information through a feature extractor of the object shape reconstruction network, and input the feature information into a generator of the object shape reconstruction network to obtain a three-dimensional tensor of confidence of the predicted voxels of the object; The binarization module is used to perform binarization processing on each element in the confidence three-dimensional tensor based on a preset threshold value to obtain a voxel representing the three-dimensional shape of the object.

8. A device for reconstructing a three-dimensional shape of an object, comprising a processor and a memory coupled to the processor, wherein the memory stores program instructions executable by the processor, wherein: When the processor executes the program instructions stored in the memory, the method for reconstructing the three-dimensional shape of an object as described in any one of claims 1 to 6 is implemented.

9. A storage medium storing program instructions, wherein the program instructions, when executed by a processor, can implement the method for reconstructing the three-dimensional shape of an object as claimed in any one of claims 1 to 6.

Images