Edge- and semantic-guided 3D object reconstruction methods and systems
By using edge- and semantically guided 3D object reconstruction methods, the problems of geometric contour loss and semantic confusion in single-view 3D reconstruction technology under complex environments are solved, achieving high-precision 3D object reconstruction and improving the robustness and surface smoothness of the model.
Patent Information
- Application Number
- CN202611417593.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-14
- Publication Date
- 2026-11-17
AI Technical Summary
Existing single-view 3D reconstruction technologies are prone to losing geometric contours, confusing semantic categories of objects, and reconstructing slender structures when faced with sudden changes in lighting, occlusion, and complex backgrounds. Furthermore, the surface quality of the models is poor, making it difficult to meet the needs of practical applications.
We employ an edge- and semantically guided 3D object reconstruction method. By integrating edge prior information and hierarchical semantic constraints through an edge extraction network, a semantically aware decoder, and a dynamic deformation optimizer, we optimize the traditional encoder-decoder architecture, enhance feature extraction and semantic matching capabilities, and improve detail reconstruction and model surface quality.
It effectively reconstructs the features of occluded areas and slender structures, reduces detail breaks and artifacts, improves the accuracy and robustness of 3D reconstruction, and outputs high-fidelity, high-precision object geometry.
Smart Images

Figure CN123118917A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer graphics technology, and in particular to a method and system for reconstructing three-dimensional objects based on edge and semantic guidance. Background Technology
[0002] In emerging technology fields such as metaverse digital asset generation, autonomous driving scene perception, and embodied intelligent environment interaction, the technology of reconstructing the three-dimensional geometric structure of objects based on a single two-dimensional image is a key research direction in the current field of 3D vision reconstruction. Currently, mainstream single-view 3D reconstruction technologies generally adopt an encoder-decoder architecture, directly predicting the occupancy probability of a three-dimensional (3D) voxel mesh through a network model to reconstruct the object's three-dimensional structure. However, in the application of complex real-world scenarios, the aforementioned traditional single-view 3D reconstruction schemes still have many inherent technical defects, and their specific technical bottlenecks are reflected in the following three aspects:
[0003] First, existing reconstruction models have a single dimension of information input, relying solely on RGB (red, green, blue) color features to complete feature extraction and reconstruction inference. When faced with complex conditions such as sudden changes in lighting, extreme shadow occlusion, and messy background textures, the models are prone to over-reliance on local pixel information, falling into local pixel traps, resulting in the loss of the true geometric contours and boundary features of the reconstructed object, and a significant reduction in reconstruction integrity and accuracy.
[0004] Secondly, existing networks can only achieve basic perception of local image semantics and lack high-order cognitive ability of global object categories. For objects such as tables and chairs with similar geometric outlines but completely different semantic categories, the topological structures of objects are prone to intersection and confusion. As a result, the reconstruction results can only replicate the surface shape of the object and cannot match the real semantic structure of the object, resulting in the distortion defect of "similar in form but not in spirit".
[0005] Finally, the feature extraction and loss constraint mechanisms of existing models have shortcomings. On the one hand, the receptive field of traditional 3D convolution is fixed and rigid, making it difficult to adaptively capture and associate slender local topological structures such as gun barrels and furniture support legs, which easily leads to missing or broken reconstructions of fine structures. On the other hand, existing reconstruction models mostly use binary cross-entropy or intersection-over-union (IoU) as optimization constraints, focusing only on the matching accuracy of voxels and failing to introduce constraints on the physical smoothness of the 3D model surface. This results in the final reconstructed 3D model having obvious step-like and jagged artifacts, which are difficult to adapt to the application requirements of actual 3D rendering. Summary of the Invention
[0006] This application aims to mitigate the shortcomings of existing technologies in reconstructing 3D objects from single-view RGB images, such as loss of geometric contours, cross-category shape confusion, breakage of slender topological structures, and poor model surface quality, in order to achieve high-fidelity and robust 3D object reconstruction.
[0007] According to a first aspect of this application, a method for reconstructing three-dimensional objects based on edge and semantic guidance is provided, comprising: acquiring a single-view RGB image of a three-dimensional object; inputting the single-view RGB image into a trained three-dimensional object reconstruction network to obtain a target three-dimensional voxel model of the three-dimensional object. The 3D object reconstruction network comprises a feature fusion encoder for edge guidance, a multi-level alignment decoder for semantic awareness, and a dynamic deformation optimizer for topology adaptation, connected sequentially. The feature fusion encoder includes an edge extraction network, an RGB image feature extraction branch, an edge image feature extraction branch, and a residual space attention module. The edge extraction network generates corresponding high-frequency edge images based on the input single-view RGB image, and the edge image feature extraction branch generates an edge image feature tensor based on the high-frequency edge images. The RGB image feature extraction branch generates the original image feature tensor based on the single-view RGB image. The residual spatial attention module is based on the original image feature tensor and the edge image feature tensor Generate two-dimensional fusion features The multi-level alignment decoder performs a fusion of the two-dimensional features. Perform up-dimensional decoding to generate a coarse three-dimensional voxel model. The dynamic deformation optimizer optimizes the rough 3D voxel model. The model is refined to output the target 3D voxel model of the 3D object. .
[0008] Preferably, the edge extraction network includes multiple cascaded extraction modules and a stitching module connected to each extraction module. Each extraction module includes a multi-scale convolutional flow submodule and a corresponding upsampling alignment submodule. The multi-scale convolutional flow submodule includes cascaded downsampling layers and densely connected convolutional blocks. The downsampling layers are used to reduce the spatial resolution of the input features, and the densely connected convolutional blocks are used to extract image features at the current scale. Each multi-scale convolutional flow submodule outputs a multi-scale image feature tensor. The upsampling alignment submodule uses deconvolution to upsample and align the corresponding image feature tensors at each scale to the size of the single-view RGB image. The stitching module stitches the aligned image feature tensors together in the channel dimension and outputs a single-channel high-frequency edge image through a single-layer convolution. The RGB image feature extraction branch is a 16-layer visual geometry group convolutional neural network. The structure of the edge image feature extraction branch is symmetrical to the structure of the RGB image feature extraction branch. The first convolutional layer of the 16-layer convolutional neural network of the visual geometry group has 1 input channel, and the weights of the convolutional kernel of the first convolutional layer are initialized to the arithmetic mean of the original pre-trained weights in the input channel dimension.
[0009] Preferably, the residual spatial attention module is based on the original image feature tensor. and the edge image feature tensor Generate two-dimensional fusion features This includes: using the edge image feature tensor Using the spatial attention layer as input, a two-dimensional spatial weight map is obtained. The spatial attention layer comprises cascaded 1×1 convolutional layers with unbiased terms and a sigmoid activation function, wherein... Represents a 1×1 convolutional layer function; the two-dimensional spatial weight map With the original image feature tensor Element-wise multiplication and then with the original image feature tensor Perform residual jump aggregation to obtain two-dimensional fused features. , ,in This indicates element-wise multiplication. During element-wise multiplication, the single-channel two-dimensional spatial weight map is used. Broadcast along the channel dimension to the original image feature tensor Same number of channels.
[0010] Preferably, during the training phase of the 3D object reconstruction network, the edge extraction network performs synchronous random erasure processing on the input sample data: the input sample data includes a single-view RGB image and a corresponding high-frequency edge image; the overlapping area of the single-view RGB image and the corresponding high-frequency edge image is randomly occluded with a preset trigger probability; a rectangular area with an area accounting for a preset percentage of the single-view RGB image is randomly generated in the overlapping area, wherein the preset percentage ranges from 2% to 15%; the rectangular area is filled with pixels of a preset grayscale value, and the area in the high-frequency edge image corresponding to the rectangular area is filled with pixels of a grayscale value of 0.
[0011] Preferably, the dynamic deformation optimizer includes a cascaded initial 3D feature extraction module, a 3D convolutional spatial offset prediction module, a 3D trilinear interpolation resampling module, a standard 3D convolutional thinning layer, a feature normalization and activation layer, a multi-level stacked 3D convolutional layer, and a feature mapping module, wherein: the initial 3D feature extraction module is used to perform 3D convolution and pooling processing on the coarse 3D voxel model to obtain a 3D feature tensor at the current scale; the input of the 3D convolutional spatial offset prediction module is the 3D feature tensor at the current scale, and the 3D convolutional spatial offset prediction module is used to perform a 3D convolution operation with a kernel size of 3×3×3, a padding size of 1, and an output channel number of 3, outputting a 3D offset field tensor of the 3D spatial coordinates of the object surface. The three-dimensional trilinear interpolation resampling module uses the current-scale three-dimensional feature tensor and the adaptive deformation sampling mesh. The output is a synchronous input, and a differentiable three-dimensional trilinear interpolation operation is performed to output reconstructed features, the adaptive deformable sampling mesh. The expression is ,in, The standard three-dimensional normalized base sampling grid represents the spatial coordinate distribution within the range [-1, 1], and the three-dimensional offset field tensor is... The output of the 3D convolutional spatial offset prediction module is h, which is a preset deformation adjustment factor. The standard 3D convolutional thinning layer is used to perform feature dimension transformation on the reconstructed features output by the 3D trilinear interpolation resampling module. The feature normalization and activation layer includes cascaded 3D batch normalization operators and nonlinear activation functions. The multi-level stacked 3D convolutional layer includes downsampling feature extraction units, fully connected bottleneck units, and upsampling reconstruction units, and fuses 3D features of the same spatial scale through skip connections to perform multi-scale spatial feature extraction and resolution restoration on the reconstructed features. The multi-level stacked 3D convolutional layer performs spatial feature extraction and dimensionality upsampling on the output of the feature normalization and activation layer. The feature mapping module includes cascaded single-layer 3D deconvolution and Sigmoid activation functions to map the feature tensors output by the multi-level stacked 3D convolutional layer to occupancy probability values between 0 and 1, thereby obtaining a refined 3D voxel model. and the refined three-dimensional voxel model With rough 3D voxel model Equal weight fusion is performed to finally output the target 3D voxel model. ,in, .
[0012] Preferably, the multi-level aligned decoder includes a cascaded 3D spatial feature reconstruction layer, a first 3D deconvolution decoding layer, a second 3D deconvolution decoding layer, a third 3D deconvolution decoding layer, a fourth 3D deconvolution decoding layer, a 3D feature refinement layer, and an occupancy probability mapping layer, wherein: the 3D spatial feature reconstruction layer modifies the input 2D fused features through tensor reshaping operations. The mapping is to an initial 3D voxel feature, the initial 3D voxel feature having a spatial resolution of 2×2×2 and 2048 channels; the first 3D deconvolution decoding layer has a convolution kernel size of 4×4×4 and 512 channels; the second 3D deconvolution decoding layer has a convolution kernel size of 4×4×4 and 128 channels; the third 3D deconvolution decoding layer has a convolution kernel size of 4×4×4 and 64 channels; the fourth 3D deconvolution decoding layer has a convolution kernel size of 4×4×4 and 32 channels; the 3D feature refinement layer includes convolution... The system comprises a 3D convolutional layer with a kernel size of 3×3×3, a padding size of 1, 32 input channels, and 8 output channels, and a 3D batch normalization layer and a nonlinear activation function connected sequentially to the 3D convolutional layer; the occupancy probability mapping layer includes a 3D convolutional layer with a kernel size of 1×1×1, 8 input channels, and 1 output channel, and a Sigmoid activation function connected to the 3D convolutional layer, used to map the feature tensor output by the 3D feature refinement layer to voxel occupancy probabilities between 0 and 1, thereby obtaining a coarse 3D voxel model.
[0013] Preferably, during the training phase of the 3D object reconstruction network, the optimization objective is to minimize the mixed total loss. : ;in, This represents the hierarchical semantic alignment constraint loss; 3D focus loss. Dice loss Laplace smoothing loss ;in, This represents the total number of voxels within a single 3D voxel grid. Indicates the voxel point index; This indicates that the 3D object reconstruction network predicts the first... The probability that an individual element is an entity. Indicates the first The true voxel label at the location corresponding to the voxel, and ; Indicates the first The probability that an individual element is correctly predicted, and: ;when hour, ;when hour, ; Indicates based on real voxel labels Selected preset asymmetric category balance weights; This indicates the preset focus attenuation parameter; This indicates a preset numerical smoothing term used to prevent division by zero; This represents a preset 3D discrete Laplacian convolution kernel used to extract local second-order discrete difference features of the target 3D voxel model; This represents the computation of discrete convolution in three-dimensional space. Describes the norm 1. This represents the mean absolute value of the calculation results for all voxel positions.
[0014] Preferably, the hierarchical semantic alignment constraint loss ;in, For single-layer semantic projection loss The value of s when it is 8 represents the single-layer semantic projection loss corresponding to the second three-dimensional deconvolution decoding layer, and the scale identifier of the second three-dimensional deconvolution decoding layer is 8; For single-layer semantic projection loss The value of s when s is 16 The single-layer semantic projection loss corresponding to the third 3D deconvolutional decoding layer is represented by 16, and s represents the scale identifier.
[0015] ;
[0016] in, The size of the current training batch. This refers to the sample index within the current training batch. The intermediate layer 3D semantic vector is obtained by passing the output of the 3D deconvolution decoding layer with sample index b and scale identifier s through a 3D semantic projection head. for With deep text semantic vectors Cosine similarity between them for With image feature vectors Cosine similarity between them; The sample index within the current training batch is The loss weighting coefficient, This is a preset numerical stability term used to prevent the denominator from being zero, and >0; wherein, the three-dimensional semantic projection head includes a cascaded three-dimensional adaptive average pooling layer, a flattening layer and a fully connected linear layer. The three-dimensional adaptive average pooling layer is used to compress the spatial dimension of the input features of the three-dimensional semantic projection head to 1×1×1. After the output of the three-dimensional adaptive average pooling layer is processed by the flattening layer, the fully connected linear layer maps the output of the flattening layer to a 512-dimensional feature vector space.
[0017] According to a second aspect of this application, a three-dimensional object reconstruction system based on edge and semantic guidance is provided, comprising a single-view RGB image acquisition unit and a three-dimensional object reconstruction network, wherein the single-view RGB image acquisition unit is used to acquire a single-view RGB image of a three-dimensional object; the three-dimensional object reconstruction network comprises a feature fusion encoder for edge guidance, a multi-level alignment decoder for semantic awareness, and a dynamic deformation optimizer for topology adaptation, connected in sequence, wherein the feature fusion encoder comprises an edge extraction network, an RGB image feature extraction branch, an edge image feature extraction branch, and a residual space attention module, wherein the edge extraction network is used to generate a corresponding high-frequency edge image using the single-view RGB image as input; the RGB image feature extraction branch is used to generate an original image feature tensor using the single-view RGB image as input. The edge image feature extraction branch is used to generate an edge image feature tensor using the high-frequency edge image as input. The residual spatial attention module is used to perform the following based on the original image feature tensor: and the edge image feature tensor Generate two-dimensional fusion features The multi-level alignment decoder is used for the two-dimensional fusion features. Perform up-dimensional decoding to generate a coarse three-dimensional voxel model. The dynamic deformation optimizer is used to optimize the rough 3D voxel model. The model is refined to generate a target 3D voxel model of the 3D object. .
[0018] According to a third aspect of this application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of the first aspect of this application.
[0019] According to a fourth aspect of this application, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method of the first aspect of this application.
[0020] Compared with the prior art, this application mainly has the following advantages:
[0021] This application proposes an edge and semantic guided 3D object reconstruction framework, ESGNet (Edge and Semantic Guided Network). Addressing the inherent limitations of traditional 3D object reconstruction networks, it integrates edge prior information, hierarchical semantic alignment constraints, and deformable convolution optimization strategies to optimize and upgrade the traditional encoder-decoder reconstruction architecture, overcoming shortcomings in feature extraction, semantic matching, detail reconstruction, and model optimization. Compared to traditional reconstruction schemes, this application can effectively mine feature information from occluded regions and slender topological structures, providing usable feature support for the 3D reconstruction of slender and easily missing components such as gun barrels and furniture support legs, as well as occluded regions. This helps alleviate the problems of broken slender structures and distortion in occluded region reconstruction found in existing models. Simultaneously, through multi-dimensional optimization constraints, it effectively suppresses cluttered noise on the 3D voxel surface, improving defects such as stepped and jagged artifacts present in traditional models, enhancing the surface smoothness and integrity of the 3D reconstruction model, thereby outputting high-fidelity, high-precision 3D geometric structures of objects. This application can mine the potential topological features of the three-dimensional shape of an object, improve the shortcomings of the traditional model's ability to perceive local features, effectively establish the relationship between image features, text semantics and three-dimensional shape, improve the accuracy, robustness and generalization ability of single-view three-dimensional reconstruction, and be better applicable to the three-dimensional reconstruction and rendering needs in different scenarios. Attached Figure Description
[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0023] Figure 1 A schematic diagram of the architecture of the 3D object reconstruction network in the edge- and semantically guided 3D object reconstruction method provided in the embodiments of this application;
[0024] Figure 2 This is a flowchart illustrating the edge- and semantically guided 3D object reconstruction method provided in this application embodiment. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Furthermore, the technical features involved in the various embodiments described below can be combined with each other as long as they do not conflict with each other.
[0026] The following will describe in detail, with reference to the accompanying drawings, a method for reconstructing 3D objects based on edge and semantic guidance according to embodiments of this application. The method uses a 3D object reconstruction network, which employs the ESGNet framework, such as... Figure 1 As shown, the 3D object reconstruction network includes an edge-guided feature fusion encoder, a semantically aware multi-level alignment decoder, and a topology-adaptive dynamic deformation optimizer. The edge-guided feature fusion encoder includes an edge extraction network, an RGB image feature extraction branch, an edge image feature extraction branch, and a residual space attention module. Figure 2 This is a flowchart illustrating the edge- and semantically guided 3D object reconstruction method provided in this application embodiment. The following is in conjunction with... Figure 1 and Figure 2 The embodiments of this application are described in detail.
[0027] like Figure 1 and Figure 2 As shown, the method includes:
[0028] Step S101: Obtain a single-view RGB image of the 3D object;
[0029] Step S102: Using a single-view RGB image as input, an edge extraction network is used to generate the corresponding high-frequency edge image;
[0030] Step S103: Using a single-view RGB image as input, generate the original image feature tensor using the RGB image feature extraction branch. And using high-frequency edge images as input, the edge image feature extraction branch is used to generate edge image feature tensors. ;
[0031] Step S104: Use the residual space attention module based on the original image feature tensor and edge image feature tensor Generate two-dimensional fusion features ;
[0032] Step S105: Use a semantically aware multi-level alignment decoder to process the two-dimensional fused features. Perform up-dimensional decoding to generate a coarse three-dimensional voxel model. ;
[0033] Step S106: Use a topology-adaptive dynamic deformation optimizer to optimize the rough 3D voxel model. Refine the model to generate a target 3D voxel model of the 3D object. .
[0034] In some embodiments, step S101 may be receiving a single-view RGB image sent by a camera, reading from an image database, or receiving a single-view RGB image uploaded by a user.
[0035] In some embodiments, in step S102, a single-view RGB image is input into an edge extraction network to generate a corresponding high-frequency edge image. The edge extraction network includes six cascaded extraction modules and a stitching module connected to the six extraction modules. Each extraction module includes a multi-scale convolutional flow submodule and a corresponding upsampling alignment submodule. The multi-scale convolutional flow submodule includes cascaded downsampling layers and densely connected convolutional blocks. The downsampling layers reduce the spatial resolution of the input features (i.e., the features input to the multi-scale convolutional flow submodules), and the densely connected convolutional blocks extract image features at the current scale. Each multi-scale convolutional flow submodule outputs a multi-scale image feature tensor. The upsampling alignment submodule upsamples and aligns the image feature tensor output by the corresponding multi-scale convolutional flow submodule to the size of the original single-view RGB image by deconvolution. The stitching module stitches the aligned multiple image feature tensors in the channel dimension, and then outputs the final single-channel high-frequency edge image through a single-layer convolution. Using the above structure, the edge extraction network can accurately capture the global contour and local high-frequency details of an object from a multi-level receptive field, and output a denoised high-frequency edge image.
[0036] To enhance the geometric reasoning ability of the edge extraction network for incomplete and occluded shapes, the edge extraction network performs synchronous random erasure processing on the input sample data during the training phase of the 3D object reconstruction network. The input sample data includes a single-view RGB image and the corresponding high-frequency edge image. Specifically, the synchronous random erasure of the input sample data includes: randomly occluding the overlapping area of the single-view RGB image and the corresponding high-frequency edge image with a preset trigger probability (e.g., 0.5), that is, randomly generating a rectangular area in the overlapping area with a preset trigger probability, which accounts for a preset percentage of the area of the single-view RGB image, with the preset percentage ranging from [2%, 15%]; filling the rectangular area with pixels of a preset gray value to simulate local stains or occlusions of objects in the real world. In a specific embodiment, the preset gray value is 0.5; and filling the area in the high-frequency edge image corresponding to the rectangular area with pixels of a gray value of 0, artificially creating physical structural breaks, forcing the 3D reconstruction network to learn topological reasoning ability when boundary cues are incomplete, thereby improving the robustness of the overall reconstruction model. The preset percentage is preferably 15%, which can create obvious physical breakage without causing the complete erasure of slender structural features such as chair legs and gun barrels, thus preventing training failure.
[0037] In some embodiments, in step S103, the single-view RGB image is input to the RGB image feature extraction branch to generate the original image feature tensor. ; and input the high-frequency edge image generated by the edge extraction network into the edge image feature extraction branch to generate the edge image feature tensor. The RGB image feature extraction branch is a general VGG16 (VisualGeometry Group 16-layer Convolutional Neural Network). Specifically, it extracts the first 27 feature extraction layers of the general VGG16, including batch normalization. The single-view RGB image is processed by the above RGB image feature extraction branch to generate an original image feature tensor of size 256×8×8. The structure of the edge image feature extraction branch is completely symmetrical to that of the RGB image feature extraction branch, and it also uses VGG16. To adapt to the single-channel input of high-frequency edge images, the number of input channels in the first convolutional layer of VGG16 is changed from 3 channels to 1 channel, and the kernel weights of this first convolutional layer are initialized to the arithmetic mean of the original pre-trained weights along the input channel dimension. The high-frequency edge image generates a 256×8×8 edge image feature tensor through the aforementioned edge image feature extraction branch. .
[0038] In some embodiments, in step S104, the original image feature tensor is... and edge image feature tensor Input the Residual Spatial Attention (RSA) module to generate two-dimensional fused features. Specifically, step S104 includes: converting the edge image feature tensor... The input is fed into a spatial attention layer to obtain a two-dimensional spatial weight map. , The spatial attention layer consists of cascaded 1×1 convolutional layers and a sigmoid activation function; the 1×1 convolutional layers have no bias term; the two-dimensional spatial weight map... With the original image feature tensor Element-wise multiplication is performed, element-wise feature activation is applied, and the result is compared with the original image feature tensor. Perform residual jump aggregation to obtain two-dimensional fused features. , ,in This indicates element-wise multiplication. During element-wise multiplication, the single-channel two-dimensional spatial weight map is used. Broadcast along the channel dimension to the original image feature tensor The same number of channels. The residual spatial attention module includes a spatial attention layer, an element-wise multiplication calculation submodule, and a residual jump aggregation submodule. The spatial attention layer is used to extract edge image feature tensors. Spatial topological distribution (i.e.) Figure 1 Spatial attention recalibration in edge image, i.e., based on the edge image feature tensor Generate a two-dimensional spatial weight graph The element-wise multiplication calculation submodule is used to calculate the weight graph in two-dimensional space. With the original image feature tensor Element-wise multiplication is used to highlight high-frequency contour lines in the original single-view RGB image and suppress high-noise background impurities; the residual jump aggregation submodule is used to combine the output of the element-wise multiplication calculation submodule with... In addition, by introducing residual skip connection paths, the depth information and lighting texture within the original single-view RGB image are preserved, preventing the object's interior from being hollowed out. Through the aforementioned residual spatial attention module, a two-dimensional fused feature with highlighted boundaries and intact internal topology is finally obtained. .
[0039] In some embodiments, step S105 uses two-dimensional fusion features. The input to the semantically perceptual multi-level alignment decoder is used to map the features back to the three-dimensional physical space through successive three-dimensional deconvolutions of the multi-level alignment decoder, generating a coarse three-dimensional voxel model. Specifically, the semantic-aware multi-level alignment decoder includes a cascaded 3D spatial feature reconstruction layer, a first 3D deconvolution decoding layer, a second 3D deconvolution decoding layer, a third 3D deconvolution decoding layer, a fourth 3D deconvolution decoding layer, a 3D feature refinement layer, and an occupancy probability mapping layer. The 3D spatial feature reconstruction layer maps the input 2D fused features to initial 3D voxel features through tensor reshaping operations, and the spatial resolution of the initial 3D voxel features is 2×2×2 with 2048 channels. The first 3D deconvolution decoding layer has a convolution kernel size of 4×4×4 and 512 channels, the second 3D deconvolution decoding layer has a convolution kernel size of 4×4×4 and 128 channels, and the third 3D deconvolution decoding layer has a convolution kernel size of 4×4×4 and 128 channels. The kernel size of the fourth 3D deconvolution decoding layer is 4×4×4, and the number of channels is 64. The kernel size of the third 3D deconvolution decoding layer is 4×4×4, and the number of channels is 32. The 3D feature refinement layer includes a 3D convolutional layer with a kernel size of 3×3×3, a padding size of 1, 32 input channels and 8 output channels, as well as a 3D batch normalization layer and a non-linear activation function connected in sequence with the 3D convolutional layer. The occupancy probability mapping layer includes a 3D convolutional layer with a kernel size of 1×1×1, 8 input channels and 1 output channel, as well as a Sigmoid activation function connected with the 3D convolutional layer. This function is used to map the feature tensor output by the 3D feature refinement layer to voxel occupancy probabilities between 0 and 1, thus obtaining a coarse 3D voxel model.
[0040] In some embodiments, step S106 involves roughening the three-dimensional voxels. The model is input into a topology-adaptive dynamic deformation optimizer for refinement, generating an optimized 3D voxel model of the target 3D object. Specifically, the topology-adaptive dynamic deformation optimizer includes a cascaded initial 3D feature extraction module, a 3D convolutional spatial offset prediction module, a 3D trilinear interpolation resampling module, a standard 3D convolutional thinning layer, a feature normalization and activation layer, a multi-level stacked 3D convolutional layer, and a feature mapping module. The 3D convolutional spatial offset prediction module takes as input the current-scale 3D feature tensor output by the initial 3D feature extraction module. It employs a 3D convolution operation with a kernel size of 3×3×3, a padding size of 1, and 3 output channels, outputting a 3D offset field tensor of the object's surface spatial coordinates. In one specific embodiment, a coarse three-dimensional voxel model First, a 3D feature tensor at the current scale is obtained through 3D convolution and pooling, and then the normalized coordinate offset is predicted at scales of 16×16×16 and 8×8×8 respectively; deformation adjustment factor Out-of-boundary coordinates are filled with boundary values and aligned using corner points. Features at each scale are refined step-by-step through fully connected bottlenecks and skip connections to obtain a refined voxel model. The final output is
[0041] The 3D trilinear interpolation resampling module receives the current-scale 3D feature tensor and the adaptive deformation sampling mesh. The output is taken as synchronous input from both, and a differentiable 3D trilinear interpolation operation is performed to output reconstructed features, with an adaptive deformable sampling mesh. The expression is ,in, The standard three-dimensional normalized basic sampling grid represents the spatial coordinate distribution within the range of [-1, 1]. A preset deformation adjustment factor h is used to limit the maximum receptive field offset amplitude of a single sampling to prevent excessive distortion of the topology. The three-dimensional offset field tensor is adjusted by introducing the deformation adjustment factor. Linear action on standard 3D normalized base sampling grid On top of this, an adaptive deformable sampling mesh with topological adaptability can be reconstructed. Adaptive deformation sampling mesh The output is also used as the input of the three-dimensional trilinear interpolation resampling module, which enables the topology-adaptive dynamic deformation optimizer to adaptively stretch and wrap around easily broken detail areas (such as table legs, armrests, etc.), which helps to alleviate the topology breakage problem caused by the fixed receptive field in conventional convolution operations.
[0042] The standard 3D convolutional thinning layer is used to transform the feature dimension of the reconstructed features output by the 3D trilinear interpolation resampling module. A conventional standard 3D convolutional thinning layer can be used. The feature normalization and activation layer contains cascaded 3D batch normalization operators and nonlinear activation functions. The multi-level stacked 3D convolutional layer contains multiple stacked 3D convolutional modules. The number of 3D convolutional modules can be determined experimentally or empirically. Each 3D convolutional module includes cascaded 3D pooling layers and 3D deconvolutional layers. The multi-level stacked 3D convolutional layer is used to extract and upscale deep spatial features from the output of the feature normalization and activation layer. The feature mapping module contains cascaded single-layer 3D deconvolution and a sigmoid activation function, used to map the feature tensors output by the multi-level stacked 3D convolutional layer to occupancy probability values between 0 and 1, outputting a refined 3D voxel model. and compared it with a rough three-dimensional voxel model By performing equal-weighted fusion, we can obtain .
[0043] In some embodiments, during the training phase of the 3D object reconstruction network, the original input data for the training samples is a single-view RGB image (randomly erased during training). The supervisory data used to guide network optimization includes real 3D voxel labels and text semantic vectors and image feature vectors extracted by a pre-trained Contrastive Language-Image Pre-training (CLIP) model. During training and inference, the edge extraction network generates corresponding high-frequency edge images based on the single-view RGB images, and the edge-guided feature fusion encoder performs feature extraction and fusion using the single-view RGB images and high-frequency edge images as input. In one specific embodiment, high-frequency edge images can be pre-generated and stored using the edge extraction network, and directly read during training and inference. The high-frequency edge images are generated from the corresponding single-view RGB images and are not used as an independent data source. The text semantic vectors and image feature vectors extracted by the pre-trained CLIP are only used to calculate the hierarchical semantic alignment constraint loss during the training phase; during the inference phase, CLIP does not need to be invoked, nor do text prompts need to be input.
[0044] In some embodiments, the 3D object reconstruction network is trained to minimize the mixed total loss. To optimize the objective, ,in, 3D focus loss is used to measure the imbalance between the number of empty background voxels and the number of solid object voxels in a 3D voxel mesh. Dice loss is used to quantify the degree of spatial overlap between the 3D voxels predicted by the 3D object reconstruction network and the real labeled voxels. For hierarchical semantic alignment constraint loss; The Laplacian smoothing loss is used as a surface quality constraint term to optimize the microscopic surface quality of 3D voxel reconstruction, which helps to alleviate the problems of step-like and jagged artifacts in traditional voxelized 3D reconstruction.
[0045] In some embodiments, three-dimensional focus loss ,in, This represents the total number of voxels within a single 3D voxel grid. Indicates the voxel point index; This indicates that the 3D object reconstruction network predicts the first... The probability that an individual element is an entity. Indicates the first The true voxel label at the location corresponding to the voxel, and ; Indicates the first The probability that an individual element is correctly predicted, and: .when hour, ;when hour, . Indicates based on real voxel labels Selected preset asymmetric category balance weights; This indicates the preset focus attenuation parameter.
[0046] ,in, To prevent division by zero, a preset small numerical smoothing term is used.
[0047] , This represents a preset 3D discrete Laplacian convolution kernel used to extract second-order discrete difference features of local surface morphology (reflecting the curvature undulation of the local surface). This represents the computation of discrete convolution in three-dimensional space. This represents the absolute mean of the calculated results for all voxel positions. In one specific embodiment, the three-dimensional discrete Laplacian convolution kernel... The size is The convolution stride is 1, the padding size is 1, and the coefficient at the center position is the negative of the sum of the coefficients at the other positions, so that the sum of all coefficients of the convolution kernel is 0.
[0048] To overcome the limitations of traditional 3D object reconstruction networks that rely solely on local geometric features and lack higher-order semantic cognition, a Hierarchical Semantic Alignment Constraint (HSAC) mechanism is introduced during the training phase of the 3D object reconstruction network. This means that a hierarchical semantic alignment constraint loss is added to the optimization objective. (like Figure 1 As shown, this is also known as the cross-modal cosine similarity loss, which is a hierarchical semantic alignment constraint loss. The cosine similarity is calculated based on the cross-modal semantic features (i.e., image features and text features) extracted by the contrastive language-image pre-trained model CLIP, thus constructing the model.
[0049] ;
[0050] in, For single-layer semantic projection loss The value of s=8 represents the single-layer semantic projection loss corresponding to the second three-dimensional deconvolution decoding layer, and the scale identifier of the second three-dimensional deconvolution decoding layer is 8. For single-layer semantic projection loss The value of s when s is 16 This represents the single-layer semantic projection loss corresponding to the third 3D deconvolution decoding layer. The scale identifier of the third 3D deconvolution decoding layer is 16, and s represents the scale identifier.
[0051] In some embodiments, single-layer semantic projection loss :
[0052] ;
[0053] in, The size of the current training batch. This refers to the sample index within the current training batch. for With deep text semantic vectors Cosine similarity between them for With image feature vectors Cosine similarity between them The intermediate layer 3D semantic vector is obtained by passing the output of the 3D deconvolution decoding layer with sample index b and scale identifier s through a 3D semantic projection head. The sample index within the current training batch is The loss weight coefficients are used to indicate whether the sample participates in the calculation of the current semantic projection loss. In some embodiments, The value range is [0,1]; This is a preset numerical stability term used to prevent the denominator from being zero, and >0. The 3D semantic projection head is constructed using conventional operators in series, including a cascaded 3D adaptive average pooling layer, a flattening layer, and a fully connected linear layer. The 3D adaptive average pooling layer is used to compress the spatial dimension of the input features of the 3D semantic projection head to 1×1×1. After the output of the 3D adaptive average pooling layer is processed by the flattening layer, the fully connected linear layer maps the output of the flattening layer to a 512-dimensional feature vector space.
[0054] In some embodiments, deep text semantic vectors The text prompts are extracted by the text encoder of the contrastive language-image pre-trained model CLIP. These text prompts are dynamically constructed in a continuous feature embedding space. The specific construction process is as follows: First, the discrete text sequence containing the target category is transformed into an initial word embedding vector sequence. Then, the inherent word vectors corresponding to the target category in this initial word embedding vector sequence are retained. Next, the initial word embedding vectors at preset context positions in this initial word embedding vector sequence are replaced with independently initialized continuous learnable context vectors. This combination in the sequence dimension constitutes the final text prompt. This lightweight text prompt and context optimization strategy can avoid the computational overhead of full parameter fine-tuning while introducing a multimodal large model (i.e., the aforementioned CLIP) for generalization cognition. Furthermore, the continuous learnable context vector includes a shallow learnable cue vector set in the input layer of the text encoder, and deep learnable cue vectors set in at least one Transformer encoding layer. Before feature processing is performed on the corresponding Transformer encoding layer, the intermediate features at the preset context position are replaced by the deep learnable cue vectors corresponding to the layer, and the output features at the position corresponding to the text end marker are used to generate a deep text semantic vector.
[0055] In some embodiments, image feature vectors This is the result extracted from the visual large model encoder of CLIP, which inputs a single-view RGB image into it.
[0056] In some embodiments, during the training phase of the 3D object reconstruction network, the weights of the pre-trained CLIP text encoder and the visual large model encoder are forcibly locked and do not participate in parameter updates. The gradients from backpropagation are only used to update the aforementioned shallow and deep learnable cue vectors. That is, during the training phase of the 3D object reconstruction network, CLIP itself does not participate in training (does not update parameters), but is treated as a fixed feature extractor (weights are forcibly locked). Only the gradients from backpropagation are used to update the learnable context vectors concatenated at the front of the text, so that the optimization update of CLIP is completely converged to the learnable context vectors. This improves training efficiency while avoiding the huge computational overhead caused by full parameter fine-tuning of the visual large model. By adding a hierarchical semantic alignment constraint mechanism, the 3D object reconstruction network (especially for its semantically aware multi-level alignment decoder) can be forcibly guided to generate in the correct cross-class semantic topology direction, which helps to reduce cross-class shape confusion and improve the accuracy of cross-class shape reconstruction.
[0057] The following experimental examples illustrate the implementation effect of the edge- and semantically guided 3D object reconstruction method proposed in this application. Specifically, the publicly available large-scale 3D object dataset ShapeNet is used. This dataset contains 3D CAD models of multiple categories. This experiment selects 13 categories with complex topological structures, such as airplanes, benches, and cabinets, for testing. The specific 13 categories are shown in Tables 1 and 2. The sample set of each category is divided into training set, validation set, and test set, with the total number of test set samples exceeding 8700, used for model performance evaluation. Each training data sample uses a single-view RGB image as the input source and the corresponding target 3D voxel as the true supervision label. Each voxel is represented as a binary tensor in physical space: a value of 1 indicates that the voxel is occupied by a physical entity, and a value of 0 indicates that the voxel is an unoccupied empty background. The edge- and semantically guided 3D object reconstruction method proposed in this application (represented by ESGNet in Tables 1 and 2) is compared with existing single-view 3D reconstruction techniques (Matryoshka, IM-Net, AttSets, 3D-C2FT, Pix2Vox, Pix2Vox++, SWAGE-3D, and EBNet, etc.). The comparison results are shown in Tables 1 and 2. Table 1 compares the 3D reconstruction results of different methods on different categories of ShapeNet using Intersection-over-Union (IoU) as the evaluation index. Table 2 compares the 3D surface reconstruction results of different methods on different categories of ShapeNet using the 3D surface F-Score (i.e., F-Score@1%) as the evaluation index. IoU represents the spatial overlap similarity between the 3D voxels predicted by the 3D object reconstruction network and the real physical voxels. The larger the IoU value, the higher the spatial overlap between the 3D voxels predicted by the 3D object reconstruction network and the real 3D voxels on the current test data, and the more accurate and complete the reconstructed 3D shape. The expression for IoU is: , This represents the total number of voxels contained in a single 3D voxel mesh (set to in this experiment). ), Index representing a voxel and They are the first The network predicts the occupancy probability value and the actual physical label value of the 3D object reconstruction of individual elements.
[0058] Table 1
[0059] category Matryoshka IM-Net AttSets 3D-C2FT Pix2Vox SWAGE-3D ESGNet airplane 0.647 0.702 0.594 0.634 0.684 0.572 0.701 bench 0.577 0.564 0.552 0.562 0.616 0.401 0.617 cabinet 0.776 0.680 0.783 0.758 0.792 0.726 0.788 car 0.850 0.756 0.844 0.835 0.854 0.835 0.857 Chair 0.547 0.644 0.559 0.533 0.567 0.460 0.583 monitor 0.532 0.585 0.565 0.499 0.537 0.419 0.553 lamp 0.408 0.433 0.445 0.475 0.443 0.423 0.481 Speakers 0.701 0.683 0.721 0.683 0.714 0.693 0.718 rifle 0.616 0.723 0.601 0.596 0.615 0.463 0.654 sofa 0.681 0.694 0.703 0.674 0.709 0.662 0.704 table 0.573 0.621 0.590 0.565 0.601 0.524 0.627 Telephone 0.756 0.762 0.743 0.717 0.776 0.731 0.797 Water vehicles 0.591 0.607 0.601 0.572 0.594 0.544 0.622 overall 0.635 0.659 0.642 0.631 0.661 0.573 0.676
[0060] Table 1 compares the 3D reconstruction results (IoU index) of different methods on different categories of ShapeNet. All IoU results in Table 1 are rounded to three decimal places. As shown in the experimental data in Table 1, compared to the baseline model of existing technologies, the edge- and semantically guided 3D object method provided in this application achieves the highest result in sample-weighted average IoU across 13 categories, and achieves the best results for categories such as benches, cars, lights, tables, telephones, and water vehicles. Specifically, for challenging categories such as chairs, tables, and rifles, which contain numerous slender topologies and are prone to structural breakage, the IoU metric of this application's method is improved, validating its effectiveness. First, the topology-adaptive dynamic deformation optimizer introduces an adaptive deformation mesh, enabling the network's receptive field to adaptively extend and repair the topological breakage problem of slender structures. Second, the edge-guided feature fusion encoder fuses high-frequency edge images through a residual space attention mechanism, providing the network with strong geometric boundary priors resistant to background interference, thus recovering lost contours in low light and complex backgrounds. Furthermore, for geometric objects with similar shapes but different categories, the semantically aware multi-level alignment decoder enhances the utilization of cross-modal semantic information by combining the deep textual semantics of the multimodal large model (CLIP), mitigating cross-category shape confusion. Moreover, under evaluation conditions on a test set containing nearly ten thousand multi-category test samples, the overall average IoU of this application's method reaches 0.6763, demonstrating a higher generalization ability than the comparative methods listed in the table, indicating that this application's method can improve the 3D reconstruction accuracy of single-view RGB images.
[0061] Table 2
[0062] category Matryoshka IM-Net AttSets 3D-C2FT Pix2Vox++ EBNet ESGNet airplane 0.446 0.598 0.489 0.504 0.550 0.585 0.619 bench 0.424 0.361 0.406 0.401 0.446 0.453 0.482 cabinet 0.381 0.345 0.367 0.356 0.406 0.383 0.401 car 0.481 0.304 0.497 0.503 0.542 0.537 0.558 Chair 0.302 0.442 0.334 0.242 0.286 0.278 0.303 monitor 0.400 0.466 0.310 0.247 0.286 0.295 0.296 lamp 0.276 0.371 0.315 0.317 0.320 0.329 0.324 Speakers 0.279 0.200 0.211 0.242 0.283 0.270 0.250 rifle 0.514 0.407 0.524 0.530 0.547 0.576 0.615 sofa 0.326 0.354 0.334 0.317 0.376 0.361 0.357 table 0.374 0.461 0.419 0.323 0.379 0.377 0.411 Telephone 0.598 0.423 0.469 0.489 0.613 0.606 0.611 Water vehicles 0.360 0.369 0.315 0.354 0.384 0.396 0.403 overall 0.391 0.405 0.395 0.375 0.418 0.419 0.433
[0063] IoU focuses on evaluating macroscopic volume occupancy, while F-Score combines precision and recall to measure the geometric matching accuracy and completeness between the reconstructed 3D surface and the real 3D surface within a preset spatial distance threshold. Table 2 compares the 3D surface reconstruction results of different methods on different categories of ShapeNet (F-Score index). As can be seen from the experimental data in Table 2, under the evaluation of the F-Score index, which is sensitive to surface step artifacts, the edge- and semantically guided 3D object method provided in this application embodiment exhibits superior surface reconstruction capabilities. Specifically, under the experimental setup, the proposed method achieved an average F-Score of 0.4330 across 13 test categories, higher than the average results of other comparative methods listed in Table 2. Furthermore, the proposed method achieved the highest F-Score in five categories: aircraft, benches, automobiles, rifles, and watercraft, demonstrating overall superior performance compared to other baseline networks. This confirms that the proposed method can improve the overall geometric matching accuracy and integrity between the reconstructed 3D surface and the real 3D surface. Specifically, the Laplacian smoothing loss, by calculating and constraining the second-order discrete difference features of local curvature, helps suppress step artifacts and edge jaggedness on the voxel surface. Simultaneously, the combination of 3D focus loss to optimize extreme volume imbalances and the synergistic effect of adaptive deformable meshes further enhances the performance. The edge- and semantically guided 3D object reconstruction method provided in this application not only accurately locks the macroscopic volume occupancy of objects but also helps improve the geometric fitting accuracy of 3D microscopic boundaries, effectively enhancing the fidelity of the 3D reconstruction model and making it more suitable for practical rendering needs.
[0064] According to another embodiment of this application, a three-dimensional object reconstruction system based on edge and semantic guidance is provided, including a single-view RGB image acquisition unit and a pre-trained three-dimensional object reconstruction network. The three-dimensional object reconstruction network includes an edge-guided feature fusion encoder, a semantically aware multi-level alignment decoder, and a topology-adaptive dynamic deformation optimizer. The single-view RGB image acquisition unit is used to acquire a single-view RGB image of the three-dimensional object; the three-dimensional object reconstruction network includes a feature fusion encoder for edge guidance, a multi-level alignment decoder for semantic awareness, and a dynamic deformation optimizer for topology adaptation, connected sequentially. The feature fusion encoder includes an edge extraction network, an RGB image feature extraction branch, an edge image feature extraction branch, and a residual space attention module. The edge extraction network is used to generate corresponding high-frequency edge images from the single-view RGB image as input; the RGB image feature extraction branch is used to generate an original image feature tensor from the single-view RGB image as input. The edge image feature extraction branch is used to generate edge image feature tensors from high-frequency edge images as input. The residual spatial attention module is used to perform attention based on the feature tensor of the original image. and edge image feature tensor Generate two-dimensional fusion features Multi-level alignment decoders are used for 2D fusion features. Perform up-dimensional decoding to generate a coarse three-dimensional voxel model. The dynamic deformation optimizer is used for rough 3D voxel models. Refine the model to generate a target 3D voxel model of the 3D object. .
[0065] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the system described above can be referred to the corresponding processes in the foregoing method embodiments, and therefore will not be repeated here.
[0066] It should be noted that the edge- and semantically guided 3D object reconstruction system provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of this application can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of this application are only for distinguishing the various modules or steps and are not considered as an improper limitation of this application.
[0067] The edge- and semantic-guided 3D object reconstruction method and system provided in this application proposes an edge- and semantic-guided 3D object reconstruction framework, ESGNet. Addressing the shortcomings of traditional 3D object reconstruction networks, it integrates edge priors, hierarchical semantic alignment constraints, and deformable convolution optimization strategies to upgrade the traditional encoder-decoder architecture, thus overcoming the deficiencies of existing technologies in feature extraction, semantic matching, detail reconstruction, and model optimization. It can effectively extract feature information of occluded areas and slender topological structures, helping to alleviate the problems of fractures in slender structures such as gun barrels and furniture support legs, as well as reconstruction distortion in occluded areas. By relying on multi-dimensional optimization constraints to suppress voxel noise, it reduces the step-like and jagged artifacts of traditional models, improving the surface smoothness and structural integrity of 3D models. Simultaneously, it alleviates the limitations of local feature perception, mines potential topological features of objects, establishes the correlation between image, text semantics, and 3D shapes, and improves the accuracy, robustness, and generalization ability of single-view 3D reconstruction, making it applicable to the needs of 3D reconstruction and rendering in different scenarios.
[0068] The embodiments provided in this application can also be computer program products. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this application. A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. Computer-readable storage media may include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.
[0069] This application uses specific embodiments to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the solution and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for reconstructing 3D objects based on edge and semantic guidance, characterized in that, include: A single-view RGB image of a 3D object is acquired, and the single-view RGB image is input into a trained 3D object reconstruction network to obtain a target 3D voxel model of the 3D object. ; The 3D object reconstruction network includes a feature fusion encoder for edge guidance, a multi-level alignment decoder for semantic awareness, and a dynamic deformation optimizer for topology adaptation, which are connected in sequence. The edge-guided feature fusion encoder includes an edge extraction network, an RGB image feature extraction branch, an edge image feature extraction branch, and a residual space attention module. The edge extraction network generates corresponding high-frequency edge images based on the input single-view RGB image, and the edge image feature extraction branch generates an edge image feature tensor based on the high-frequency edge images. The RGB image feature extraction branch generates the original image feature tensor based on the single-view RGB image. The residual spatial attention module is based on the original image feature tensor and the edge image feature tensor Generate two-dimensional fusion features ; The semantically aware multi-level alignment decoder applies the two-dimensional fusion features. Perform up-dimensional decoding to generate a coarse three-dimensional voxel model. ; The topology-adaptive dynamic deformation optimizer optimizes the rough 3D voxel model. The model is refined to output the target 3D voxel model of the 3D object. .
2. The 3D object reconstruction method based on edge and semantic guidance according to claim 1, characterized in that, The edge extraction network includes multiple cascaded extraction modules and splicing modules that are connected one-to-one with the extraction modules. Each extraction module includes a multi-scale convolutional flow submodule and a corresponding upsampling alignment submodule. The multi-scale convolutional flow submodule includes cascaded downsampling layers and densely connected convolutional blocks. The downsampling layers are used to reduce the spatial resolution of the input features, and the densely connected convolutional blocks are used to extract image features at the current scale. Each of the multi-scale convolutional flow submodules outputs a multi-scale image feature tensor. The upsampling alignment submodule uses deconvolution operations to upsample the corresponding image feature tensors at each scale in terms of spatial resolution and align them to the size of the single-view RGB image. The stitching module is used to stitch together the aligned multiple image feature tensors in the channel dimension and then output a single-channel high-frequency edge image through a single-layer convolution. The RGB image feature extraction branch is a 16-layer convolutional neural network with a visual geometry group; The structure of the edge image feature extraction branch is symmetrical to that of the RGB image feature extraction branch. The edge image feature extraction branch is a 16-layer convolutional neural network of visual geometry group. The number of input channels of the first convolutional layer of the 16-layer convolutional neural network of visual geometry group is 1. The weights of the convolutional kernel of the first convolutional layer are initialized to the arithmetic mean of the original pre-trained weights in the input channel dimension.
3. The 3D object reconstruction method based on edge and semantic guidance according to claim 1, characterized in that, The residual spatial attention module is based on the original image feature tensor. and the edge image feature tensor Generate two-dimensional fusion features ,include: Using the edge image feature tensor Using the spatial attention layer as input, a two-dimensional spatial weight map is obtained. The spatial attention layer comprises cascaded 1×1 convolutional layers with unbiased terms and a sigmoid activation function, wherein... This represents a 1×1 convolutional layer function. It is a natural constant; The two-dimensional spatial weight map With the original image feature tensor Element-wise multiplication and then with the original image feature tensor Perform residual jump aggregation to obtain two-dimensional fused features. , ,in This indicates element-wise multiplication. When performing element-wise multiplication, the single-channel two-dimensional spatial weight map is used. Broadcast along the channel dimension to the original image feature tensor Same number of channels.
4. The 3D object reconstruction method based on edge and semantic guidance according to claim 1, characterized in that, The method also includes, during the training phase of the 3D object reconstruction network, the edge extraction network performing synchronous random erasure processing on the input sample data: The input sample data includes a single-view RGB image and the corresponding high-frequency edge image; The overlapping area of a single-view RGB image and its corresponding high-frequency edge image is randomly occluded with a preset trigger probability. A rectangular area is randomly generated in the overlapping area, which accounts for a preset percentage of the area of the single-view RGB image. The preset percentage ranges from 2% to 15%. The rectangular region is filled with pixels of a preset grayscale value, and the region in the high-frequency edge image corresponding to the rectangular region is filled with pixels of a grayscale value of 0.
5. The edge- and semantically guided 3D object reconstruction method according to claim 1, characterized in that, The dynamic deformation optimizer includes a cascaded initial 3D feature extraction module, a 3D convolutional spatial offset prediction module, a 3D trilinear interpolation resampling module, a standard 3D convolutional thinning layer, a feature normalization and activation layer, a multi-level stacked 3D convolutional layer, and a feature mapping module, wherein: The initial 3D feature extraction module is used to process the coarse 3D voxel model. Three-dimensional convolution and pooling are performed to obtain a three-dimensional feature tensor at the current scale. The input to the three-dimensional convolutional spatial offset prediction module is the three-dimensional feature tensor at this scale. The three-dimensional convolutional spatial offset prediction module is used to perform a three-dimensional convolution operation with a kernel size of 3×3×3, a padding size of 1, and an output channel number of 3, outputting a three-dimensional offset field tensor of the three-dimensional spatial coordinates of the object surface. ; The three-dimensional trilinear interpolation resampling module uses the current-scale three-dimensional feature tensor and an adaptive deformation sampling mesh. The output is a synchronous input, and a differentiable three-dimensional trilinear interpolation operation is performed to output reconstructed features, the adaptive deformable sampling mesh. The expression is ,in, The standard three-dimensional normalized base sampling grid represents the spatial coordinate distribution within the range [-1, 1], and the three-dimensional offset field tensor is... The output of the three-dimensional convolutional spatial offset prediction module is h, which is a preset deformation adjustment factor. The standard 3D convolutional thinning layer is used to perform feature dimension transformation on the reconstructed features output by the 3D trilinear interpolation resampling module; The feature normalization and activation layer includes a cascaded three-dimensional batch normalization operator and a nonlinear activation function; The multi-level stacked 3D convolutional layer includes a downsampling feature extraction unit, a fully connected bottleneck unit, and an upsampling reconstruction unit. It fuses 3D features of the same spatial scale through skip connections to perform multi-scale spatial feature extraction and resolution restoration on the reconstructed features. The multi-level stacked 3D convolutional layer performs spatial feature extraction and dimensionality upsampling on the output of the feature normalization and activation layers. The feature mapping module includes cascaded single-layer 3D deconvolution and a sigmoid activation function, used to map the feature tensors output by the multi-level stacked 3D convolutional layers to occupancy probability values between 0 and 1, thus obtaining a refined 3D voxel model. and the refined three-dimensional voxel model With rough 3D voxel model Equal weight fusion is performed to finally output the target 3D voxel model. ,in, .
6. The 3D object reconstruction method based on edge and semantic guidance according to claim 1, characterized in that, The multi-level alignment decoder includes a cascaded 3D spatial feature reconstruction layer, a first 3D deconvolution decoding layer, a second 3D deconvolution decoding layer, a third 3D deconvolution decoding layer, a fourth 3D deconvolution decoding layer, a 3D feature refinement layer, and an occupancy probability mapping layer, wherein: The three-dimensional spatial feature reconstruction layer uses tensor reshaping operations to transform the input two-dimensional fused features. The mapping is to an initial three-dimensional voxel feature, the initial three-dimensional voxel feature having a spatial resolution of 2×2×2 and a channel number of 2048; The kernel size of the first three-dimensional deconvolution decoding layer is 4×4×4, and the number of channels is 512; The second three-dimensional deconvolution decoding layer has a kernel size of 4×4×4 and 128 channels; The kernel size of the third 3D deconvolution decoding layer is 4×4×4, and the number of channels is 64. The kernel size of the fourth 3D deconvolution decoding layer is 4×4×4, and the number of channels is 32. The three-dimensional feature refinement layer includes a three-dimensional convolutional layer with a kernel size of 3×3×3, a padding size of 1, 32 input channels and 8 output channels, as well as a three-dimensional batch normalization layer and a non-linear activation function connected in sequence with the three-dimensional convolutional layer. The occupancy probability mapping layer includes a 3D convolutional layer with a kernel size of 1×1×1, 8 input channels, and 1 output channel, and a Sigmoid activation function connected to the 3D convolutional layer. This function maps the feature tensor output by the 3D feature refinement layer to voxel occupancy probabilities between 0 and 1, resulting in a coarse 3D voxel model. .
7. The edge- and semantically guided 3D object reconstruction method according to claim 6, characterized in that, Also includes: During the training phase of the 3D object reconstruction network, the optimization objective is to minimize the mixed total loss. : ; in, This represents the hierarchical semantic alignment constraint loss; 3D focus loss ; Dice loss ; Laplace smoothing loss ; in, This represents the total number of voxels within a single 3D voxel grid. Indicates the voxel point index; This indicates that the 3D object reconstruction network predicts the first... The probability that an individual element is an entity. Indicates the first The true voxel label at the location corresponding to the voxel, and ; Indicates the first The probability that an individual element is correctly predicted, and: ,when hour, ,when hour, ; Indicates based on real voxel labels Selected preset asymmetric category balance weights; This indicates the preset focus attenuation parameter; This indicates a preset numerical smoothing term used to prevent division by zero; This represents a preset 3D discrete Laplacian convolution kernel used to extract local second-order discrete difference features of the target 3D voxel model; This represents the computation of discrete convolution in three-dimensional space. Describes the norm, This represents the mean absolute value of the calculation results for all voxel positions.
8. The 3D object reconstruction method based on edge and semantic guidance according to claim 7, characterized in that, The hierarchical semantic alignment constraint loss ; in, For single-layer semantic projection loss The value of s=8 represents the single-layer semantic projection loss corresponding to the second three-dimensional deconvolution decoding layer, and the scale identifier of the second three-dimensional deconvolution decoding layer is 8. For single-layer semantic projection loss The value of s when s is 16 The single-layer semantic projection loss corresponding to the third 3D deconvolutional decoding layer is represented by 16, and s represents the scale identifier. ; in, The size of the current training batch. This refers to the sample index within the current training batch. The sample index is The intermediate layer 3D semantic vector is obtained by passing the output of the 3D deconvolution decoding layer with time scale identifier s through a 3D semantic projection head. for With deep text semantic vectors Cosine similarity between them for With image feature vectors Cosine similarity between them; The sample index within the current training batch is The loss weighting coefficient; This is a preset numerical stability term used to prevent the denominator from being zero, and >0; The three-dimensional semantic projection head includes a cascaded three-dimensional adaptive average pooling layer, a flattening layer, and a fully connected linear layer. The three-dimensional adaptive average pooling layer is used to compress the spatial dimension of the input features of the three-dimensional semantic projection head to 1×1×1. After the output of the three-dimensional adaptive average pooling layer is processed by the flattening layer, the fully connected linear layer maps the output of the flattening layer to a 512-dimensional feature vector space.
9. A three-dimensional object reconstruction system based on edge and semantic guidance, characterized in that, It includes a single-view RGB image acquisition unit and a 3D object reconstruction network, wherein: The single-view RGB image acquisition unit is used to acquire a single-view RGB image of a three-dimensional object; The 3D object reconstruction network comprises a feature fusion encoder for edge guidance, a multi-level alignment decoder for semantic awareness, and a dynamic deformation optimizer for topology adaptation, connected in sequence. The feature fusion encoder includes an edge extraction network, an RGB image feature extraction branch, an edge image feature extraction branch, and a residual space attention module. The edge extraction network generates corresponding high-frequency edge images using the single-view RGB image as input. The RGB image feature extraction branch generates an original image feature tensor using the single-view RGB image as input. The edge image feature extraction branch is used to generate an edge image feature tensor using the high-frequency edge image as input. The residual spatial attention module is used to perform the following based on the original image feature tensor: and the edge image feature tensor Generate two-dimensional fusion features ; The multi-level alignment decoder is used for the two-dimensional fusion feature. Perform up-dimensional decoding to generate a coarse three-dimensional voxel model. ; The dynamic deformation optimizer is used for the rough 3D voxel model. The model is refined to generate a target 3D voxel model of the 3D object. .
10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-8.
