Multi-view fusion and neural network combined 3D object reconstruction method

Abstract

The invention discloses a 3D object reconstruction method combining multi-view fusion and a neural network, and the method comprises the following steps: collecting a multi-view image of a target object, carrying out the geometric calibration and view parameter calibration, and generating a standardized image sequence; inputting the image sequence into a feature extraction and voxel fusion module to obtain a preliminary three-dimensional space representation body as a coarse reconstruction model; performing uncertainty evaluation on the coarse reconstruction model, generating a voxel-level confidence coefficient heat map, and dividing the voxel-level confidence coefficient heat map into a plurality of confidence coefficient intervals; based on the confidence interval, constructing an adaptive repair network with a multi-scale residual path, and outputting and activating different repair paths as required by using a path gating mechanism; and fusing the residual output of each repair path with the coarse reconstruction model to generate an optimized final three-dimensional reconstruction model. According to the method, the risk of excessive repair or error repair can be effectively reduced, the adaptability of the model to complex areas such as sheltered areas is enhanced, and the integrity and precision of the whole three-dimensional reconstruction model are improved.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a 3D object reconstruction method combining multi-view fusion with a neural network. Background Art

[0002] With the continuous development of computer vision and 3D reconstruction technologies, 3D object reconstruction methods based on multi-view images have become an important research direction in fields such as automated modeling, virtual reality, and robotic vision. Existing multi-view 3D reconstruction techniques typically rely on structured light, LiDAR, or traditional voxel fusion strategies, acquiring images or depth information from multiple angles to generate a 3D model of the target object.

[0003] However, in practical applications, existing technologies still face numerous challenges when constructing 3D models. Issues such as incomplete view coverage, severe occlusion, texture loss, and varying illumination often result in low geometric accuracy and structural restoration capabilities in the initial reconstructed model. Traditional methods often employ a unified fusion strategy to generate voxel models, which struggles to accurately describe complex geometric structures. This is particularly true in areas with boundary details and occluded regions, where blurring, artifacts, or structural loss can occur, compromising the integrity and accuracy of the final 3D reconstructed model.

[0004] Furthermore, most reconstruction methods lack regional reliability analysis mechanisms, making it difficult to effectively characterize reconstruction uncertainty in different regions of the model. This makes it difficult to clearly identify areas of low confidence in the reconstruction results, making targeted optimization or repair difficult. Furthermore, existing deep learning reconstruction methods often use fixed-structure network architectures to uniformly optimize the entire model, lacking differentiated processing strategies for different regions. This can easily lead to problems such as redundant computation, incorrect repairs, or overcompensation of local structures, leaving limited room for improvement in overall robustness and accuracy. Summary of the Invention

[0005] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0006] To solve the above technical problems, the present invention provides the following technical solution: a 3D object reconstruction method combining multi-view fusion and neural network, comprising the following steps: Collect multi-view images of the target object and perform geometric calibration and perspective parameter calibration to generate a standardized image sequence; Inputting the image sequence into a feature extraction and voxel fusion module to obtain a preliminary three-dimensional spatial representation as a coarse reconstruction model; performing uncertainty assessment on the coarse reconstruction model, generating a voxel-level confidence heat map and dividing the confidence heat map into a plurality of confidence intervals; Based on the confidence interval, an adaptive repair network with multi-scale residual paths is constructed, and different repair paths are output and activated on demand using a path gating mechanism; The residual output of each repair path is fused with the rough reconstruction model to generate an optimized final three-dimensional reconstruction model.

[0007] As a preferred embodiment of the 3D object reconstruction method combining multi-view fusion and neural network of the present invention, the image sequence is input into a feature extraction and voxel fusion module to obtain a preliminary 3D space representation as a coarse reconstruction model, including the following steps: The standardized image sequence is input into the feature extraction module, which extracts image features with deep semantic and geometric expression capabilities from the two-dimensional image through the feature extraction network. The image features include but are not limited to edge features, texture features, and spatial structure information; Construct a 3D voxel grid of a predefined size as the initial structure of the 3D space representation.

[0008] A fusion mechanism is used to form a 3D voxel representation containing initial occupancy probability, spatial semantic information and coarse geometric structure features for features mapped to the same voxel position from multiple perspectives.

[0009] The fused 3D voxel representation is used as a coarse reconstruction model.

[0010] As a preferred solution of the 3D object reconstruction method combining multi-view fusion and neural network described in the present invention, the fusion mechanism adopts a spatial attention enhanced voxel fusion strategy, which realizes feature weighting and consistency correction under different perspectives by introducing cross-view attention maps and spatial context encoders.

[0011] As a preferred solution of the 3D object reconstruction method combining multi-view fusion and neural network of the present invention, wherein: performing uncertainty assessment on the coarse reconstruction model, generating a voxel-level confidence heat map and dividing it into multiple confidence intervals, includes the following steps: A voxel uncertainty quantification model is constructed to measure the uncertainty of each voxel unit in the coarse reconstruction model. The model input is the fused three-dimensional space voxel representation, and the output is the confidence score of each voxel. Construct a voxel-level confidence heatmap to spatially express the uncertainty distribution of the entire 3D voxel representation.

[0012] According to the confidence heat map, the 3D voxel representation is divided into multiple confidence intervals.

[0013] As a preferred solution of the 3D object reconstruction method combining multi-view fusion and neural network of the present invention, the confidence score calculation formula is as follows:

[0014] in: Indicates the confidence scores for each voxel; Indicates the number of views mapped to the voxel, that is, how many images produce valid feature projections on it; Indicates the Viewpoint-to-voxel Extracted image feature vector; represents the average value of all view features, ; represents the variance of the voxel’s features under multiple viewing angles, ; Indicates the The weight of perspective; Represents extremely small positive numbers to prevent division by zero errors; Indicates the boundary complexity of the voxel in space. The closer to the boundary or detail area, the larger the value. represents the balance parameter, which is used to control the influence of structural complexity on the overall confidence; is the Sigmoid function, which is used to map the final confidence to the 0-1 interval; The interactive fusion score representing cross-view feature consistency and voxel spatial complexity; It is the L2 norm squared, which is used to measure the gap between the image features of the view and the average features, reflecting the stability of the view; Represents the feature variance of all views at this voxel, which serves as a residual normalization factor to suppress the influence of data scale on the score; It represents the joint penalty term of the number of feature perspectives and structural complexity. The more complex the structure, the lower the score. range: ,in: when When , it means that the voxel is covered by multiple high-confidence view-stable features and the spatial structure is smooth; when When , it means that the voxel has a large cross-view feature deviation, or is located near the structure boundary and has high uncertainty.

[0015] As a preferred solution of the 3D object reconstruction method combining multi-view fusion and neural network of the present invention, wherein: according to the confidence heat map, the three-dimensional space voxel representation volume is divided into multiple confidence intervals, including the following steps: One or more confidence thresholds are determined according to the fuzzy grading algorithm, and the range of voxel confidence values is divided into a high confidence interval, a medium confidence interval, and a low confidence interval. The range of voxel confidence values is , the division rules are as follows: 0<low confidence interval<first threshold, this interval indicates that the area has high uncertainty and serious defects, and needs to be reconstructed; The first threshold ≤ medium confidence interval < second threshold, this interval indicates that the area has medium uncertainty and needs moderate repair; The second threshold is ≤ high confidence interval < 1, which indicates that the regional structure is stable and the semantics are clear; The confidence value of each voxel unit in the voxel-level confidence heat map is compared according to the set threshold, and each voxel is assigned a corresponding interval label; The confidence interval labels of each voxel obtained are integrated to generate an uncertainty partition map that corresponds one-to-one with the original three-dimensional space voxel representation in terms of structure and position information. Each voxel unit in the partition map has a corresponding confidence interval label.

[0016] As a preferred solution of the 3D object reconstruction method combining multi-view fusion and neural network of the present invention, wherein: based on the confidence interval, an adaptive restoration network with multi-scale residual paths is constructed, and a path gating mechanism is used to activate different restoration paths on demand, including the following steps: An adaptive inpainting network driven by uncertainty partition maps is constructed. The network consists of multiple multi-scale convolutional blocks with different receptive fields, each of which is responsible for inpainting uncertain regions at a specific scale. The network receives the original 3D voxel representation and the uncertainty partition map as joint input. The uncertainty labels guide the convolutional blocks of different scales to focus on extracting structural information and completing details in high, medium, and low confidence regions, and generate corresponding residual outputs at each scale path. A path gating mechanism is introduced into the adaptive repair network, which is embedded in the selection module between multi-scale convolution paths; According to the decision of the path gating mechanism, the corresponding repair path is activated, and different repair strategies are performed on voxel areas in low, medium, and high confidence intervals.

[0017] As a preferred solution of the 3D object reconstruction method combining multi-view fusion and neural network of the present invention, wherein: the residual output of each repair path is fused with the rough reconstruction model to generate the optimized final 3D reconstruction model, including the following steps: According to the activation results of the path gating mechanism, the residual outputs of each restoration path corresponding to the low, medium, and high confidence intervals are extracted, and the residual outputs are weightedly fused with the coarse reconstruction results of the corresponding positions layer by layer to obtain a preliminary fused 3D model; During the fusion process, different weights are assigned to each repair path based on the decision and confidence value of the path gating mechanism; Perform post-processing optimization operations based on the fused 3D model; Through multiple iterative repair and fusion processes, the final optimized high-quality 3D reconstruction model is generated.

[0018] As an optimal solution for the 3D object reconstruction method combining multi-view fusion and neural network described in the present invention, the layer-by-layer weighted fusion adopts a residual attention fusion module, and the residual attention fusion module interactively aligns the residual output of the repair path with the original voxel feature map through a joint mechanism of spatial attention and channel attention.

[0019] As an optimal solution for the 3D object reconstruction method combining multi-view fusion and neural network described in the present invention, the final optimized high-quality 3D reconstruction model is optimized by a joint loss function during the training phase, and the joint loss function includes three parts: structural similarity loss, geometric reconstruction error and uncertainty guidance loss.

[0020] Beneficial effects of the present invention: 1. This invention performs uncertainty assessment on the coarse reconstruction model and generates a voxel-level confidence heat map, which can clearly identify unstable or unoptimized areas within the coarse reconstruction model. The voxel-level confidence heat map is further divided into multiple confidence intervals, which are used to drive an adaptive inpainting network with multi-scale residual paths. A path gating mechanism is used to activate different inpainting paths on demand, achieving differentiated detail compensation for different confidence intervals. Furthermore, the path gating mechanism effectively reduces the risk of over-inpainting or erroneous inpainting, enhances the model's adaptability to complex areas such as occlusion, and improves the integrity and accuracy of the overall 3D reconstructed model.

[0021] 2. The present invention collects multi-view images of the target object and performs geometric calibration and view parameter calibration, then inputs the standardized image sequence into the feature extraction and fusion module, which can fully obtain the image semantics and geometric features from multiple angles and realize the effective mapping from two-dimensional images to three-dimensional structures; the preliminary three-dimensional spatial representation subsequently formed serves as a coarse reconstruction model, which retains the spatial consistency and basic morphological structure from multiple perspectives, providing a solid foundation for subsequent uncertainty assessment and refined reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them: Figure 1 This is a flowchart of a 3D object reconstruction method combining multi-view fusion and neural network according to the present invention. DETAILED DESCRIPTION

[0023] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0024] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0025] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0026] Furthermore, the present invention is described in detail with reference to schematic diagrams. For ease of illustration, when describing the embodiments of the present invention, cross-sectional views illustrating device structures may be partially enlarged and not to scale. Furthermore, the schematic diagrams are merely illustrative and should not limit the scope of protection of the present invention. Furthermore, in actual production, the three-dimensional dimensions of length, width, and depth should be included.

[0027] Example 1 Reference Figure 1 , which is a first embodiment of the present invention, provides a 3D object reconstruction method combining multi-view fusion and neural network, comprising the following steps: S1: Collect multi-view images of the target object and perform geometric calibration and perspective parameter calibration to generate a standardized image sequence.

[0028] It should be noted that an image acquisition system with automatic calibration capabilities for intrinsic and extrinsic parameters is used to capture images of the target object from multiple viewpoints. The resulting images are geometrically calibrated to eliminate distortion errors and achieve uniform image resolution. Simultaneously, the camera's extrinsic parameters (camera pose matrix) and intrinsic parameters (focal length, optical center parameters) are extracted to generate a standardized image sequence with spatial consistency markers. This standardized sequence not only ensures strict spatial alignment of the multi-view images but also provides a coordinate-consistent input foundation for subsequent steps such as feature extraction, voxel fusion, and uncertainty assessment, thereby ensuring the accuracy and robustness of cross-view feature matching during 3D reconstruction.

[0029] S2: Input the image sequence into the feature extraction and voxel fusion module to obtain a preliminary three-dimensional spatial representation as a coarse reconstruction result.

[0030] It should be noted that this step is a key transition link from two-dimensional images to three-dimensional space modeling in this application. Its purpose is to efficiently encode the semantic and geometric information in the standardized image sequence and project it into a unified three-dimensional space to form a coarse-grained spatial occupancy representation, thereby constructing a preliminary structural framework of the three-dimensional model and providing input basis for subsequent uncertainty assessment and adaptive repair networks.

[0031] The specific implementation includes the following steps: S2.1: The standardized image sequence is input into the feature extraction module. This module extracts image features with deep semantic and geometric expression capabilities from the two-dimensional image through the feature extraction network. The image features include but are not limited to edge features, texture features, and spatial structure information.

[0032] It should be noted that the feature extraction network uses a convolutional neural network as its basic architecture, incorporating residual connections to enhance deep feature extraction capabilities. It further introduces an attention mechanism or Transformer architecture to enhance the network's perception of local key regions (such as edges, structural discontinuities, and texture transitions), achieving multi-scale, multi-semantic image representation. Furthermore, to ensure consistency of cross-view images in the subsequent 3D spatial mapping, the network is designed to support parameter sharing, allowing images acquired from multiple viewpoints to be processed within a network with the same structure, thus maintaining consistency and alignment of feature representations. Furthermore, after dimensionality reduction and encoding, the multi-level image features output by the network are organized into a set of semantic vectors with spatial semantic labels. This set not only carries information about the local structure of the image but also contains information about its relationships that can be projected into 3D space. The semantic vectors of this set are combined with the camera view parameters (including intrinsic and extrinsic parameters) obtained in step S1 and mapped to 3D space in the world coordinate system via a differentiable back-projection function, serving as the input for the subsequent 3D voxel grid construction.

[0033] This feature extraction network not only extracts key image features such as edges, textures, and spatial structures, but also improves the accuracy of voxel feature fusion in subsequent 3D reconstruction while maintaining spatial consistency between multi-view images. The parallel extraction of multi-scale and multi-semantic features enables the coarse reconstruction process to possess stronger spatial representation capabilities.

[0034] S2.2: Construct a 3D voxel grid of a predefined size as the initial structure of the 3D spatial representation.

[0035] It should be noted that the purpose of constructing a three-dimensional voxel grid is to provide a structured carrier unit for the spatial projection of multi-view features, ensuring that the semantic vectors from different image perspectives in subsequent steps can be accurately landed and fused in space.

[0036] The 3D voxel grid uniformly discretizes the target reconstruction space at a fixed resolution, dividing the continuous 3D space into a number of voxel units, each representing a minimum spatial occupancy unit. The 3D voxel grid is constructed based on the camera intrinsic and extrinsic parameters extracted from S1. A unified world coordinate system is established around the target object and this coordinate system is rasterized in 3D, ensuring that image features from different viewpoints can be accurately mapped to the same coordinate reference.

[0037] In step S2.1, the semantic vector set will be used as the direct input for the construction of the three-dimensional voxel grid. Specifically, the image-space projection relationship is utilized and a differentiable perspective projection or back-projection mechanism is adopted to map the semantic vector to a three-dimensional space coordinate point according to its image coordinates and viewing angle information, and the voxel unit in which it is located is determined accordingly.

[0038] The voxel units to determine where it is located include: Based on the semantic vector's image coordinates, camera view parameters, and 3D back-projection function, the semantic vector's spatial position in the world coordinate system is determined. Combined with the preset 3D voxel grid resolution, the discretized voxel unit index is generated, thus binding the semantic information to the 3D spatial position. This determination process not only achieves a precise mapping of the semantic vector to spatial voxels but also provides a unified spatial reference and alignment foundation for subsequent steps such as multi-view semantic fusion, local uncertainty assessment, and final model optimization.

[0039] It should be further explained that, considering that when semantic vectors from different perspectives are mapped to the spatial voxel grid through back projection, there may be multiple image perspectives that may aggregate different semantic information and geometric expressions in the same voxel unit, forming a typical "multi-source feature conflict" phenomenon. Therefore, in order to effectively integrate feature information from different perspectives and improve the robustness and consistency of spatial expression, it is necessary to build a fusion mechanism on this basis to generate a unified three-dimensional spatial voxel representation.

[0040] S2.3: For features mapped to the same voxel position from multiple perspectives, a fusion mechanism is used to form a 3D voxel representation that includes initial occupancy probability, spatial semantic information, and coarse geometric structure features.

[0041] It should be noted that when semantic vectors from different viewpoints are mapped to the spatial voxel grid through back-projection, there may be multiple image viewpoints that may aggregate different semantic information and geometric expressions within the same voxel unit, forming a typical "multi-source feature conflict" phenomenon. Therefore, in order to effectively integrate feature information from different viewpoints and improve the robustness and consistency of spatial expression, it is necessary to build a fusion mechanism on this basis to generate a unified three-dimensional spatial voxel representation. Specifically, the fusion mechanism adopts a spatial attention-enhanced voxel fusion strategy. This strategy achieves feature weighting and consistency correction from different viewpoints by introducing a cross-view attention map and a spatial context encoder, thereby improving the expressiveness of the fusion result in edges and occluded areas.

[0042] S2.4: Output the fused 3D voxel representation as a coarse reconstruction model. The output is used as the input object of step S3 to perform voxel-level uncertainty assessment and confidence heatmap generation.

[0043] It should be noted that while the fused voxel grid formed in S2.3 possesses basic geometric structure and semantic information, some voxel units still have structural uncertainty or semantic ambiguity due to uncontrollable factors such as occlusion, blur, and reflection interference in the input image. Therefore, in this step, the fused 3D voxel grid is uniformly output as a "coarse reconstruction model" and handed off in a structured manner to step S3 for targeted uncertainty modeling and repair processing.

[0044] S3: Uncertainty assessment is performed on the coarse reconstruction model, a voxel-level confidence heat map is generated and divided into multiple confidence intervals.

[0045] It should be noted that this step is mainly used to identify and quantify potential uncertainty areas in the rough reconstruction model and build a structured confidence heat map based on this. It specifically includes the following steps: S3.1: Construct a voxel uncertainty quantification model to measure the uncertainty of each voxel unit in the coarse reconstruction model; It should be noted that due to multi-view feature fusion errors, image occlusions, uneven sampling density, or image quality fluctuations in the coarse reconstruction model, the semantic or geometric representation of some voxel units may have high uncertainty. To quantify this uncertainty, a voxel-level uncertainty quantification model is constructed. The model input is the 3D voxel grid (3D spatial voxel representation) fused in S2.4, and the output is a confidence score for each voxel.

[0046] The confidence score calculation formula is as follows:

[0047] in: Indicates the confidence scores for each voxel; Indicates the number of views mapped to the voxel, that is, how many images produce valid feature projections on it; Indicates the Viewpoint-to-voxel Extracted image feature vector; represents the average value of all view features, ; represents the variance of the voxel’s features under multiple viewing angles, ; Indicates the The weight of the viewing angle can take into account the viewing angle difference, clarity, exposure, etc., and after normalization, it satisfies ; Represents a very small positive number to prevent division by zero errors. The typical value is ; Indicates the boundary complexity of the voxel in space. The closer to the boundary or detail area, the larger the value is. It is calculated by the voxel spatial gradient; represents the balance parameter, which is used to control the influence of structural complexity on the overall confidence; is the Sigmoid function, which is used to map the final confidence to the 0-1 interval; The interactive fusion score representing cross-view feature consistency and voxel spatial complexity; It is the L2 norm squared, which is used to measure the gap between the image features of the view and the average features, reflecting the stability of the view; Represents the feature variance of all views at this voxel, which serves as a residual normalization factor to suppress the influence of data scale on the score; It represents the joint penalty term of the number of feature perspectives and structural complexity. The more complex the structure, the lower the score. range: ,in: when When , it means that the voxel is covered by multiple high-confidence view-stable features and the spatial structure is smooth; when When , it means that the voxel has a large cross-view feature deviation, or is located near the structure boundary and has high uncertainty.

[0048] When implementing: Assume that a voxel is mapped by images from 5 perspectives, namely , the image features are as follows: ; ; ; ; ; Other parameter settings are as follows: , , ; Find the average of the viewing angle features :

[0049] Calculate the square of the characteristic residual : For example, for :

[0050] Calculate all The squared residuals up to 5 are as follows: ; ; ; ; Calculating variance :

[0051] Normalized residual similarity term, For example:

[0052] All calculations are as follows: , , ,

[0053] Final calculation results:

[0054] The above examples show that the feature perspectives vary greatly and the spatial structure complexity is relatively high.

[0055] S3.2: Construct a voxel-level confidence heatmap to spatially express the uncertainty distribution of the entire 3D voxel representation.

[0056] It should be noted that in this step, in order to more intuitively characterize the spatial distribution characteristics of uncertainty in the three-dimensional voxel representation and provide a spatial perception basis for the selection and activation of subsequent repair paths, the voxel confidence score obtained in step S3.1 is mapped to the corresponding voxel position in the three-dimensional voxel representation according to its spatial coordinates, and then a voxel-level confidence heat map is constructed.

[0057] This voxel-level confidence heatmap uses a 3D voxel grid as its base unit structure, with each voxel corresponding to a confidence value, thus forming a structured confidence field with position encoding in 3D space. In this way, explicit spatial modeling of uncertainty information is achieved.

[0058] S3.3: According to the confidence heat map, the three-dimensional voxel representation is divided into multiple confidence intervals and output to the S4 step as the basis for activating the repair path.

[0059] Specifically, according to the confidence heat map, the three-dimensional voxel representation volume is divided into multiple confidence intervals, including the following steps: S3.31: Determine one or more confidence thresholds based on a fuzzy hierarchical algorithm, and divide the range of voxel confidence values into a high confidence interval, a medium confidence interval, and a low confidence interval. The range of voxel confidence values is , the division rules are as follows: 0<low confidence interval<first threshold, this interval indicates that the area has high uncertainty and serious defects, and needs to be reconstructed; The first threshold ≤ medium confidence interval < second threshold, this interval indicates that the area has medium uncertainty and needs moderate repair; The second threshold is ≤ high confidence interval < 1, which indicates that the regional structure is stable and the semantics are clear.

[0060] The first threshold is set to 0.5, and the second threshold is set to 0.8.

[0061] S3.32: Compare the confidence values of each voxel in the voxel-level confidence heatmap against the thresholds set in S3.31 and assign each voxel a corresponding interval label. For example, if the confidence value of a voxel is 0.85, it is assigned to the high confidence interval; if it is 0.6, it is assigned to the medium confidence interval; and if it is 0.3, it is assigned to the low confidence interval.

[0062] This step converts the confidence heat map into a structured uncertainty distribution map, providing precise control signals for the path gating mechanism in S4, ensuring that different uncertainty regions can activate corresponding repair paths.

[0063] S3.33: Integrate the confidence interval labels of each voxel obtained in S3.32 to generate an uncertainty partition map that corresponds one-to-one with the original three-dimensional space voxel representation in terms of structure and position information, and each voxel unit in the partition map has a corresponding confidence interval label.

[0064] It should be noted that the partition map not only intuitively displays the uncertainty distribution of each region in the three-dimensional space, but also serves as one of the inputs of the adaptive repair network in the subsequent S4, ensuring that the repair processing of the entire system is based on data-driven precise partitioning.

[0065] S3.34: Output the generated uncertainty partition map and the corresponding voxel confidence values and interval label data and pass them to step S4 to guide the dynamic decision-making of the path gating mechanism in the adaptive repair network.

[0066] S4: Based on confidence intervals, an adaptive repair network with multi-scale residual paths is constructed, and a path gating mechanism is used to activate different repair paths on demand. It should be noted that the purpose of this step is to dynamically repair the uncertainty areas in the rough reconstruction model through the adaptive repair network, thereby improving the accuracy and quality of the final 3D reconstruction. Specifically, it includes the following steps: S4.1: Construct an adaptive inpainting network driven by uncertainty partition maps. The network consists of multiple multi-scale convolutional blocks with different receptive fields, each corresponding to the inpainting task of an uncertain region at a specific scale. The network receives the original 3D voxel representation and the uncertainty partition map as joint inputs. The uncertainty labels guide the convolutional blocks of different scales to focus on extracting structural information and completing details in high, medium, and low confidence regions. A corresponding residual output is generated at each scale path, representing the contribution of that path to the structural restoration of the original coarse reconstruction model.

[0067] It should be noted that the residual output refers to the residual tensor or voxel compensation value output by the residual path of each scale when performing local repair, which is used to reflect the contribution of the path to the local repair of the coarse reconstruction volume. The residual will participate in the final model fusion.

[0068] S4.2: A path gating mechanism is introduced into the adaptive repair network. This mechanism is embedded in the selection module between multi-scale convolutional paths. Based on the confidence interval labels in the uncertainty partition map, the confidence level of each voxel region is determined. The repair path that matches the confidence level is activated as needed (for example, high-uncertainty regions activate deeper paths, while low-uncertainty regions activate lightweight paths). This enables on-demand resource allocation and precise scheduling of repair tasks. The gating mechanism can implement path control using an attention mechanism or a hard mask mechanism.

[0069] S4.3: Based on the decision of the path gating mechanism, the corresponding repair path is activated, and different repair strategies are performed on voxel regions in the low, medium, and high confidence intervals.

[0070] It should be noted that the repair strategy in the low-confidence interval mainly includes using higher-level network layers for feature refinement and fine repair, such as filling in details through deep convolutional layers; the repair strategy in the medium-confidence interval focuses on balancing repair accuracy and efficiency, and uses moderate network layers for local repair; the repair strategy in the high-confidence interval mainly focuses on surface smoothing and boundary optimization to reduce unnecessary repair operations.

[0071] S5: Fusing the residual output of each repair path with the rough reconstruction model to generate an optimized final three-dimensional reconstruction model.

[0072] Specifically, the residual output of each repair path is fused with the rough reconstruction result to generate an optimized final 3D reconstruction model, including the following steps: S51: Based on the activation results of the path gating mechanism, the residual outputs of each repair path corresponding to the low, medium, and high confidence intervals are extracted, and the residual outputs are weightedly fused with the coarse reconstruction results of the corresponding positions layer by layer to obtain a preliminary fused 3D model, ensuring that the repair information of different confidence intervals can be accurately mapped back to the coarse reconstruction model.

[0073] It should be noted that the layer-by-layer weighted fusion adopts the residual attention fusion module. The residual attention fusion module uses the joint mechanism of spatial attention and channel attention to interactively align the residual output of the repair path with the original voxel feature map, thereby improving the fusion accuracy and structural restoration.

[0074] S52: During the fusion process, each inpainting path is assigned a different weight based on the decision of the path gating mechanism and the confidence value. Inpainting paths in low-confidence areas may require more weight to add finer details, while inpainting paths in high-confidence areas can have their weight appropriately reduced to avoid over-inpainting.

[0075] S53: Based on the fused 3D model, perform post-processing optimization operations, including but not limited to noise removal, surface smoothing, boundary optimization, and other steps, to further improve the quality of the 3D reconstruction results and ensure that the final model is more consistent with the geometric features and details of the actual object.

[0076] S54: Through multiple iterations of the restoration and fusion process, a high-quality, optimized 3D reconstruction model is generated. This model not only has high reconstruction accuracy but also effectively eliminates errors caused by factors such as occlusion and noise, providing more reliable 3D reconstruction results.

[0077] It should be noted that the final optimized 3D reconstruction model is optimized through a joint loss function during the training phase. The joint loss function includes three parts: structural similarity loss, geometric reconstruction error, and uncertainty guidance loss, so as to take into account structural fidelity, local detail restoration capability, and model robustness at the same time.

[0078] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A 3D object reconstruction method combining multi-view fusion and neural network, characterized in that: The following steps are involved: Collect multi-view images of the target object and perform geometric calibration and perspective parameter calibration to generate a standardized image sequence; Inputting the image sequence into a feature extraction and voxel fusion module to obtain a preliminary three-dimensional spatial representation as a coarse reconstruction model; performing uncertainty assessment on the coarse reconstruction model, generating a voxel-level confidence heat map and dividing the confidence heat map into a plurality of confidence intervals; Based on the confidence interval, an adaptive repair network with multi-scale residual paths is constructed, and different repair paths are output and activated on demand using a path gating mechanism; The residual output of each repair path is fused with the rough reconstruction model to generate an optimized final three-dimensional reconstruction model.

2. The 3D object reconstruction method combining multi-view fusion and neural network according to claim 1, characterized in that: The image sequence is input into a feature extraction and voxel fusion module to obtain a preliminary three-dimensional space representation as a coarse reconstruction model, including the following steps: The standardized image sequence is input into the feature extraction module, which extracts image features with deep semantic and geometric expression capabilities from the two-dimensional image through the feature extraction network. The image features include but are not limited to edge features, texture features, and spatial structure information; Construct a 3D voxel grid of a predefined size as the initial structure of the 3D space representation. A fusion mechanism is used to form a 3D voxel representation containing initial occupancy probability, spatial semantic information and coarse geometric structure features for features mapped to the same voxel position from multiple perspectives. The fused 3D voxel representation is used as a coarse reconstruction model.

3. The 3D object reconstruction method combining multi-view fusion and neural network according to claim 2, characterized in that: The fusion mechanism adopts a spatial attention enhanced voxel fusion strategy, which realizes feature weighting and consistency correction under different perspectives by introducing a cross-view attention map and a spatial context encoder.

4. The 3D object reconstruction method combining multi-view fusion and neural network according to claim 1, characterized in that: The uncertainty assessment of the coarse reconstruction model is performed to generate a voxel-level confidence heat map and divide it into a plurality of confidence intervals, including the following steps: Construct a voxel uncertainty quantification model to measure the uncertainty of each voxel unit in the coarse reconstruction model. The model input is the fused three-dimensional space voxel representation, and the output is the confidence score of each voxel. Construct a voxel-level confidence heatmap to spatially express the uncertainty distribution of the entire 3D voxel representation. According to the confidence heat map, the 3D voxel representation is divided into multiple confidence intervals.

5. The 3D object reconstruction method combining multi-view fusion and neural network according to claim 4, characterized in that: The confidence score calculation formula is as follows: in: C i represents the confidence score of the i-th voxel; N i Indicates the number of views mapped to the voxel, that is, how many images produce valid feature projections on it; represents the image feature vector extracted from voxel i at the j-th viewpoint; μ i represents the average value of all view features, represents the variance of the voxel’s features under multiple viewing angles, ω j represents the weight of the j-th perspective; ∈ represents a very small positive number to prevent division by zero errors; R i Indicates the boundary complexity of the voxel in space. The closer to the boundary or detail area, the larger the value. λ represents the balance parameter, which is used to control the influence of structural complexity on the overall confidence; exp(·) is the Sigmoid function, which is used to map the final confidence level to the range of 0-1; The interactive fusion score representing cross-view feature consistency and voxel spatial complexity; It is the L2 norm squared, which is used to measure the gap between the image features of the view and the average features, reflecting the stability of the view; Represents the feature variance of all views at this voxel, which serves as a residual normalization factor to suppress the influence of data scale on the score; N i +λ·R i It represents the joint penalty term of the number of feature perspectives and structural complexity. The more complex the structure, the lower the score. range: C i ∈(0,1), where: When C i When ≈1, it means that the voxel is covered by multiple high-confidence view-stable features and the spatial structure is smooth; When C i When ≈0, it means that the voxel has a large cross-view feature deviation, or is located near the structure boundary and has high uncertainty.

6. The 3D object reconstruction method combining multi-view fusion and neural network according to claim 4, characterized in that: Dividing the three-dimensional voxel representation into a plurality of confidence intervals according to the confidence heat map comprises the following steps: One or more confidence thresholds are determined based on the fuzzy hierarchical algorithm, and the range of voxel confidence values is divided into a high confidence interval, a medium confidence interval, and a low confidence interval. The range of voxel confidence values is 0, 1). The division rules are as follows: 0<low confidence interval<first threshold, this interval indicates that the area has high uncertainty and serious defects, and needs to be reconstructed; The first threshold ≤ medium confidence interval < second threshold, this interval indicates that the area has moderate uncertainty and needs moderate repair; The second threshold is ≤ high confidence interval < 1, which indicates that the regional structure is stable and the semantics are clear; The confidence value of each voxel unit in the voxel-level confidence heat map is compared according to the set threshold, and each voxel is assigned a corresponding interval label; The confidence interval labels of each voxel obtained are integrated to generate an uncertainty partition map that corresponds one-to-one with the original three-dimensional space voxel representation in terms of structure and position information. Each voxel unit in the partition map has a corresponding confidence interval label.

7. The 3D object reconstruction method combining multi-view fusion and neural network according to claim 6, characterized in that: Based on the confidence interval, an adaptive repair network with multi-scale residual paths is constructed, and a path gating mechanism is used to activate different repair paths on demand, including the following steps: An adaptive inpainting network driven by uncertainty partition maps is constructed. The network consists of multiple multi-scale convolutional blocks with different receptive fields, each of which is responsible for inpainting uncertain regions at a specific scale. The network receives the original 3D voxel representation and the uncertainty partition map as joint input. The uncertainty labels guide the convolutional blocks of different scales to focus on extracting structural information and completing details in high, medium, and low confidence regions, and generate corresponding residual outputs at each scale path. A path gating mechanism is introduced into the adaptive repair network, which is embedded in the selection module between multi-scale convolution paths; According to the decision of the path gating mechanism, the corresponding repair path is activated, and different repair strategies are performed on voxel areas in low, medium, and high confidence intervals.

8. The 3D object reconstruction method combining multi-view fusion and neural network according to claim 1, characterized in that: The process of fusing the residual outputs of each repair path with the rough reconstruction model to generate an optimized final 3D reconstruction model includes the following steps: According to the activation results of the path gating mechanism, the residual outputs of each restoration path corresponding to the low, medium, and high confidence intervals are extracted, and the residual outputs are weightedly fused with the coarse reconstruction results of the corresponding positions layer by layer to obtain a preliminary fused 3D model; During the fusion process, different weights are assigned to each repair path based on the decision and confidence value of the path gating mechanism; Based on the fused 3D model, perform post-processing optimization operations; Through multiple iterative repair and fusion processes, the final optimized high-quality 3D reconstruction model is generated.

9. The 3D object reconstruction method combining multi-view fusion and neural network according to claim 8, characterized in that: The layer-by-layer weighted fusion adopts a residual attention fusion module, which interactively aligns the residual output of the repair path and the original voxel feature map through a joint mechanism of spatial attention and channel attention.

10. The 3D object reconstruction method combining multi-view fusion and neural network according to claim 8, characterized in that: The final optimized high-quality 3D reconstruction model is optimized during the training phase using a joint loss function, which includes three parts: structural similarity loss, geometric reconstruction error, and uncertainty guidance loss.

Images