Aviation part refined three-dimensional reconstruction method based on generative model

Abstract

The invention discloses an aviation part refined three-dimensional reconstruction method based on a generative model, and the method comprises the following steps: S1, constructing an aviation part data set, and carrying out the preprocessing of the data set; s2, establishing a coding and decoding framework based on structured potential representation, and in the established coding and decoding framework, realizing three-dimensional data coding through visual feature aggregation and a sparse variational auto-encoder; s3, based on the data set preprocessed in the step S1 and the coding and decoding framework established in the step S2, training of a correction flow model is completed, the correction flow model is used for two-stage generation, sparse structure representation is generated in the first stage, and structured potential representation is generated based on the sparse structure representation and image features in the second stage; and S4, decoding the structured potential representation generated in the step S3 into a three-dimensional representation form through a multi-format decoder, and completing refined three-dimensional reconstruction of the aviation part.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology, specifically to a method for refined 3D reconstruction of aerospace components based on generative models. Background Technology

[0002] Aerospace components are the core foundation for the safety and performance of aircraft and other aircraft. Their design, manufacturing, and testing directly affect the safety of aircraft during flight, the service life of aircraft components, and economic costs. Highly detailed 3D modeling of components is indispensable in process design, digital twin manufacturing, and error detection.

[0003] Currently, the design, inspection, and repair of most components are primarily based on standard engineering drawings or theoretical 3D models. However, in the research and manufacturing of certain cutting-edge components, there is a lack of detailed engineering drawings, with only a basic physical model or image available for reference. Existing 3D reconstruction technologies for components mainly suffer from the following problems:

[0004] (1) Vision systems face the challenges of high equipment costs and high manpower requirements for completing 3D scanning of parts. Most existing 3D reconstruction technologies in industrial scenarios adopt camera-based and point cloud scanning equipment. However, high-precision 3D reconstruction requires high-resolution cameras and scanning equipment, resulting in high hardware costs. The use of scanning equipment is affected by environmental factors such as temperature and lighting changes, making the operating conditions quite harsh. In addition, the 3D reconstruction process involves long manual operation time and high operator training costs, and the scanning results may have defects such as alignment errors, simplification errors, systematic errors, and human errors.

[0005] (2) Three-dimensional reconstruction technology faces challenges such as low accuracy and missing textures in the generated three-dimensional models of parts. Existing methods such as laser scanners establish a three-dimensional reconstruction coordinate system based on camera parameters and scanning light, and perform calculations using traditional stereo vision methods such as geometric relationships, but problems such as missing textures or rough reconstructed surfaces still exist. During the measurement process, changes in temperature, humidity, and light intensity in the scanning environment can lead to a large error between the scanned three-dimensional model and the original part model.

[0006] (3) Deep learning-based 3D reconstruction algorithms suffer from problems such as insufficient generalization ability, insufficient precision, poor texture effect, high memory consumption, and excessive computational cost. Such methods require a large amount of aerospace component data, but the process of collecting aerospace component data samples is often cumbersome, costly, and difficult to label. Even if the dataset problem is solved, accurate 3D reconstruction still faces problems such as excessive computation and poor generalization ability.

[0007] Screws are among the most common components in aerospace parts, serving the functions of structural connection and load transfer. An aircraft uses hundreds of thousands of screws, and the threads and regular structures of these screws encapsulate the main characteristics of most of the components. However, existing 3D reconstruction methods lack the ability to model intricate structures such as threads and grooves, making it difficult to meet the high-precision requirements of aerospace component process design and error detection. Summary of the Invention

[0008] The purpose of this invention is to provide a refined 3D reconstruction method for aerospace components based on generative models, in order to solve the technical problems existing in the background art.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0010] A method for refined 3D reconstruction of aerospace components based on generative models includes the following steps:

[0011] S1. Construct an aviation parts dataset and preprocess the dataset;

[0012] S2. Establish an encoding and decoding framework based on structured latent representation. In the established encoding and decoding framework, three-dimensional data encoding is achieved through visual feature aggregation and sparse variational autoencoder.

[0013] S3. Based on the preprocessed dataset in step S1 and the encoding / decoding framework established in step S2, complete the training of the modified flow model. The modified flow model is used for two-stage generation. The first stage generates sparse structure representation, and the second stage generates structured latent representation based on sparse structure representation and image features.

[0014] S4. The structured latent representation generated in step S3 is decoded into a three-dimensional representation using a multi-format decoder to complete the refined three-dimensional reconstruction of the aerospace parts.

[0015] Furthermore, in step S1, the preprocessing of the dataset specifically includes:

[0016] Multi-view rendering: Based on the Hammersley sampling method, uniformly distributed camera views are generated to perform multi-view image rendering on the 3D data. The rendered image resolution is 192×192 pixels and the color channel is RGB.

[0017] 3D structure voxelization: Converts a continuous polygonal mesh model into a discrete 3D representation composed of voxels. The default voxel mesh resolution is set to 64×64×64, and the average number of active voxels is 20K.

[0018] Image feature extraction: The DINO-V2 model is used to extract features from the rendered image. The three-dimensional voxel points are projected onto the two-dimensional image plane. The image features are mapped to the corresponding voxel positions through bilinear interpolation. The multi-view features associated with the same voxel are aggregated by average pooling.

[0019] Sparse structure encoding: compresses the binary grid of voxel features into a low-resolution feature grid through 3D convolutional blocks;

[0020] Structured latent representation encoding: Combining image depth features and sparse structural representation features, a pre-trained encoder generates structured latent representations;

[0021] Rendering Image Conditions: Generates conditional multi-view rendering images, using randomized view generation logic, with a field of view range of 10 to 70 degrees, for conditional or randomized multi-view rendering.

[0022] Furthermore, in step S2, the structured latent representation is a unified three-dimensional latent representation, which is applicable to three-dimensional structures. Structured latent representation Defined as a set of local latent features on a 3D mesh:

[0023] ;

[0024] in, It is in the three-dimensional mesh and The position index of active voxels at surface intersections. This indicates the local potential features attached to the corresponding voxel. It is the spatial length of the three-dimensional mesh. It is the total number of active voxels. Outline the rough structure and potential features of the three-dimensional structure. Capture the details of appearance and shape.

[0025] Further, in step S2, the visual feature aggregation process is as follows:

[0026] First, each three-dimensional structure Converted to voxelized features ,in Refers to the active voxels of a three-dimensional structure. It is a visual feature that records detailed structural and appearance information of a local area;

[0027] Images of randomly sampled camera views are rendered on a sphere. Feature maps are extracted using a pre-trained DINO-V2 encoder. Each voxel is then projected onto the multi-view feature map to obtain features at the corresponding location, and their average is used as the... .

[0028] Furthermore, in step S2, when establishing the encoding and decoding framework based on structured latent representation, specifically:

[0029] The encoder encodes voxelized features into a structured latent representation, and the decoder transforms the structured latent representation into a specific form of 3D representation;

[0030] A sparse variational autoencoder based on the Transformer architecture is adopted;

[0031] The encoder and decoder share the same Transformer architecture, which serializes the input features of the active voxels and adds sinusoidal position encoding based on their voxel positions, and processes them through Transformer blocks.

[0032] Furthermore, in step S3, the modified flow model learns the optimal transmission path from noise to target data by mapping the complex data distribution to a simple Gaussian distribution. The training of the modified flow model adopts conditional flow matching target.

[0033] Furthermore, in step S3, the specific steps for generating the modified flow model in two stages are as follows:

[0034] The first stage is sparse structure generation. The input image is processed by extracting features through DINO-V2 and fed into the Transformer model based on modified flow along with noise to generate a low-resolution feature grid. The grid is then decoded by the decoder into the position information of active voxels as the sparse structure.

[0035] The second stage is the generation of structured latent representations. The sparse structured representations and image features are input into the sparse modified flow Transformer model to generate structured latent representations.

[0036] Both stages of the Transformer model employ time-step conditional modulation and cross-attention mechanisms to inject image conditional information.

[0037] Further, in step S4, the multi-format decoder includes:

[0038] 3D Gaussian Decoder: Decodes each latent feature into K 3D Gaussian representations, each Gaussian having a position offset, color, scale, opacity, and rotation. The final position of the Gaussian representation is constrained to the vicinity of the active voxel.

[0039] Radiation field decoder: decodes the latent representation into a CP decomposition form of local radiation voxels;

[0040] Mesh Decoder: Decodes the latent representation into flexible cube parameters and signed distance values, improving resolution to [resolution value] through convolutional upsampling blocks. Extract the mesh from the zero isosurface.

[0041] Furthermore, the step S4 is followed by the following step:

[0042] S5. Deploy the trained generative model on a computing device to achieve automatic 3D reconstruction of aerospace parts.

[0043] Furthermore, in step S5, the trained generative model supports both single-view input and multi-view input modes:

[0044] Single-view input mode: Input a single part image, and the model will directly perform two-stage generation after extracting image features;

[0045] Multi-view input mode: Input multiple images of components from different perspectives, and obtain more complete spatial and texture information through multi-view feature fusion.

[0046] Compared with existing technologies, the refined 3D reconstruction method for aerospace components based on generative artificial intelligence models provided by this invention has the following significant technical effects:

[0047] (1) It eliminates the dependence on 3D scanning equipment. Only ordinary cameras are needed to take digital images of parts to complete the fine 3D reconstruction, which greatly reduces hardware costs and operational difficulty;

[0048] (2) By using structured latent representation and two-stage generation method, the spatial structure information and geometric texture information of parts are effectively captured, so as to achieve accurate modeling of fine features such as threads and nail grooves;

[0049] (3) Supports single-view and multi-view input. Multi-view input has better reconstruction effect in terms of detail and geometric consistency compared to single-view input;

[0050] (4) The modified flow model is used for generation, which has a faster convergence speed and lower computational cost compared with the traditional diffusion model;

[0051] (5) Supports outputting various three-dimensional representations such as three-dimensional Gaussian, radiation field, and mesh to meet the needs of different application scenarios;

[0052] (6) It has good generalization ability and can also perform detailed modeling of other aerospace parts such as nuts and gears, in addition to screws.

[0053] Therefore, this invention provides a highly autonomous, high-precision, and practical 3D reconstruction solution for scenarios such as process design and error detection of aerospace components, which has significant engineering application value and promising prospects for promotion. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in this embodiment, the accompanying drawings used in the description of the embodiment will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart illustrating a method for refined 3D reconstruction of aerospace components based on a generative model, provided by the present invention.

[0056] Figure 2 This is a schematic diagram summarizing the overall method in a specific embodiment of the present invention concerning screws;

[0057] Figure 3 A schematic diagram for multi-view rendering;

[0058] Figure 4 This is a schematic diagram of the structure of a sparse VAE encoder / decoder;

[0059] Figure 5 A schematic diagram of the modified flow Transformer structure;

[0060] Figure 6 A schematic diagram of the sparse modified flow Transformer structure;

[0061] Figure 7 This is a single-view screw reconstruction result image;

[0062] Figure 8 This is a multi-view screw reconstruction result image;

[0063] Figure 9 Image showing the result of nut reconstruction;

[0064] Figure 10 This is a diagram showing the result of gear reconstruction. Detailed Implementation

[0065] To make the technical means, creative features, objectives and effects of this invention easier to understand, the following description, in conjunction with the accompanying drawings and specific embodiments, further explains how this invention is implemented.

[0066] Reference Figure 1 As shown, this invention provides a method for refined 3D reconstruction of aerospace components based on generative models, comprising the following steps:

[0067] S1. Construct an aviation parts dataset and preprocess the dataset;

[0068] S2. Establish an encoding and decoding framework based on structured latent representation. In the established encoding and decoding framework, three-dimensional data encoding is achieved through visual feature aggregation and sparse variational autoencoder.

[0069] S3. Based on the preprocessed dataset in step S1 and the encoding / decoding framework established in step S2, complete the training of the modified flow model. The modified flow model is used for two-stage generation. The first stage generates sparse structure representation, and the second stage generates structured latent representation based on sparse structure representation and image features.

[0070] S4. The structured latent representation generated in step S3 is decoded into a three-dimensional representation using a multi-format decoder to complete the refined three-dimensional reconstruction of the aerospace parts.

[0071] Screws are the fundamental connecting elements of core components such as aircraft fuselages, wings, and engines, playing a crucial role in structural connection and load transfer. An aircraft fuselage and wings use hundreds of thousands of screws. The threads and regular structures of screws embody the main characteristics of most components; therefore, the ability of a 3D reconstruction model to create detailed models of screws, to a certain extent, represents the ability to create detailed models of all aerospace components.

[0072] Reference Figure 2 As shown, the following uses a three-dimensional reconstruction model of a screw as an example to introduce a specific embodiment of the present invention.

[0073] In step S1, an aerospace parts (screws) dataset is constructed and preprocessed.

[0074] A screw consists of three parts: the head, the shank, and the end. To enable the model to have good 3D reconstruction capabilities for screws and even the entire component, it is necessary to systematically analyze the structural features of the screw based on its three components, ensuring that the constructed dataset covers the common features of screws.

[0075] Screw head structure analysis: Screw heads can be classified according to their external shape and the groove. Based on head shape, common types include flat head, round head, countersunk head, and hexagonal head, generally resembling regular symmetrical geometric shapes. Based on groove shape, common types include slotted groove, Phillips head, Torx head, internal hexagonal groove, square groove, and internal triangular groove, generally resembling regular grooves with a certain depth.

[0076] Shaft Structure Analysis: The shank of a screw can be divided into two parts: the shank body and the thread. Common shank body types include straight shanks, stepped shanks, knurled shanks, and hollow shanks. The shank body embodies the main structure of the screw and is generally straight and symmetrical about its central axis. Common thread types include full threads and partial threads. The thread reflects the screw's key textural features; whether the thread is continuous, smooth, and conforms to a pattern is crucial for judging the accuracy of a screw's 3D reconstruction result.

[0077] End structure analysis: The ends of screws can be classified as tapered, flat, pointed, and ball-shaped. Although the screw end only accounts for a small part of the overall 3D model of the screw, the reconstruction result of the screw end greatly affects the geometric consistency of the reconstruction result.

[0078] When constructing the screw dataset, to ensure good 3D modeling capabilities for the screws, the dataset needs to consider various features covering the screw head, shank, and end. This embodiment collects and constructs a dataset containing 3D data for 102 screws, ensuring that the number of screws in each category is as similar as possible.

[0079] To ensure good geometric consistency between the model output and the actual screw, some real screw scan 3D data were collected. The overall structure of the screw is not completely regular, and there are some imperfections in the texture, in order to ensure that the model output is consistent with the real situation.

[0080] To ensure the model's structural accuracy and good geometric consistency, as well as the detailed texture of the reconstructed result, some 3D data from CAD modeling was also created and collected. Because these 3D data are from CAD modeling, they generally have regular, aligned structures and detailed textures, effectively improving the model's ability to 3D model screws.

[0081] For a multi-stage generative 3D reconstruction model, a 3D dataset alone is insufficient for training each stage; multi-step preprocessing of the 3D data is required. To avoid confusion in data organization and classification, metadata is constructed to organize the processed data. The basic identifiers of the metadata are the hash value corresponding to the 3D data and the local file path; the processing flow attribute records whether the 3D data underwent rendering, voxelization, or conditional rendering operations; the feature encoding attribute records whether the 3D data has undergone image depth feature extraction using DINO-V2, whether sparse latent representation has been extracted, and whether structured latent representation has been extracted.

[0082] Preprocessing of the dataset specifically includes:

[0083] 1) Multi-view rendering: Generating the sparse latent representation of the model and extracting image depth features requires images of the 3D screw data from different camera angles. Therefore, the 3D screw data needs to be rendered first to generate multi-view images. The fundamental principle of multi-view rendering is to project the 3D model onto multiple 2D image planes through the transformation matrices of different cameras. The correspondence between camera coordinates and the world coordinates of the 3D data is as follows:

[0084] ;

[0085] in A point in the world coordinate system. A point in the camera coordinate system. for

[0086] Transformation matrix, For rotation matrix, It is a translation vector.

[0087] The generation of multi-view rendering perspectives is based on the Hammersley sampling method, a deterministic sampling method based on low-discrepancy sequences that ensures the uniform distribution of the generated viewpoint set in multi-dimensional space. The rendered image resolution is fixed at 192×192 pixels, the color channels are RGB, and the sampling frequency is set to once per simulation step. The multi-view rendering effect of the screw is as follows: Figure 3 As shown.

[0088] 2) 3D Structure Voxelization: Voxelization is the process of converting a continuous polygonal mesh model into a discrete 3D representation composed of voxels. Discretized voxel representations are needed when generating a sparse structure latent representation of the model; therefore, the dataset needs to be voxelized. The default voxel mesh resolution is set to 64×64×64, and the average number of active voxels is 20K.

[0089] 3) Image feature extraction:

[0090] To further capture feature information in the rendered image, feature extraction is needed on the rendered multi-view images to capture global and local information of objects. In this embodiment, the DINO-V2 model is used to extract features from the rendered image. DINO-V2 is a self-supervised learning-based visual feature extraction model whose backbone network is based on VisionTransformer, enabling it to efficiently capture features in the input image. The image feature extraction process mainly consists of the following steps:

[0091] First, image feature extraction is performed, using the DINO-V2 model to learn global semantic features and local detail features;

[0092] Then, the voxels are projected onto the image plane, forming three-dimensional voxel points. Projected onto two-dimensional image plane coordinates The process is achieved through the following formula:

[0093] ;

[0094] in Here are the parameters of the rotation matrix. For translation vector parameters, For the camera intrinsic parameter matrix, and This refers to the camera's external parameters.

[0095] Next, bilinear interpolation is used to map the image features to the corresponding voxel locations to smoothly obtain sub-pixel level features:

[0096] ;

[0097] in For target voxel features, For image features, Weights are assigned to neighboring pixels;

[0098] Finally, average pooling of voxel features is performed to aggregate multiple features associated with the same voxel into a unified voxel feature representation:

[0099] .

[0100] in The voxel characteristics are those after average pooling polymerization.

[0101] 4) Sparse structure encoding: The binary grid of voxel features is compressed into a low-resolution feature grid through three-dimensional convolutional blocks.

[0102] 5) Structured latent representation encoding: Combining image depth features and sparse structure representation features, a pre-trained encoder generates a structured latent representation.

[0103] 6) Rendering Image Conditions: Unlike the fixed perspective generation of multi-view rendering, image conditions use randomized perspective generation logic. Its function is to generate conditional multi-view rendering images with a field of view range of 10 to 70 degrees. It is used for conditional or randomized multi-view rendering to inspire the model to explore details more finely.

[0104] In step S2, an encoding and decoding framework based on structured latent representation is established. In the established encoding and decoding framework, three-dimensional data encoding is achieved through visual feature aggregation and sparse variational autoencoder.

[0105] The structured latent representation is a unified 3D latent representation used for high-quality, multifunctional 3D generation, and is suitable for 3D structures. Structured latent representation Defined as a set of local latent features on a 3D mesh:

[0106] ;

[0107] in, It is in the three-dimensional mesh and The position index of active voxels at surface intersections. This indicates the local potential features attached to the corresponding voxel. It is the spatial length of the three-dimensional mesh. It is the total number of active voxels. Outline the rough structure and potential features of the three-dimensional structure. Capture details of appearance and shape. This embodiment uses default settings. ,average value .

[0108] The visual feature aggregation process is as follows:

[0109] First, each three-dimensional structure Converted to voxelized features ,in Refers to the active voxels of a three-dimensional structure. These are visual features that record detailed structural and appearance information of local regions. This is used to derive the characteristics of each active voxel. The image of a randomly sampled camera view is rendered on a sphere. A feature map is extracted using a pre-trained DINO-V2 encoder. Then, each voxel is projected onto the multi-view feature map to obtain the features at the corresponding location, and their average is used as the... .

[0110] When establishing an encoding and decoding framework based on structured latent representations, specifically:

[0111] encoder voxelization features Encoding as a structured latent representation decoder Will The representation is transformed into a specific form of 3D representation. A sparse variational autoencoder based on the Transformer architecture is employed. The encoder and decoder share the same Transformer architecture. To handle sparse voxels, the input features of active voxels are serialized, and sinusoidal positional encodings are added according to their voxel positions to create variable-length tokens of context length L, which are then processed through Transformer blocks. Furthermore, a sliding window attention mechanism is added in 3D space to enhance the interaction of local information, which improves efficiency compared to the full attention implementation.

[0112] During training, a reconstruction loss is applied between the decoded 3D data and the baseline ground truth to train the encoder and decoder in an end-to-end manner, while simultaneously adjusting the output. Applying KL penalty makes the distribution tend to follow a normal distribution regularization.

[0113] Furthermore, the established decoder is a multi-format decoder, including a 3D Gaussian decoder. Radiation field decoder and mesh decoder All three have the same structure except for the output layer.

[0114] 1) 3D Gaussian Decoder: This decodes each... Decoded as Each of the three-dimensional Gaussian representations has a position offset. ,color ,scale Opacity and rotation In order to maintain Positionality, Gauss's final position It is limited to the vicinity of its active voxels. The reconstruction loss includes the difference between the rendered image, which is a 3D Gaussian representation, and the reference ground truth image. Loss, D-SSIM loss, and LPIPS loss.

[0115] 2) Radiation field decoder: Decodes the structured latent representation into CP decomposition of local radiation voxels. The reconstruction loss is similar to that of the 3D Gaussian representation.

[0116] 3) Mesh Decoder: Decodes the structured latent representation into flexible cube parameters and signed distance values ​​at the eight vertices of the corresponding voxels. The mesh decoder appends two convolutional upsampling blocks after the Transformer backbone to increase the final output resolution. Extract the mesh from the zero isosurface.

[0117] In subsequent training, due to the refinement and high efficiency of the 3D Gaussian representation, it is used for end-to-end learning of the encoder and decoder. For other output formats, the learned encoder is frozen, and the corresponding decoder is trained.

[0118] In step S3, the training of the modified flow model is completed based on the preprocessed dataset in step S1 and the encoding / decoding framework established in step S2. The modified flow model is used for two-stage generation: the first stage generates sparse structure representation, and the second stage generates structured latent representation based on sparse structure representation and image features.

[0119] The modified flow model learns the optimal transmission path from noise to target data by mapping complex data distribution to a simple Gaussian distribution. The training of the modified flow model adopts conditional flow matching target.

[0120] The modified flow model is a novel generative method that can significantly improve computational efficiency while increasing generation accuracy. Modified flow is a time-varying generative approach. Ordinary differential model on:

[0121] ;

[0122] It will follow Empirical observations of distribution Transform into compliance Empirical observations of distribution Drift force function The setting is to make the flow follow the path from... point to The direction of flow along a linear path. Drift force. Obtaining this requires solving the least squares regression problem:

[0123] ;

[0124] Training uses conditional flow matching of targets:

[0125] ;

[0126] in This is the original data sample. This is a noise vector, typically sampled from a standard Gaussian distribution. For time steps, the range of values ​​is... , For the velocity field predicted by the model, These are the model parameters.

[0127] The specific steps for generating the modified flow model in two stages are as follows:

[0128] Phase 1: Sparse structure generation, refer to... Figure 2 The part about the modified flow Transformer extracts features from the input image using DINO-V2, and inputs them along with noise into the modified flow-based Transformer model to generate a low-resolution feature grid. This grid is then decoded by a decoder into the position information of active voxels as a sparse structure.

[0129] The second stage is the generation of structured latent representations. The sparse structured representations and image features are input into the sparse modified flow Transformer model to generate structured latent representations.

[0130] Both stages of the Transformer model employ time-step conditional modulation and cross-attention mechanisms to inject image conditional information.

[0131] In the first stage, sparse structure refers to the positional information of voxels intersecting with the surface of the 3D data contained in the image. During the sparse structure generation stage, image features initially encoded and extracted from the input image, along with noise, are input into a modified flow model for sampling, resulting in a rough, low-resolution compressed mesh of the 3D data surface. These features are then input into the sparse structure decoder to decode the positional information of the active voxels on the 3D data surface.

[0132] Specifically, the goal of the first-stage generative model is to generate sparse structures. To achieve this goal using tensor neural networks, the sparse active voxels first need to be converted into a dense binary 3D mesh. If the voxel is an active voxel, set its value to 1; otherwise, set it to 0. Generate a dense mesh directly. The computational cost is very high, therefore a simple sparse VAE encoder / decoder is introduced, referencing... Figure 4 As shown, it is compressed into a low-resolution feature grid using 3D convolutional blocks. .

[0133] Reference Figure 5 As shown, a Transformer backbone based on modified flow is introduced. Used to generate low-resolution feature meshes The input high-density noisy mesh is serialized, combined with positional encoding, and fed into a multi-head attention layer for denoising. For image conditions, DINO-V2 is used to extract visual features. Adaptive layer normalization and gating mechanisms are employed to incorporate temporal step information into the network, while conditional information (visual features extracted by DINO-V2) is injected as keys and values ​​through a cross-attention layer. The denoised feature mesh is then presented. Decoded into discrete grid And further converted back into active voxels. As the final sparse structure.

[0134] In the second stage, after obtaining the rough and sparse structure of the 3D data surface, the model further generates a structured latent representation by combining the sparse structure with the image feature information obtained from multi-view rendering, DINO-V2 feature extraction, and feature aggregation through a sparse correction flow Transformer. At this point, the structured latent representation already contains all features of the target to be reconstructed, including its spatial structure and specific texture.

[0135] Reference Figure 6 As shown, specifically, a sparse modified flow Transformer is used. Representing sparse structures Generate structured latent representations Efficiency is improved by packaging the input noisy latent representation into shorter sequences before serialization. This is achieved by applying downsampling blocks with sparse convolutions for packaging. The latent representation within a local region is then added, followed by multiple temporally modulated Transformer blocks. Convolutional upsampling blocks are appended to the ends of the Transformers, with skip connections to the downsampling blocks to facilitate spatial information flow. Temporal steps are integrated through adaptive normalization layers, and image conditions are injected via cross-attention.

[0136] After model training, structured latent representation It can be generated sequentially from two models.

[0137] In step S4, the structured latent representation generated in step S3 is decoded into various three-dimensional formats by a multi-format decoder to complete the refined three-dimensional reconstruction of aerospace components.

[0138] Referring to the multi-format decoder established in step S2 above, step S4 specifically uses a three-dimensional Gaussian decoder. Radiation field decoder and mesh decoder Decodes into various 3D formats.

[0139] The step S4 is followed by the following step:

[0140] S5. Deploy the trained generative model on a computing device to achieve automatic 3D reconstruction of aerospace components. The trained generative model supports both single-view and multi-view input modes.

[0141] Single-view input mode: Input a single image of a component; the model extracts image features and directly performs a two-stage generation. The single-view screw reconstruction result is shown below. Figure 7 As shown.

[0142] Multi-view input mode: Input multiple images of the component from different perspectives. Through multi-view feature fusion, more complete spatial and textural information is obtained, improving reconstruction accuracy and geometric consistency. The multi-view screw reconstruction result is shown below. Figure 8 As shown.

[0143] The method proposed in this invention has the capability to create detailed models of aerospace components. For screws, the model can accurately reproduce the overall structure. It is understood that, in other embodiments, this invention can also be applied to other aerospace components such as nuts and gears. Figure 9 The results of nut reconstruction are shown. Figure 10The gear reconstruction results are shown, demonstrating that the model possesses strong capabilities for detailed 3D reconstruction of these components. Compared to existing methods, the method of this invention significantly improves reconstruction accuracy, detail fidelity, and computational efficiency.

[0144] In summary, the generative model-based refined 3D reconstruction method for aerospace components provided by this invention, by introducing a modified flow model and structured latent representation into the 3D reconstruction task of aerospace components, achieves stable, efficient, and refined 3D reconstruction relying solely on image observation. This significantly improves the automation capability and accuracy of 3D reconstruction of aerospace components, and is suitable for refined 3D modeling tasks of complex structural components before process design, error detection, or pre-assembly. As a novel, intelligent, and highly reliable 3D reconstruction method for aerospace components, this invention provides technical support for improving the intelligence and automation level of aerospace manufacturing.

[0145] Finally, it should be noted that the above description is only an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for refined 3D reconstruction of aerospace components based on generative models, characterized in that, Includes the following steps: S1. Construct an aviation parts dataset and preprocess the dataset; S2. Establish an encoding and decoding framework based on structured latent representation. In the established encoding and decoding framework, three-dimensional data encoding is achieved through visual feature aggregation and sparse variational autoencoder. S3. Based on the preprocessed dataset in step S1 and the encoding / decoding framework established in step S2, complete the training of the modified flow model. The modified flow model is used for two-stage generation. The first stage generates sparse structure representation, and the second stage generates structured latent representation based on sparse structure representation and image features. S4. The structured latent representation generated in step S3 is decoded into a three-dimensional representation using a multi-format decoder to complete the refined three-dimensional reconstruction of the aerospace parts.

2. The method for refined 3D reconstruction of aerospace components based on generative models according to claim 1, characterized in that, In step S1, the preprocessing of the dataset specifically includes: Multi-view rendering: Based on the Hammersley sampling method, uniformly distributed camera views are generated to perform multi-view image rendering on the 3D data. The rendered image resolution is 192×192 pixels and the color channel is RGB. 3D structure voxelization: Converts a continuous polygonal mesh model into a discrete 3D representation composed of voxels. The default voxel mesh resolution is set to 64×64×64, and the average number of active voxels is 20K. Image feature extraction: The DINO-V2 model is used to extract features from the rendered image. The three-dimensional voxel points are projected onto the two-dimensional image plane. The image features are mapped to the corresponding voxel positions through bilinear interpolation. The multi-view features associated with the same voxel are aggregated by average pooling. Sparse structure encoding: compresses the binary grid of voxel features into a low-resolution feature grid through 3D convolutional blocks; Structured latent representation encoding: Combining image depth features and sparse structural representation features, a pre-trained encoder generates structured latent representations; Rendering Image Conditions: Generates conditional multi-view rendering images, using randomized view generation logic, with a field of view range of 10 to 70 degrees, for conditional or randomized multi-view rendering.

3. The method for refined 3D reconstruction of aerospace components based on generative models according to claim 2, characterized in that, In step S2, the structured latent representation is a unified three-dimensional latent representation, which is applicable to three-dimensional structures. Structured latent representation Defined as a set of local latent features on a 3D mesh: ； in, It is in the three-dimensional mesh and The position index of active voxels at surface intersections. This indicates the local potential features attached to the corresponding voxel. It is the spatial length of the three-dimensional mesh. It is the total number of active voxels. Outline the rough structure and potential features of the three-dimensional structure. Capture the details of appearance and shape.

4. The method for refined 3D reconstruction of aerospace components based on generative models according to claim 3, characterized in that, In step S2, the visual feature aggregation process is as follows: First, each three-dimensional structure Converted to voxelized features ,in Refers to the active voxels of a three-dimensional structure. It is a visual feature that records detailed structural and appearance information of a local area; Images of randomly sampled camera views are rendered on a sphere. Feature maps are extracted using a pre-trained DINO-V2 encoder. Each voxel is then projected onto the multi-view feature map to obtain features at the corresponding location, and their average is used as the... .

5. The method for refined 3D reconstruction of aerospace components based on generative models according to claim 4, characterized in that, In step S2, when establishing the encoding and decoding framework based on structured latent representation, specifically: The encoder encodes voxelized features into a structured latent representation, and the decoder transforms the structured latent representation into a specific form of 3D representation; A sparse variational autoencoder based on the Transformer architecture is adopted, and the encoder and decoder share the same Transformer architecture. The input features of the active voxels are serialized and sinusoidal position codes are added according to their voxel positions, and then processed by Transformer blocks.

6. The method for refined 3D reconstruction of aerospace components based on generative models according to claim 5, characterized in that, In step S3, the modified flow model learns the optimal transmission path from noise to target data by mapping the complex data distribution to a simple Gaussian distribution. The training of the modified flow model adopts conditional flow matching target.

7. The method for refined 3D reconstruction of aerospace components based on generative models according to claim 6, characterized in that, In step S3, the specific steps for generating the modified flow model in two stages are as follows: The first stage is sparse structure generation. The input image is processed by extracting features through DINO-V2 and fed into the Transformer model based on modified flow along with noise to generate a low-resolution feature grid. The grid is then decoded by the decoder into the position information of active voxels as the sparse structure. The second stage is the generation of structured latent representations. The sparse structured representations and image features are input into the sparse modified flow Transformer model to generate structured latent representations. Both stages of the Transformer model employ time-step conditional modulation and cross-attention mechanisms to inject image conditional information.

8. The method for refined 3D reconstruction of aerospace components based on generative models according to claim 7, characterized in that, In step S4, the multi-format decoder includes: 3D Gaussian Decoder: Decodes each latent feature into K 3D Gaussian representations, each Gaussian having a position offset, color, scale, opacity, and rotation. The final position of the Gaussian representation is constrained to the vicinity of the active voxel. Radiation field decoder: decodes the latent representation into a CP decomposition form of local radiation voxels; Mesh Decoder: Decodes the latent representation into flexible cube parameters and signed distance values, improving resolution to [resolution value] through convolutional upsampling blocks. Extract the mesh from the zero isosurface.

9. The method for refined 3D reconstruction of aerospace components based on generative models according to claim 1, characterized in that, The step S4 is followed by the following step: S5. Deploy the trained generative model on a computing device to achieve automatic 3D reconstruction of aerospace parts.

10. The method for refined 3D reconstruction of aerospace components based on generative models according to claim 9, characterized in that, In step S5, the trained generative model supports both single-view input and multi-view input modes: Single-view input mode: Input a single part image, and the model will directly perform two-stage generation after extracting image features; Multi-view input mode: Input multiple images of components from different perspectives, and obtain more complete spatial and texture information through multi-view feature fusion.

Images