3D model synthesis method based on improved transformer

Abstract

The application provides a 3D model synthesis method based on an improved Transformer, comprising the following steps: collecting multi-dimensional 2D images of a large number of aircrafts and corresponding real 3D model data, and establishing a 2D image dataset and a corresponding real 3D model dataset; extracting data features of 2D images in the 2D image dataset and 3D sub-models of the corresponding real 3D model dataset; training an improved Transformer model based on the data features; inputting preprocessed 2D image data of an aircraft to be synthesized into the trained improved Transformer model for processing, and obtaining a synthesized 3D model of the aircraft. The application solves the problems of high acquisition cost, poor scene adaptability, low acquisition efficiency and model precision that cannot meet the requirements of the aviation profession when 3D data of an aircraft is acquired in the aviation field. The improved Transformer model does not need to rely on expensive 3D acquisition equipment, and can quickly and accurately synthesize a three-dimensional model only through 2D images of an aircraft.

Description

Technical Field

[0001] This invention relates to the field of digital modeling technology for aircraft, and in particular to a 3D model synthesis method based on an improved Transformer, which can be applied to aerospace engineering and research fields such as reverse design of aircraft parts, construction of digital twins of the whole aircraft, aircraft maintenance and inspection and damage assessment, airworthiness certification, and retrofitting and upgrading of old aircraft models. Background Technology

[0002] As high-end and complex equipment, aircraft rely heavily on high-precision, high-fidelity 3D digital models of the entire aircraft and its components throughout their entire lifecycle, from research and development, design, manufacturing, operation and maintenance, modification and upgrading, to airworthiness certification. In the forward development phase, 3D digital models are the core data foundation for aerodynamic simulation, structural strength verification, multiphysics coupling analysis, and avionics system layout design. In manufacturing, 3D models are the core basis for CNC machining, tooling and fixture design, assembly process planning, and quality inspection. In operation and maintenance, accurate 3D models are essential for detecting wear, deformation, and impact damage to critical components such as fuselage skin, engine blades, landing gear, and fuselage frames of in-service aircraft, enabling quantitative damage assessment, remaining service life prediction, and maintenance plan development. In the modification and airworthiness certification of older aircraft, many in-service older models lack complete original digital models, necessitating reverse engineering to obtain compliant 3D digital models to meet the rigid requirements of civil aviation airworthiness review and modification design.

[0003] Currently, the acquisition of 3D models of aircraft and components in the aviation field mainly relies on specialized 3D acquisition equipment, including industrial-grade LiDAR, high-precision blue light 3D scanners, and RGB-D depth cameras. For the overall shape of an aircraft, ground-based LiDAR is typically used for multi-station scanning and point cloud stitching to obtain the overall point cloud data and then reconstruct the 3D model. For small and complex components such as engine turbine blades, precision connectors, and hydraulic valve bodies, industrial-grade 3D scanners are used for sub-millimeter precision scanning. For the internal structure of the aircraft cabin and local structures at maintenance sites, portable RGB-D cameras are used for on-site data acquisition and model reconstruction.

[0004] However, the aforementioned existing technologies have significant limitations in aircraft research and engineering applications, and cannot meet the needs of the aviation field for full-scenario, low-cost, high-efficiency, and high-precision 3D model construction. The core defects are reflected in the following aspects: The acquisition costs are extremely high, and the adaptability to different scenarios is extremely poor: the procurement costs of industrial-grade 3D scanning equipment and high-precision lidar applicable to the aviation field are often hundreds of thousands to millions of yuan, and the operation of the equipment requires professionally trained technicians; scanning the entire aircraft requires a closed special site and multiple rounds of point calibration and environmental matting processing, which has an extremely high implementation threshold; for small and enclosed spaces such as engine cavities, wing fuel tanks, and fuselage frames, existing 3D sensors cannot effectively enter to complete data acquisition, resulting in the inability to acquire 3D data for a large number of special aviation scenarios, and the availability of effective data is extremely low.

[0005] Low data acquisition efficiency cannot meet the needs of aviation emergency scenarios: For emergency scenarios such as line maintenance, scheduled maintenance, and aircraft structural damage assessment at accident sites for in-service aircraft, existing 3D sensors need to complete multiple processes such as equipment debugging, multi-station scanning, point cloud denoising, stitching and fusion, and coordinate calibration. Data acquisition for a single civil aircraft takes several hours to several days, which cannot achieve rapid 3D model construction and is difficult to support the timeliness requirements of aviation emergency maintenance, rapid accident analysis, and flight support.

[0006] The accuracy of model reconstruction is limited by the quality of the acquired data, and its robustness is seriously insufficient: For typical aerospace structures such as complex free-form surfaces of aircraft skin, composite curved and twisted surfaces of engine blades, and irregular landing gear structures, existing 3D sensors are prone to problems such as scanning blind spots, missing point cloud data, and environmental noise interference. This directly leads to problems such as surface distortion, loss of structural features, and disordered assembly topology in the subsequently reconstructed 3D model. It cannot meet the aerospace field's requirements for millimeter-level or even sub-millimeter-level accuracy of models, and it is difficult to support core aerospace research work such as aerodynamic simulation, structural strength verification, and fatigue life analysis.

[0007] Fourth, existing general 2D to 3D conversion technologies cannot meet the professional technical requirements of the aviation field. Current general 2D image to 3D model conversion technologies are mostly designed for general civilian objects and simple geometric structures. They do not take into account the industry characteristics of high precision, high complexity, strong topological correlation, and strict aerodynamic and structural constraints of aircraft structures. As a result, they have problems such as poor model generalization ability, low accuracy in reproducing complex curved surfaces, and inability to adapt to the precision assembly relationships of aerospace components. The synthesized 3D models cannot meet the professional requirements of aviation research and engineering applications, and have not yet been effectively applied in the aviation field.

[0008] Therefore, this invention proposes to develop a 3D model synthesis method based on an improved Transformer. Summary of the Invention

[0009] This invention provides a 3D model synthesis method based on an improved Transformer, addressing the problems of high acquisition costs, poor scene adaptability, low acquisition efficiency, and insufficient model accuracy to meet aviation professional requirements when acquiring 3D aircraft data in the aviation field. Using the improved Transformer model, 3D models can be synthesized quickly and with high precision from 2D aircraft images without relying on expensive 3D acquisition equipment.

[0010] This invention provides a 3D model synthesis method based on an improved Transformer, comprising: Step 1: Collect a large number of multi-dimensional 2D images of aircraft and their corresponding real 3D model data, and establish a 2D image dataset and its corresponding real 3D model dataset. Step 2: Extract data features from the 2D images in the 2D image dataset and the corresponding 3D sub-models in the real 3D model dataset; Step 3: Train the improved Transformer model based on the data features; Step 4: After preprocessing the 2D image data of the aircraft to be synthesized, input it into the improved Transformer model based on the training for processing to obtain the synthesized 3D aircraft model.

[0011] Preferably, in a 3D model synthesis method based on an improved Transformer, step 1 includes: Based on big data collection, a massive amount of 2D images and corresponding real 3D model data of aircraft, fuselage skin, engines, landing gear, avionics systems, hydraulic systems and various precision components are collected. 2D images are detected and screened to obtain high-quality 2D images, and these high-quality 2D images are then preprocessed to obtain optimized 2D images. Based on optimizing the correspondence between the entire aircraft or its components in the 2D images and the 3D model data, as well as the association of the assembly topology data between the various 3D models, a 2D image dataset and its corresponding real 3D model dataset are established.

[0012] Preferably, in a 3D model synthesis method based on an improved Transformer, step 3 includes: Based on optimizing the correspondence between common pixels on 2D images and point clouds on 3D models, the data features of 2D images and their corresponding 3D sub-models are paired to generate model training sets and model validation sets. The improved Transformer model is trained using the model training set and the model validation set to obtain a well-trained improved Transformer model.

[0013] Preferably, in a 3D model synthesis method based on an improved Transformer, the improved Transformer model includes: A fixed window attention mechanism is added to the self-attention module of the improved Transformer model. The data features input into the improved Transformer model are divided into windows of a preset size, and the self-attention of the data features in each window is calculated to generate a feature map. The feature map is aggregated to obtain a reduced feature map, which is then sent to a multilayer perceptron for processing.

[0014] Preferably, in a method for synthesizing 3D models based on an improved Transformer, an improved Transformer model is trained using a model training set and a model validation set to obtain a trained improved Transformer model, including: The model training set is input into the improved Transformer model to synthesize the target 3D model. The target 3D model is then compared with the real 3D model corresponding to the model validation set to obtain the model differences. Based on the aforementioned model differences, a preset loss function is used to continuously adjust the model parameters of the improved Transformer model through backpropagation algorithm until the optimal solution is obtained. Then, the training of the improved Transformer model ends, and a well-trained improved Transformer model is obtained.

[0015] Preferably, in a method for synthesizing 3D models based on an improved Transformer, training an improved Transformer model based on a model training set and a model validation set further includes: Based on the type of real 3D model corresponding to the model validation set, the model training set and the model validation set are classified, and convertible model labels are generated based on the classification results. Clustering is performed on the model training set and model validation set based on convertible model labels to obtain the final model training set and model validation set corresponding to the whole aircraft, various systems and component scenarios.

[0016] Preferably, in a 3D model synthesis method based on an improved Transformer, the target 3D model is compared with the corresponding real 3D model on the model validation set to obtain model differences, including: The model network structure is traversed to obtain the intersection lines between different faces of the target 3D model and the real 3D model, and the coordinate information of the intersection lines is obtained. The coordinate information of the intersecting lines corresponding to the target 3D model and the real 3D model is compared to obtain the intersecting line error. Based on the surfaces corresponding to the intersecting lines, the intersecting lines are clustered to obtain multiple intersecting line groups. Based on the curvature of each surface and the complexity of the intersecting lines, the error coefficient corresponding to the intersecting line group for each surface is determined. The average intersecting line error corresponding to each intersecting line group is calculated, and the error degree corresponding to the current surface is determined by combining the error coefficient. Furthermore, by combining the structural position of each face within the aircraft or its components, the relationships between the faces are determined, and error weights are allocated based on these relationships to determine the corresponding error weight for each face. Based on the error weights and the error degree of each face, the final error of each face is determined and labeled on each face.

[0017] Preferably, in a 3D model synthesis method based on an improved Transformer, 2D images are detected and screened to obtain high-quality 2D images, including: The gradient magnitude of pixels in a 2D image is calculated based on the Sobel operator, and the gradient magnitude is compared with a preset magnitude. Pixels with a gradient magnitude greater than or equal to the preset magnitude are considered as qualified points. Pixels with gradient magnitudes less than a preset magnitude are designated as blurred points. Calculate the proportion of blurred points in the 2D image. When the proportion of blurred points is greater than a preset threshold, the 2D image is determined to be blurred, the 2D image is deleted, and the image data in the 2D image dataset is updated to obtain a new 2D image dataset. Based on the real 3D model dataset, the model framework lines of the real 3D model and the assembly connection relationship between each model framework line are determined. The endpoint features of the model framework lines are extracted and marked with serial numbers on the 2D images in the corresponding 2D image dataset of the real 3D model data. Based on the endpoint features of the model framework lines, it is determined whether each line is fully displayed on the 2D image. If so, the fully displayed lines are marked in green; otherwise, the incompletely displayed lines are marked in red. The 2D images marked in red are taken as the images to be processed. Based on the serial number markings of the model framework lines, the correspondence between the lines in each 2D image is determined, and the associated images corresponding to the incomplete lines in the images to be processed are obtained. The shooting angles of the image to be processed and the associated image are obtained respectively. Based on the shooting angles and the line presentation effect of the incomplete display line on the real 3D model, it is determined whether the incomplete line is fully presented by the combination of the image to be processed and the associated image. If so, the incomplete lines on the image to be processed and the associated image are marked in green, and the connection points of the incomplete lines are determined in the image to be processed and the associated image respectively, as clipping reference points; If not, then based on the assembly and connection relationship between the lines of each model framework, determine the associated lines of the incomplete lines, and infer the endpoints of the incomplete display lines based on the associated lines. If the inference is successful, then determine that the incomplete lines are inferable complete lines and mark them in green. The 2D images with all green calibration lines are considered high-quality 2D images, and the 2D image dataset is updated accordingly.

[0018] Preferably, in a 3D model synthesis method based on an improved Transformer, when there are red marker lines on the 2D image, a re-acquisition area is determined on the real 3D model based on the line number corresponding to the red marker lines, and an acquisition command is sent to the image acquisition module to re-acquire the 2D image corresponding to the re-acquisition area.

[0019] Preferably, a 3D model synthesis method based on an improved Transformer further includes: Model data packets generated based on the synthetic 3D model are sent to the user according to a preset communication address.

[0020] Compared with the prior art, the present invention has at least the following beneficial effects: This invention collects massive amounts of multi-dimensional 2D images of aircraft (including the entire aircraft, fuselage skin, engines, landing gear, avionics systems, hydraulic systems, and various precision components) and their corresponding real 3D model data, establishing a 2D image dataset and its corresponding real 3D model dataset. Then, it extracts data features from the 2D images within the 2D image dataset and the 3D sub-models within the corresponding real 3D model dataset. Based on these data features, it trains an improved Transformer model, obtaining an improved Transformer model capable of converting from 2D to 3D. Only 2D images need to be input to synthesize 3D models, significantly reducing acquisition costs and eliminating reliance on expensive 3D acquisition equipment. This greatly reduces the cost of acquiring 3D models in the aviation field, making the synthesis of aircraft 3D models more flexible and improving the efficiency and scene adaptability of aircraft 3D model synthesis. Only the input 2D images need to be adjusted to generate various 3D models required for different scenarios throughout the entire lifecycle of an aircraft.

[0021] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in this application.

[0022] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0023] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 A flowchart of a 3D model synthesis method based on an improved Transformer; Figure 2 This is a flowchart of step 1 of a 3D model synthesis method based on an improved Transformer; Figure 3 This is a flowchart of step 3 of a 3D model synthesis method based on an improved Transformer. Detailed Implementation

[0024] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0025] Example 1: This invention provides a 3D model synthesis method based on an improved Transformer, such as... Figure 1 As shown, it includes: Step 1: Collect a large number of multi-dimensional 2D images of aircraft and their corresponding real 3D model data, and establish a 2D image dataset and its corresponding real 3D model dataset. Step 2: Extract data features from the 2D images in the 2D image dataset and the corresponding 3D sub-models in the real 3D model dataset; Step 3: Train the improved Transformer model based on the data features; Step 4: After preprocessing the 2D image data of the aircraft to be synthesized, input it into the improved Transformer model based on the training for processing to obtain the synthesized 3D aircraft model.

[0026] The beneficial effects of the above technical solution are as follows: This invention collects massive amounts of multi-dimensional 2D images of aircraft (including the entire aircraft, fuselage skin, engines, landing gear, avionics systems, hydraulic systems, and various precision components) and their corresponding real 3D model data, and establishes a 2D image dataset and its corresponding real 3D model dataset. Then, it extracts data features from the 2D images within the 2D image dataset and the 3D sub-models within the corresponding real 3D model dataset. Based on these data features, it trains an improved Transformer model to obtain an improved Transformer model capable of converting from 2D to 3D. Only 2D images need to be input to synthesize 3D models, greatly reducing acquisition costs and eliminating reliance on expensive 3D acquisition equipment. This significantly reduces the cost of acquiring 3D models in the aviation field, making the synthesis of aircraft 3D models more flexible and improving the efficiency and scene adaptability of aircraft 3D model synthesis. Only the input 2D images need to be adjusted to generate various 3D models required for different scenarios throughout the entire lifecycle of an aircraft.

[0027] Example 2: Based on Example 1, step 1, as follows Figure 2 As shown, it includes: Step 101: Based on big data, collect massive amounts of 2D images and corresponding real 3D model data of aircraft, fuselage skin, engines, landing gear, avionics systems, hydraulic systems and various precision components; Step 102: Detect and filter 2D images to obtain high-quality 2D images, and preprocess the high-quality 2D images to obtain optimized 2D images; Step 103: Based on the correspondence between the entire aircraft or its components in the optimized 2D image and the 3D model data, as well as the association of the assembly topology data between the various 3D models, establish a 2D image dataset and its corresponding real 3D model dataset.

[0028] In this embodiment, preprocessing includes, but is not limited to, operations such as cropping, scaling and grayscale conversion, noise reduction, and contrast enhancement.

[0029] The beneficial effects of the above technical solution are as follows: This invention detects and filters the acquired 2D images to obtain high-quality 2D images, ensuring the image quality used for model training and thus improving the model training effect; and based on optimizing the correspondence between 2D images and objects / scenes and 3D model data, as well as the data association between various 3D models, it establishes a 2D image dataset and its corresponding real 3D model dataset, establishing a correspondence between 2D image data and 3D model data of aircraft, which is beneficial for constructing 3D models that meet actual engineering requirements based on real aircraft data, and also beneficial for quickly locating model error positions during model training, improving the training optimization effect of local model structures.

[0030] Example 3: Based on Example 2, 2D images are detected and screened to obtain high-quality 2D images, including: The gradient magnitude of pixels in a 2D image is calculated based on the Sobel operator, and the gradient magnitude is compared with a preset magnitude. Pixels with a gradient magnitude greater than or equal to the preset magnitude are considered as qualified points. Pixels with gradient magnitudes less than a preset magnitude are designated as blurred points. Calculate the proportion of blurred points in the 2D image. When the proportion of blurred points is greater than a preset threshold, the 2D image is determined to be blurred, the 2D image is deleted, and the image data in the 2D image dataset is updated to obtain a new 2D image dataset. Based on the real 3D model dataset, the model framework lines of the real 3D model and the connection relationship between each model framework line are determined. The endpoint features of the model framework lines are extracted and marked with serial numbers on the 2D images in the corresponding 2D image dataset of the real 3D model data. Based on the endpoint features of the model framework lines, it is determined whether each line is fully displayed on the 2D image. If so, the fully displayed lines are marked in green; otherwise, the incompletely displayed lines are marked in red. The 2D images marked in red are taken as the images to be processed. Based on the serial number markings of the model framework lines, the correspondence between the lines in each 2D image is determined, and the associated images corresponding to the incomplete lines in the images to be processed are obtained. The shooting angles of the image to be processed and the associated image are obtained respectively. Based on the shooting angles and the line presentation effect of the incomplete line on the real 3D model, it is determined whether the incomplete line is completely presented by the combination of the image to be processed and the associated image. If so, the incomplete lines on the image to be processed and the associated image are marked in green, and the connection points of the incomplete lines are determined in the image to be processed and the associated image respectively, as clipping reference points; If not, then based on the connection relationship between the lines of each model framework, determine the associated lines of the incomplete line, and infer the endpoints of the incomplete line based on the associated lines. If the inference is successful, then determine that the incomplete line is a predictable complete line and mark it in green. 2D images with all green calibration lines are considered high-quality 2D images, and the 2D image dataset is updated accordingly. When red marker lines exist on the 2D image, a re-acquisition area is determined on the real 3D model based on the line number corresponding to the red marker lines. An acquisition command is then sent to the image acquisition module to re-acquire the 2D image corresponding to the re-acquisition area.

[0031] In this embodiment, the preset amplitude value range is [10, 50], which can be flexibly set according to the accuracy requirements of aviation scenarios. For high-precision component images such as engine blades and precision connectors, the preset amplitude is set to 40 by default; for conventional scenario images such as the overall shape of the aircraft and the internal structure of the cabin, the preset amplitude is set to 30 by default.

[0032] In this embodiment, the preset threshold value ranges from [0.8, 1] and can be flexibly set according to the accuracy requirements of aviation scenarios. The default value is 0.85. For high-requirement scenarios such as aviation airworthiness certification and reverse engineering of precision parts, the preset threshold value is set to 0.95.

[0033] In this embodiment, endpoint features refer to the positions of the two endpoints of the model construction lines and the distance between the two endpoints and the assembly constraint relationship.

[0034] In this embodiment, the associated image refers to a 2D image containing incomplete lines. The 2D image may contain all incomplete lines or only some complete lines.

[0035] In this embodiment, the line rendering effect refers to the display effect of incomplete lines in a real 3D model.

[0036] In this embodiment, "combined complete presentation" means that the image to be processed and the associated images are combined to fully display the incomplete line. This can be either any one of the associated images fully displaying the incomplete line, or a portion of the image to be processed and the associated images being displayed, which together complete the presentation of the incomplete line.

[0037] In this embodiment, the cropping reference point is used to ensure the complete display of incomplete lines during the preprocessing process of cropping the image to adapt to the model input. If multiple incomplete lines appear on a 2D image during the cropping process, priority is given to ensuring that the incomplete lines that have not been combined with the cropped 2D images are fully displayed. If an incomplete line on the image to be processed is fully rendered on any associated image, then the incomplete line does not need to be clipped. The reference point, i.e., the connection point, needs to be determined.

[0038] In this embodiment, the model framework lines include aircraft structural outlines, assembly reference lines, and component feature lines.

[0039] In this embodiment, endpoint inference refers to inferring the appearance and endpoint position of incomplete lines in a 2D image based on the assembly connection relationship between the lines of the model framework and the known lines connected by the endpoints of incomplete lines.

[0040] The beneficial effects of the above technical solution are as follows: First, this invention detects the sharpness of each pixel in a 2D image based on gradient magnitude. Then, it evaluates the overall sharpness of the image based on the sharpness of each pixel, discarding 2D images with low sharpness to obtain a new 2D image dataset. This helps improve the model's ability to accurately extract structural features of aircraft, thereby enhancing model training effectiveness and generalization ability. Next, based on the model framework lines of the real 3D model and the connection relationships between these lines and the assembly, the endpoint features of the model framework lines are extracted and marked with serial numbers on the 2D images in the corresponding 2D image dataset of the real 3D model data. This facilitates rapid location of error positions during model training, improving the efficiency of local optimization of the improved Transformer model. Then, based on the endpoint features of the model framework lines, it determines whether each line is fully displayed on the 2D image. If so, the fully displayed lines are marked in green; otherwise, the incompletely displayed lines are marked in red. The 2D images marked in red are used as the images to be processed. Based on the serial number markings of the model structure lines, the correspondence between the lines in each 2D image is determined, and the associated images corresponding to the incomplete lines in the image to be processed are obtained. The shooting angles of the image to be processed and the associated images are obtained respectively. Based on the shooting angles and the line presentation effect of the incomplete lines on the real 3D model, it is determined whether the incomplete lines are completely presented by combining the image to be processed and the associated images. This greatly reduces the probability that the improved Transformer model will learn partial or incorrect feature relationships due to the lack of key structural lines in the image. It helps the improved Transformer model to more accurately understand and grasp the structural characteristics of objects or scenes, effectively improves the performance of the improved Transformer model after training, and determines the connection points of the incomplete lines in the image to be processed and the associated images respectively when the incomplete lines are completely presented as clipping reference points, providing a basis for the preprocessing of high-quality 2D images.When incomplete lines are not fully presented, the associated lines of the incomplete lines are determined based on the connection relationships between the lines of each model framework. Endpoint inference is then performed on the incomplete lines based on these associated lines. This process, to a certain extent, enables data inference training during the 2D image to 3D model conversion, which is beneficial for the model's understanding of the constructed object scene structure. Through multi-image combination verification and endpoint inference, the integrity of the aircraft structure lines is calibrated and completed, significantly reducing the probability of the model learning partial or incorrect feature relationships due to missing key aircraft structure lines in the image. This helps the model more accurately understand and grasp the characteristics of complex aircraft structures, effectively improving the performance of the trained model. Finally, based on the line number corresponding to the red-calibrated lines, a re-acquisition area is determined on the real 3D model. An acquisition command is sent to the image acquisition module to re-acquire the 2D image corresponding to the re-acquisition area, maximizing the integrity and accuracy of the determined model training data, thereby helping to improve the model's overall performance and adaptability to different situations.

[0041] Example 4: Based on Example 1, as follows Figure 3 As shown, step 3 includes: Step 301: Based on optimizing the correspondence between common pixels on the 2D image and point cloud on the 3D model, pair the data features of the 2D image and its corresponding 3D sub-model, and generate the model training set and model validation set; Step 302: Based on the model training set and the model validation set, train the improved Transformer model to obtain the trained improved Transformer model; The improved Transformer model includes: A fixed window attention mechanism is added to the self-attention module of the improved Transformer model. The data features input into the improved Transformer model are divided into windows of a preset size, and the self-attention of the data features in each window is calculated to generate a feature map. The feature map is aggregated to obtain a reduced feature map, which is then sent to a multilayer perceptron for processing.

[0042] In this embodiment, the 3D sub-model refers to the 3D model of the local structure, system or component of the aircraft that corresponds to the area captured by the 2D image in the real 3D model.

[0043] In this embodiment, the model validation set refers to the real 3D model dataset corresponding to the aircraft / parts; the model training set refers to the 2D image dataset corresponding to the real 3D model data.

[0044] In this embodiment, the preset size of the fixed window can be flexibly set according to the structural features of the aircraft. For large scene features such as the overall shape of the aircraft, the preset window size is 16×16; for fine structural features such as engine blades and precision connectors, the preset window size is 8×8.

[0045] The beneficial effects of the above technical solution are as follows: This invention optimizes the correspondence between common pixels on 2D images and point clouds on 3D models, pairs the data features of 2D images and their corresponding 3D sub-models, and generates model training and validation sets. This facilitates the model's accurate learning of the spatial mapping relationship between 2D images of aircraft and real 3D models. Based on the model training and validation sets, the improved Transformer model is trained to obtain a well-trained improved Transformer model, realizing autonomous training of 2D data to synthesize 3D data models. A fixed window is added to the self-attention module of the improved Transformer model. The self-attention mechanism divides the data features input into the improved Transformer model into windows of a preset size, and calculates the self-attention of the data features within each window to generate feature maps. This makes the improved Transformer model more versatile and adaptable, improves the model's ability to extract features from complex curved surfaces and intricate structures of aircraft, and enhances the model's generalization ability. The feature maps are aggregated to obtain reduced feature maps, which are then sent to a multilayer perceptron for processing. This effectively deepens the network depth of the improved Transformer model, improving the model's learning ability and fitting accuracy for complex aircraft structures.

[0046] Example 5: Based on Example 4, the improved Transformer model is trained using the model training set and model validation set to obtain a trained improved Transformer model, including: The model training set is input into the improved Transformer model to synthesize the target 3D model. The target 3D model is then compared with the real 3D model corresponding to the model validation set to obtain the model differences. Based on the aforementioned model differences, a preset loss function is used to continuously adjust the model parameters of the improved Transformer model through backpropagation algorithm until the optimal solution is obtained. Then, the training of the improved Transformer model ends, and a well-trained improved Transformer model is obtained.

[0047] The beneficial effects of the above technical solution are as follows: This invention inputs the model training set into the improved Transformer model to synthesize the target 3D model, compares the target 3D model with the corresponding real 3D model in the model validation set to obtain model differences; based on the model differences, a preset loss function is used to continuously adjust the model parameters of the improved Transformer model through the backpropagation algorithm until the optimal solution is obtained, then the training of the improved Transformer model ends, and a trained improved Transformer model is obtained. This realizes the autonomous training and optimization of the improved Transformer model, ensuring the consistency between the synthesized aircraft 3D model and the real model, and meeting the accuracy requirements of the aviation field.

[0048] Example 6: Based on Example 4, the improved Transformer model is trained using the model training set and model validation set, and the training also includes: Based on the type of real 3D model corresponding to the model validation set, the model training set and the model validation set are classified, and convertible model labels are generated based on the classification results. Clustering is performed on the model training set and model validation set based on convertible model labels to obtain the final model training set and model validation set corresponding to the whole aircraft, various systems and component scenarios.

[0049] In this embodiment, the convertible model labels include aircraft whole-body labels, aerodynamic shape labels, engine component labels, landing gear labels, fuselage structural component labels, and cabin structure labels.

[0050] The beneficial effects of the above technical solution are as follows: Based on the type of real 3D model corresponding to the model validation set, the present invention classifies the model training set and the model validation set, and generates convertible model labels according to the classification results; based on the convertible model labels, the model training set and the model validation set are clustered respectively to obtain the final model training set and model validation set corresponding to various objects and scenarios, realizing targeted training for the synthesis of 3D models of different structures and scenarios of aircraft. While ensuring that the model can achieve high-precision synthesis of 2D data to 3D data, it effectively expands the application scope of the model in the entire life cycle of aircraft. Users can quickly switch the appropriate model weights through model labels according to actual synthesis needs to achieve accurate synthesis of 3D models of corresponding scenarios.

[0051] Example 7: Based on Example 4, the target 3D model is compared with the corresponding real 3D model in the model validation set to obtain model differences, including: The model network structure is traversed to obtain the intersection lines between different faces of the target 3D model and the real 3D model, and the coordinate information of the intersection lines is obtained. The coordinate information of the intersecting lines corresponding to the target 3D model and the real 3D model is compared to obtain the intersecting line error. Based on the surfaces corresponding to the intersecting lines, the intersecting lines are clustered to obtain multiple intersecting line groups. Based on the curvature of each surface and the complexity of the intersecting lines, the error coefficient corresponding to the intersecting line group for each surface is determined. The average intersecting line error corresponding to each intersecting line group is calculated, and the error degree corresponding to the current surface is determined by combining the error coefficient. Furthermore, by combining the structural position of each face within the aircraft or its components, the relationships between the faces are determined, and error weights are allocated based on these relationships to determine the corresponding error weight for each face. Based on the error weights and the error degree of each face, the final error of each face is determined and labeled on each face.

[0052] In this embodiment, intersecting lines refer to lines including aircraft skin surface connection lines, structural component assembly lines, and component outline lines.

[0053] In this embodiment, an interlaced line group refers to all interlaced lines corresponding to the same face.

[0054] In this embodiment, the complexity of the interlacing lines is the product of the curvature and the complexity of the interlacing lines. The more interlacing lines there are and the greater the curvature, the larger the error coefficient. For example, the complexity of the interlacing lines takes the value [0, 1]; the curvature takes the value [0.5, 1].

[0055] In this embodiment, the error degree is the product of the average cross-line error and the error coefficient.

[0056] In this embodiment, the error weight is set according to the importance of the surface in the aircraft. The error weight range for aerodynamic surfaces, load-bearing structural surfaces, and precision assembly surfaces is [0.8, 1], while the error weight range for non-critical structural surfaces is [0.3, 0.7]. The error weights for aerodynamic surfaces, load-bearing structural surfaces, and precision assembly surfaces are higher than those for non-critical structural surfaces.

[0057] The beneficial effects of the above technical solution are as follows: This invention traverses the model network structure, obtains the interlacing lines between different faces of the target 3D model and the real 3D model, and obtains the coordinate information of the interlacing lines; compares the coordinate information of the interlacing lines corresponding to the target 3D model and the real 3D model to obtain the interlacing line error; and clusters the interlacing lines according to the faces corresponding to the interlacing lines to obtain multiple interlacing line groups. Based on the curvature of each face and the complexity of the interlacing lines, the error coefficient corresponding to the interlacing line group corresponding to each face is determined, the average interlacing line error corresponding to each interlacing line group is calculated, and the error degree corresponding to the current face is determined by combining the error coefficient; and the error degree corresponding to each face is determined by combining the error coefficient with the error degree of each face. The model identifies the structural position of each facet within the aircraft or its components, determines the relationships between them, and assigns error weights based on these relationships. The final error for each facet is then determined by combining these error weights with the error degree of each facet and labeling each facet. By traversing the model's network structure, the model obtains the intersecting lines and their coordinate information, and compares these to obtain the intersecting line errors. This allows the model to focus on the errors between each facet and its corresponding facet in the real 3D model. This enables a more detailed determination of the differences between the synthesized 3D model and the real 3D model in key aircraft structures and local details, providing precise data support for iterative optimization of model parameters.

[0058] Example 8: Based on Example 1, a 3D model synthesis method based on an improved Transformer further includes: Model data packets generated based on the synthetic 3D model are sent to the user according to a preset communication address.

[0059] In this embodiment, the preset communication address can be one or more, including the designated data receiving address of aircraft design units, maintenance enterprises, airworthiness review agencies, and research institutes.

[0060] In this embodiment, the model data package is adapted to the format requirements of aerodynamic simulation, structural strength verification, maintenance process planning, and airworthiness certification review.

[0061] The beneficial effects of the above technical solution are as follows: Based on the synthesized 3D model of an aircraft, the present invention generates model data packets that are adapted to the format requirements of various professional aviation scenarios and sends them to users according to the preset communication address. This satisfies the professional needs of different users for 3D models in different scenarios such as aircraft research and development, maintenance, modification, and airworthiness, and improves the pertinence and practicality of aircraft 3D model data transmission.

[0062] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A 3D model synthesis method based on an improved Transformer, characterized in that, include: Step 1: Collect a large number of multi-dimensional 2D images of aircraft and their corresponding real 3D model data, and establish a 2D image dataset and its corresponding real 3D model dataset. Step 2: Extract data features from the 2D images in the 2D image dataset and the corresponding 3D sub-models in the real 3D model dataset; Step 3: Train the improved Transformer model based on the data features; Step 4: After preprocessing the 2D image data of the aircraft to be synthesized, input it into the improved Transformer model based on the training for processing to obtain the synthesized 3D aircraft model.

2. The 3D model synthesis method based on an improved Transformer according to claim 1, characterized in that, Step 1 includes: Based on big data collection, a massive amount of 2D images and corresponding real 3D model data of aircraft, fuselage skin, engines, landing gear, avionics systems, hydraulic systems and various precision components are collected. 2D images are detected and screened to obtain high-quality 2D images, and these high-quality 2D images are then preprocessed to obtain optimized 2D images. Based on optimizing the correspondence between the entire aircraft or its components in the 2D images and the 3D model data, as well as the association of the assembly topology data between the various 3D models, a 2D image dataset and its corresponding real 3D model dataset are established.

3. The 3D model synthesis method based on an improved Transformer according to claim 1, characterized in that, Step 3: Includes: Based on optimizing the correspondence between common pixels on 2D images and point clouds on 3D models, the data features of 2D images and their corresponding 3D sub-models are paired to generate model training sets and model validation sets. The improved Transformer model is trained using the model training set and the model validation set to obtain a well-trained improved Transformer model.

4. The 3D model synthesis method based on an improved Transformer according to claim 3, characterized in that, The improved Transformer model includes: A fixed window attention mechanism is added to the self-attention module of the improved Transformer model. The data features input into the improved Transformer model are divided into windows of a preset size, and the self-attention of the data features in each window is calculated to generate a feature map. The feature map is aggregated to obtain a reduced feature map, which is then sent to a multilayer perceptron for processing.

5. A 3D model synthesis method based on an improved Transformer according to claim 4, characterized in that, Based on the model training set and model validation set, the improved Transformer model is trained to obtain a trained improved Transformer model, including: The model training set is input into the improved Transformer model to synthesize the target 3D model. The target 3D model is then compared with the real 3D model corresponding to the model validation set to obtain the model differences. Based on the aforementioned model differences, a preset loss function is used to continuously adjust the model parameters of the improved Transformer model through backpropagation algorithm until the optimal solution is obtained. Then, the training of the improved Transformer model ends, and a well-trained improved Transformer model is obtained.

6. The 3D model synthesis method based on the improved Transformer according to claim 5, characterized in that, Training the improved Transformer model based on the model training set and model validation set also includes: Based on the type of real 3D model corresponding to the model validation set, the model training set and the model validation set are classified, and convertible model labels are generated based on the classification results. Clustering is performed on the model training set and model validation set based on convertible model labels to obtain the final model training set and model validation set corresponding to the whole aircraft, various systems and component scenarios.

7. A 3D model synthesis method based on an improved Transformer according to claim 5, characterized in that, The target 3D model is compared with the corresponding real 3D model in the model validation set to obtain model differences, including: The model network structure is traversed to obtain the intersection lines between different faces of the target 3D model and the real 3D model, and the coordinate information of the intersection lines is obtained. The coordinate information of the intersecting lines corresponding to the target 3D model and the real 3D model is compared to obtain the intersecting line error. Based on the surfaces corresponding to the intersecting lines, the intersecting lines are clustered to obtain multiple intersecting line groups. Based on the curvature of each surface and the complexity of the intersecting lines, the error coefficient corresponding to the intersecting line group for each surface is determined. The average intersecting line error corresponding to each intersecting line group is calculated, and the error degree corresponding to the current surface is determined by combining the error coefficient. Furthermore, by combining the structural position of each face within the aircraft or its components, the relationships between the faces are determined, and error weights are allocated based on these relationships to determine the corresponding error weight for each face. Based on the error weights and the error degree of each face, the final error of each face is determined and labeled on each face.

8. A 3D model synthesis method based on an improved Transformer according to claim 2, characterized in that, Detection and screening of 2D images to obtain high-quality 2D images, including: The gradient magnitude of pixels in a 2D image is calculated based on the Sobel operator, and the gradient magnitude is compared with a preset magnitude. Pixels with a gradient magnitude greater than or equal to the preset magnitude are considered as qualified points. Pixels with gradient magnitudes less than a preset magnitude are designated as blurred points. Calculate the proportion of blurred points in the 2D image. When the proportion of blurred points is greater than a preset threshold, the 2D image is determined to be blurred, the 2D image is deleted, and the image data in the 2D image dataset is updated to obtain a new 2D image dataset. Based on the real 3D model dataset, the model framework lines of the real 3D model and the assembly connection relationship between each model framework line are determined. The endpoint features of the model framework lines are extracted and marked with serial numbers on the 2D images in the corresponding 2D image dataset of the real 3D model data. Based on the endpoint features of the model framework lines, it is determined whether each line is fully displayed on the 2D image. If so, the fully displayed lines are marked in green; otherwise, the incompletely displayed lines are marked in red. The 2D images marked in red are taken as the images to be processed. Based on the serial number markings of the model framework lines, the correspondence between the lines in each 2D image is determined, and the associated images corresponding to the incomplete lines in the images to be processed are obtained. The shooting angles of the image to be processed and the associated image are obtained respectively. Based on the shooting angles and the line presentation effect of the incomplete display line on the real 3D model, it is determined whether the incomplete line is fully presented by the combination of the image to be processed and the associated image. If so, the incomplete lines on the image to be processed and the associated image are marked in green, and the connection points of the incomplete lines are determined in the image to be processed and the associated image respectively, as clipping reference points; If not, then based on the assembly and connection relationship between the lines of each model framework, determine the associated lines of the incomplete lines, and infer the endpoints of the incomplete display lines based on the associated lines. If the inference is successful, then determine that the incomplete lines are inferable complete lines and mark them in green. The 2D images with all green calibration lines are considered high-quality 2D images, and the 2D image dataset is updated accordingly.

9. A 3D model synthesis method based on an improved Transformer according to claim 8, characterized in that: When red marker lines exist on the 2D image, a re-acquisition area is determined on the real 3D model based on the line number corresponding to the red marker lines. An acquisition command is then sent to the image acquisition module to re-acquire the 2D image corresponding to the re-acquisition area.

10. A 3D model synthesis method based on an improved Transformer according to claim 1, characterized in that, Also includes: Model data packets generated based on the synthetic 3D model are sent to the user according to a preset communication address.

Images