Aero-engine augmented reality virtual-real fusion method based on depth prior scene reconstruction

Abstract

An aero-engine augmented reality virtual-real fusion method based on depth prior scene reconstruction comprises the following steps: firstly, collecting data of an aero-engine look-around RGB image and a depth image, calculating a camera frame pose by using an SFM algorithm and obtaining a sparse point cloud, and optimizing the camera pose by using an ICP algorithm and the point cloud converted from the depth image to obtain an aligned prior dense point cloud; gaussian points are initialized based on the priori dense point cloud and the sparse point cloud, and random generation of 3D Gaussian is limited by using the boundary of the dense point cloud as geometric constraint; through the three-dimensional reconstruction of the adjacent multi-view enhancement strategy, the spatial and visual consistency is improved, and an aero-engine model is reconstructed; then understanding the reconstructed aero-engine model based on semantic segmentation, identifying and marking core components, and finally performing virtual-real fusion visualization on the reconstructed aero-engine through augmented reality to guide training, assembling and maintenance operations. According to the invention, rapid high-quality three-dimensional reconstruction and augmented reality virtual-real fusion visualization of the aero-engine are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of aero-engines, and particularly to an augmented reality virtual-real fusion method for aero-engines based on depth prior scene reconstruction. Background Art

[0002] An aero-engine is a core component of an aircraft. With the rapid development of the aviation industry, the training, assembly, maintenance, etc. of aero-engines have become increasingly complex. Traditional methods require technicians to operate on-site, which is time-consuming and laborious, and it is difficult for technicians in different regions to collaborate in real time. To address this problem, there is an urgent need for a technology that can quickly and high-quality reconstruct a real aero-engine, three-dimensionally reconstruct a model and combine it with augmented reality technology for virtual-real fusion visualization to achieve more efficient training, assembly, maintenance, etc., and also enable technicians in different regions to achieve remote collaboration through augmented reality technology. Therefore, effectively improving the three-dimensional reconstruction quality and virtual-real fusion visualization effect of aero-engines can effectively promote the development of advanced aviation equipment.

[0003] Currently, there are already relevant literatures on using three-dimensional reconstruction for virtual-real fusion visualization and using augmented reality technology for aero-engine assembly, maintenance, etc., but there are problems such as slow reconstruction speed, low reconstruction quality, and difficulty in remote collaboration for assembly and maintenance. For example, the patent application with the name "Virtual Reality and Augmented Reality Fusion Method Based on Indoor Three-Dimensional Reconstruction" (Publication No.: CN118485796A) realizes three-dimensional reconstruction of the target and augmented reality visualization, but the problem of large targets with complex structures is not considered in the reconstruction process. Another example is the patent application with the name "An Augmented Reality-Assisted Assembly Method for the Blind Spot of the Field of Vision in an Aircraft Equipment Compartment" (Publication No. CN115797099A), whose principle is to visualize the picture of the blind spot in the field of vision during the assembly process according to the pose estimation result of the assembly object and combine it with augmented reality technology for assembly guidance. The modeling model is used, and the appearance texture is inconsistent with the real model.

[0004] To improve the speed and quality of augmented reality virtual-real fusion visualization of aero-engines, an augmented reality virtual-real fusion method for aero-engines based on depth prior scene reconstruction is needed. By combining technologies such as depth prior, three-dimensional reconstruction, virtual-real fusion visualization, etc. with augmented reality technology, rapid and high-quality reconstruction of a real aero-engine scene can be achieved, and efficient training, assembly, maintenance, remote collaboration, etc. of aero-engines can be realized by combining augmented reality technology, solving problems such as low reconstruction quality, slow reconstruction speed, and poor virtual-real fusion effect. Currently, there is no corresponding literature disclosure. Summary of the Invention

[0005] In order to overcome the above-mentioned disadvantages of the prior art, the purpose of the present invention is to provide an augmented reality virtual-real fusion method for aero-engine based on depth prior scene reconstruction, so as to realize fast and high-quality three-dimensional reconstruction of the aero-engine and augmented reality virtual-real fusion visualization.

[0006] In order to achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0007] An augmented reality virtual-real fusion method for aero-engine based on depth prior scene reconstruction, comprising the following steps:

[0008] Step 1) Acquisition of target object panoramic data and solution of camera pose: Use a depth camera to collect multi-view data of the target object, ensure that there is overlap between views, and obtain high-quality depth maps and RGB images; Use the SFM algorithm to process the collected RGB images, extract ORB key points and perform matching, and then calculate the camera frame pose to obtain the camera poses and sparse point clouds of each view; Use the ICP algorithm to accurately align the initially calculated camera frame pose with the depth map data, correct the pose error in the point cloud, ensure that the data of different views can be accurately aligned, and generate the aligned prior dense point cloud data;

[0009] Step 2) Initialization and constraint of Gaussian points based on depth prior point cloud and sparse point cloud: Use the prior dense point cloud obtained in Step 1 as a reference to eliminate the wrong points in the sparse point cloud; According to the spatial distribution and scale characteristics of the optimized prior dense point cloud, determine the position of the initial Gaussian points and initialize high-quality Gaussian ellipsoids; Use the boundary of the dense point cloud as a geometric constraint, introduce a constraint function, and limit the generated 3D Gaussian points within a preset geometric range, so that they always maintain consistent geometric characteristics during the training process, ensure that the Gaussian ellipsoids accurately fit the engine surface, accurately reflect the geometric characteristics of the actual object, and be consistent with the real world;

[0010] Step 3) Three-dimensional reconstruction based on adjacent multi-view enhancement strategy: Introduce a deep learning method with an adjacent multi-view enhancement strategy during the three-dimensional reconstruction training process, create a pose graph to represent the pose information of all cameras and their mutual relationships, where nodes represent the poses at each moment and edges represent relative position information; Combine the pose graph to calculate adjacent multi-view images with overlapping viewing cones, learn the geometric structure relationship and texture consistency between adjacent views, enhance the model's ability to capture complex surface details, and ensure spatial consistency and visual consistency;

[0011] Step 4) Understanding the reconstruction target based on semantic segmentation: Use the U-Net deep learning model to perform semantic segmentation on the RGB image, extract pixel category labels, map the semantic category labels to Gaussian ellipsoid points through the camera pose and projection relationship, integrate the semantic features from different perspectives through a weighted fusion strategy to form the final semantic label of the Gaussian points, and identify and label core components such as the rotor and stator;

[0012] Step 5) Reconstructing the scene for augmented reality virtual-real fusion visualization: According to the aeroengine reconstructed in Step 3) and the category information of core components such as the rotor and stator obtained in Step 4), define the display rules, display conditions, and interaction methods of virtual objects for different components, ensure the geometric alignment and fusion of virtual information with the real scene, perform virtual-real fusion rendering and dynamic display, and adjust the display state or attributes of the object according to interaction instructions and changes in the scene environment; At the same time, construct an information database for training, assembly, and maintenance to achieve the virtual-real fusion display of relevant information and guide users to perform relevant operations.

[0013] Step 1) Specifically, the panoramic data collection of the target object and the solution of the camera pose are as follows:

[0014] Step 1.1) Multi-perspective data collection of the aeroengine; Use the Realsense depth camera to collect panoramic data of the aeroengine, obtain the corresponding high-quality depth map and RGB image, and extract frames from the obtained images to ensure that the overlap rate between adjacent perspectives is not higher than 80%;

[0015] Step 1.2) SFM calculates the initial camera pose and generates a sparse point cloud; Use the SFM algorithm to process the collected RGB images, extract key points using the ORB algorithm, detect corner points in the images using the FAST algorithm, calculate the rotation BRIEF descriptor for each key point, then perform feature matching, perform descriptor matching between image pairs of adjacent perspectives, filter out high-quality matching pairs, and use the five-point method to calculate the initial camera pose; Generate a sparse point cloud using the initial pose, merge the point clouds from different perspectives to form an overall sparse point cloud model; During the merging process, remove duplicate points and noise points to maintain the sparsity and accuracy of the point cloud;

[0016] Step 1.3) Precise alignment of the camera pose; Convert each aligned depth map into three-dimensional point cloud data and perform coordinate transformation using the camera internal parameter matrix K; Use the ICP algorithm, take the camera pose obtained in Step 1.2) as the initial pose, perform ICP fine registration on the three-dimensional point clouds converted from adjacent depth maps, and correct the initial pose calculated in Step 1.2) with the precise pose result solved by the ICP algorithm to precisely align the camera pose;

[0017] Step 1.4) Generate dense point cloud data; using the accurate camera poses obtained in Step 1.3), fuse the aligned depth point cloud data of each view, and adopt voxel grid filtering and weight fusion to process the overlapping regions existing in multiple views to make the point cloud density uniform; assign a weight to each point, where the weight is based on the confidence of the point. In the overlapping region, perform weighted averaging on multiple points at the same spatial position to reduce noise and duplicate points.

[0018] Step 2) Initialize Gaussian points based on depth prior data and sparse point cloud. Specifically:

[0019] Step 2.1) Use the prior dense point cloud as a benchmark to eliminate incorrect points, and adopt statistical outlier detection to screen abnormal points in the sparse point cloud to ensure the high quality and geometric consistency of the point cloud data. Then, integrate the sparse point cloud into the prior dense point cloud.

[0020] Step 2.2) Determine the positions of the initial Gaussian points according to the spatial distribution and scale characteristics of the optimized prior dense point cloud. Adopt the K-means algorithm to perform clustering processing on the prior dense point cloud, use the center coordinates of the point cloud clusters as the initial positions of the Gaussian points, and calculate the covariance matrix of each clustering cluster to define the shape and direction of the Gaussian ellipsoid.

[0021] Step 2.3) Introduce geometric constraints to impose restrictions on the generation of Gaussian ellipsoids. Use the boundary of the prior dense point cloud as a geometric constraint, and use a constraint function to limit the generated 3D Gaussian points within a preset geometric range; when generating Gaussian ellipsoids during the training process, only allow the center position u i of the ellipsoid to be located within the geometric range defined by the boundary; for each Gaussian ellipsoid only when μ i ∈V, generation and subsequent optimization are performed;

[0022] Step 2.4) Maintain and optimize geometric consistency during the training process; during the model training process, continuously apply geometric constraints to ensure that the generated 3D Gaussian ellipsoids maintain geometric consistency.

[0023] Step 3) 3D reconstruction based on the adjacent multi-view enhancement strategy. Specifically:

[0024] Step 3.1) Pose graph construction; according to the camera poses, obtain the camera pose matrices T i and R i corresponding to each image, and construct a pose graph;

[0025] Step 3.2) Obtain adjacent views with overlapping view cones; obtain a pair of adjacent views from the pose graph, and use the Sutherland-Hodgman algorithm to calculate the intersection region of the two view cones; add the judgment of the view direction consistency. When IoU ij ≥τ and θ min<α ij ≤θ max , the current adjacent view pair meets the requirements;

[0026] Step 3.3) Multi-view loss calculation; Step 3.2) Obtain multiple camera views with overlapping frustums and use them as the training camera view list. According to the camera views in the list, combine the current Gaussian distribution and the rendering pipeline to perform image rendering, and obtain the rendering results corresponding to each camera view; Calculate the loss of the corresponding view, sum up the losses of all views in the training camera view list, and then perform backpropagation to update the parameters of the Gaussian ellipsoid.

[0027] Step 3.4) 3DGS three-dimensional scene reconstruction; Optimize spatial and visual consistency to achieve high-quality 3DGS scene reconstruction, generate the final three-dimensional scene model, and ensure that the generated three-dimensional scene has high details and consistency in each view, meeting the requirements of spatial and visual consistency.

[0028] Step 4) Understanding the reconstructed scene based on semantic segmentation is specifically as follows:

[0029] Step 4.1) Semantic segmentation model training and application; Collect and annotate an aero-engine RGB image dataset with semantic class labels for training a deep learning semantic segmentation model, and each pixel point is assigned a class label; Adopt the U-Net deep learning model architecture, use the annotated dataset, and train the semantic segmentation model through supervised learning methods to optimize the model parameters to minimize the classification error; After the model training is completed, use the pictures used for the reconstructed scene as the input, input them into the trained U-Net model, and obtain the class prediction results of each pixel point; Extract the class labels of each pixel point from the prediction results output by the model to form a semantic class label map.

[0030] Step 4.2) Projecting semantic labels into three-dimensional space; According to the pose and projection relationship of the camera, map the class label Y pred (x, y) in the semantic class label map to the corresponding Gaussian ellipsoid point X in three-dimensional space to form the semantic class label of each Gaussian point.

[0031] Step 4.3) Semantic feature weighted fusion strategy; For each Gaussian ellipsoid point X j , according to its semantic class label C j mapped to each view, generate the corresponding semantic feature vector Normalize the semantic feature vector to ensure that the feature vectors under different views have a consistent scale; According to the reliability w k of each view, calculate the weights of the semantic features of each view; After obtaining the weights of the semantic features of each view, perform semantic feature weighted fusion, and combine the semantic feature vectors of the Gaussian ellipsoids at the same position from different views. According to the weight w k perform weighted summation to obtain the final semantic feature vector f j ; For the semantically feature vector f after weighted fusion j apply the Softmax function to calculate the probability distribution P of each semantic category j (c); Take the semantic category with the highest probability as the Gaussian point X j of the final semantic label C j ;

[0032] Step 4.4) Core component recognition and labeling; From the entire scene Gaussian point set S, combine the final semantic label information obtained in Step 4.3), filter out the Gaussian ellipsoid points belonging to predefined core component categories such as the rotor and stator, group them according to spatial proximity, and group the mutually close Gaussian ellipsoid points into one group through a preset spatial threshold to classify the Gaussian points that may not have been assigned semantic labels; A group of Gaussian ellipsoid points with the same label category represents a core component, which is marked as a core component instance and assigned a unique identifier ID k for subsequent virtual-real fusion use.

[0033] Step 5) Reconstruct the scene for augmented reality virtual-real fusion visualization specifically as follows:

[0034] Step 5.1) Define the virtual models, display rules, and display conditions of the core components; Define the corresponding virtual object models and their attributes for core component categories such as the rotor and stator of each type of aero-engine, including basic dimension information and uses; Set the display attributes for the virtual models, including transparency, movability, and collision attributes; Set the display conditions for the virtual models to dynamically adjust the display state of the virtual objects; According to the current user's task requirements, adjust the display content and method of the virtual objects, allow the user to move and rotate the objects through gestures to adjust their positions and attributes, and support the user to control the display state of the virtual objects or trigger specific operations through voice commands;

[0035] Step 5.2) Virtual-real fusion rendering and dynamic interactive display; Coordinate the visual attributes such as the color and lighting of the virtual objects with the real scene to ensure natural visual fusion; Be able to respond to user interactions in real time, and adjust the position, orientation, and display attributes of the virtual objects in real time according to the user's gestures or voice commands to achieve interactive operations; At the same time, dynamically adjust the display attributes of the virtual objects according to the change of the user's perspective or the change of the scene environment to maintain consistency with the real scene;

[0036] Step 5.3) Build an information database; Establish a training, assembly, and maintenance information database, including training tutorials, operation steps, precautions, assembly processes, tool usage instructions, assembly step animations, maintenance manuals, and repair steps, and store the information using a NoSQL database;

[0037] Step 5.4) Augmented reality visualization: The user wears HoloLens glasses, the system loads the software module, and the augmented reality visualization of the aircraft engine model and operation interface is visualized by fusion of virtual and real. Multiple users can view and operate the aircraft engine virtual model synchronously through their respective HoloLens devices to achieve collaboration between users. Users can operate the virtual model and database information through gestures, voice, and menu interactions, which facilitates more intuitive training, assembly, and maintenance.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] (1) The present invention proposes a method for collecting surround view data of target objects and solving camera pose based on a depth camera and SFM algorithm. Through precise alignment with the ICP algorithm, pose errors are corrected to generate high-quality prior dense point cloud data, providing reliable initialization information for subsequent reconstruction.

[0040] (2) The present invention introduces a deep prior point cloud constraint algorithm and initializes a Gaussian ellipsoid according to the optimized spatial distribution and scale characteristics of the prior dense point cloud. The boundary of the prior dense point cloud is used as a geometric constraint to limit the generation of 3D Gaussian points, ensuring accurate fitting of the engine surface and improving the quality and accuracy of reconstruction.

[0041] (3) The present invention adopts the method of adjacent multi-view enhancement strategy for three-dimensional reconstruction, uses the pose graph to integrate the camera pose information, obtains adjacent view images with overlapping viewing cones, learns the geometric structure relationship and texture consistency between adjacent view, breaks through the limitations of single-view reconstruction, enhances the model's ability to capture complex surface details, and improves the reconstruction quality.

[0042] (4) The present invention innovatively realizes the seamless integration of image semantic segmentation and three-dimensional Gaussian model. It uses the U-Net deep learning model to extract the semantic category information of the RGB image and maps it to the Gaussian ellipsoid points. It integrates the semantic features of different perspectives through a weighted fusion strategy, identifies and labels the core components, and provides rich information for virtual-reality fusion.

[0043] (5) The present invention realizes interactive dynamic display of virtual objects and real parts through the augmented reality fusion visualization of the reconstructed scene, improves the realism and interactivity of the augmented reality visualization, and provides intuitive and efficient assistance for industrial assembly, maintenance, training and other applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of an embodiment of the present invention.

[0045] Figure 2 This is a flow chart of target object surround data collection and camera pose solution in an embodiment of the present invention.

[0046] Figure 3 This is the flowchart of 3D reconstruction based on the depth prior point cloud constraint algorithm and the adjacent multi-view enhancement strategy in the embodiments of the present invention.

[0047] Figure 4 This is the visualization screen of the 3D reconstruction scene in the embodiments of the present invention. Detailed implementation manners

[0048] The present invention will be described in detail below in conjunction with the embodiments and the drawings. The present invention can be implemented in many different forms and should not be considered limited to the embodiments described herein. On the contrary, these embodiments are provided so that the disclosure of the present invention is thorough and complete, and will fully convey the scope of the present invention to those skilled in the art.

[0049] Referring to Figure 1 , an augmented reality virtual-real fusion method for aero-engine based on depth prior scene reconstruction includes the following steps:

[0050] Step 1), referring to Figure 2 , acquisition of panoramic data of the target object and solution of the camera pose:

[0051] Step 1.1) Acquisition of multi-view data of the aero-engine; use a Realsense depth camera to collect panoramic data of the aero-engine, adjust the camera position and angle according to a predetermined surrounding path, and move slowly to ensure the continuity and integrity of the data, obtain corresponding high-quality depth maps and RGB images, and extract frames from the obtained images to ensure that the overlap rate between adjacent views is not higher than 80%, reducing the data processing volume;

[0052] Step 1.2) Calculate the initial camera pose by SFM and generate a sparse point cloud; use the SFM algorithm to process the collected RGB images, use the ORB algorithm to extract key points from the RGB images, use the FAST algorithm to detect corner points in the images, and assign a direction to each key point to achieve the rotation invariance of the features; the calculation formula of the direction θ is:

[0053]

[0054] where I(x, y) is the gray value of the pixels around the key point, and x, y are the coordinates of the key point;

[0055] Calculate the rotation BRIEF descriptor for each key point, and select a set of fixed pixel point pairs (p i , q i ) in a predefined image area around the key point, where i = 1, 2,..., N; for each pair of pixel points (p i , q i) Rotate it according to the direction θ of the key point and calculate the grayscale value difference:

[0056]

[0057] Arrange all d i in order to form a binary descriptor vector of length N; then perform feature matching, perform descriptor matching between image pairs of adjacent viewpoints, use the Hamming distance as the similarity metric, and select the best matching pair; the distance calculation formula in the matching process is:

[0058]

[0059] where d A and d B are the descriptor vectors in two images respectively, represents the bitwise exclusive OR operation;

[0060] Adopt the Lowe ratio test to screen out high-quality matching pairs, and retain those matching point pairs with ratios lower than the preset threshold of 0.80;

[0061] Based on the matched key point pairs, use the five-point method to calculate the initial pose of the camera, including the rotation matrix R and the translation vector t. The pose estimation formula is as follows:

[0062] p i =K[R|t]P i (4)

[0063] where p i is the coordinate of the key point in the image, K is the camera internal parameter matrix, R and t are the rotation matrix and translation vector of the camera respectively, and P i is the coordinate of the point in the three-dimensional space;

[0064] Generate a sparse point cloud using the initial pose; construct a preliminary sparse point cloud model through the camera pose and key point matching results obtained by the SFM algorithm; use the triangulation method to convert the matched two-dimensional key point pairs into point coordinates in the three-dimensional space. The reconstruction formula is as follows:

[0065]

[0066] where, and are the matched key points from viewpoints A and B respectively, R A ,t a and R B ,t B are the camera poses corresponding to the viewpoints. Merge the three-dimensional point cloud data from different viewpoints to form an overall sparse point cloud model; during the merging process, remove duplicate points and noise points to maintain the sparsity and accuracy of the point cloud;

[0067] Step 1.3) Precise alignment of camera pose; Convert each aligned depth map into three-dimensional point cloud data and perform coordinate transformation using the camera intrinsic matrix K;

[0068]

[0069] where (u, v) are pixel coordinates, d(u, v) is the depth value corresponding to the pixel, and P is the coordinate value of the point in three-dimensional space under the corresponding camera pose;

[0070] Using the ICP algorithm, take the camera pose obtained in Step 1.2) as the initial pose, and perform fine registration on the three-dimensional point cloud converted from the depth map. Under the initial pose, find the nearest corresponding point pairs between the point clouds under different camera poses; By minimizing the Euclidean distance between the point pairs, iteratively optimize the rotation matrix R and t, and optimize the objective function;

[0071]

[0072] In the formula, p i is the target point, q i is the source point, R is the rotation matrix, t is the translation vector, and N is the number of point pairs;

[0073] Correct the initial pose calculated in Step 1.2) with the precise pose result solved by the ICP algorithm to precisely align the camera pose;

[0074] Step 1.4) Generate dense point cloud data; Utilize the precise camera pose obtained in Step 1.3) to fuse the depth point cloud data of each aligned view, and adopt voxel grid filtering and weight fusion to process the overlapping regions existing in multiple views to make the point cloud density uniform; The voxel grid filtering first divides the three-dimensional space into voxel grids, and then selects representative points within each voxel to reduce the amount of point cloud data and maintain the uniformity of the spatial distribution of the point cloud. The formula is as follows:

[0075]

[0076] In the formula, P' is the representative point within the voxel, and N v is the number within the voxel;

[0077] Then assign a weight to each point. The weight is based on the confidence of the point. In the overlapping region, perform weighted averaging on multiple points at the same spatial position to reduce noise and duplicate points. The formula is as follows:

[0078]

[0079] where P fused is the coordinate of the fused point, and w iis the weight of the i-th point, and M is the number of points in the overlapping area, thereby generating prior dense point cloud data;

[0080] Step 2), referring to Figure 3 , initialize and constrain Gaussian points based on the depth prior point cloud:

[0081] Step 2.1) Use the prior dense point cloud as a benchmark to eliminate the wrong points in the sparse point cloud, and use statistical outlier detection to screen the abnormal points in the sparse point cloud to ensure the high quality and geometric consistency of the point cloud data. Then integrate the sparse point cloud into the prior dense point cloud;

[0082] Step 2.2) Determine the position of the initial Gaussian points according to the spatial distribution and scale characteristics of the optimized prior dense point cloud. Use the K-means algorithm to cluster the prior dense point cloud, and take the center coordinates of the point cloud cluster as the initial position of the Gaussian points. Calculate the covariance matrix of each cluster to define the shape and direction of the Gaussian ellipsoid;

[0083] Step 2.3) Introduce geometric constraints to impose restrictions on the generation of Gaussian ellipsoids. Use the boundary of the prior dense point cloud as a geometric constraint, and use a constraint function to limit the generated 3D Gaussian points within a preset geometric range; First, extract the boundary point set B of the prior dense point cloud and its normal vector n b , when generating Gaussian ellipsoids during the training process, only allow the center position u of the ellipsoid i to be within the geometric range defined by the boundary; Define the effective region V within the boundary, and use the boundary point set B to construct the concave hull of the point cloud to fit the complex surface of the engine and improve the accuracy of the constraint;

[0084]

[0085] The details of the concave hull can be controlled by adjusting the parameter ∝. A smaller ∝ value allows more depressions and is suitable for complex structures such as aeroengines; Based on the k-neighborhood of each boundary point, gradually connect to form a concave hull, and the effective region V is defined as the spatial region inside the concave hull, that is: Inside the spatial region, that is:

[0086]

[0087] For each Gaussian ellipsoid Only when μ i ∈ V, generation and subsequent optimization are performed;

[0088] Step 2.4) Maintenance and optimization of geometric consistency during the training process; During the model training process, continuously apply geometric constraints to ensure that the generated 3D Gaussian ellipsoids maintain geometric consistency and avoid random generation; In each training iteration, verify the center position of the generated Gaussian ellipsoid of Whether it is within the effective space V:

[0089]

[0090] If not satisfied, the generated Gaussian ellipsoid is removed;

[0091] Step 3), referring to Figure 3 , 3D reconstruction based on the adjacent multi-view enhancement strategy:

[0092] Step 3.1) Pose graph construction; according to the camera poses obtained in Step 1.3), obtain the camera pose matrix T i and R i , and construct a pose graph where each node represents the pose T i and R i of the camera at the i-th moment, and the edge defines the relationship between two observations of adjacent cameras, and at the same time marks whether the connection between adjacent frames is reliable; for each pair of adjacent views (v i , v j ), calculate the relative pose transformation:

[0093]

[0094] Define the edge e ij ∈ε as a connection containing the relative pose information ΔT ij , ΔR ij ; the initial camera pose is used as the initial node to ensure that the pose graph is connected so that information can be propagated and optimized throughout the graph;

[0095] Step 3.2) Obtain adjacent views with overlapping frustums; the frustum of each camera view is determined by the internal and external parameters of the camera, and the geometric representation of the frustum can be represented by the projection matrix P i of the camera:

[0096] P i = K i [R i t i (14)

[0097] where K i is the internal parameter matrix of camera i, R i is the rotation matrix of the camera, and t i is the translation vector of camera i;

[0098] Obtain a pair of adjacent views (i, j) from the pose graph, transform the frustum of camera j into the coordinate system of camera i, and use the relative pose transformation matrix T ij :

[0099]

[0100] Define the frustum vertex. Let the distance of the near clipping plane of the frustum be n, the distance of the far clipping plane be f, and the horizontal field of view angle be θ. Then the vertices of the frustums of cameras i and j in their coordinate systems can be expressed as:

[0101]

[0102] Use the Sutherland-Hodgman algorithm to calculate the intersection area of the two frustums. If the volume V of the intersection area intersection satisfies:

[0103] V intersection ≥τ·V min (17)

[0104] where τ is a preset overlap threshold, and V min =min(V i ,V′ j ). When the above conditions are met, it is considered that there is a significant view overlap between the two viewpoints;

[0105] To avoid incorrect overlaps of relative or back-to-back viewpoints, the judgment of view direction consistency is added; First, calculate the view direction vector. Let the view direction vector of each camera be d i , usually the optical axis direction of the camera, that is, the third column of the rotation matrix R i :

[0106] d i =R i [:,3] (18)

[0107] Calculate the angle α between the view direction vectors ij :

[0108]

[0109] Set an angle threshold θ max . If θ min <α ij ≤θ max , it is considered that the two view directions are consistent and suitable for multi-view reconstruction;

[0110] The intersection over union of the frustum quantifies the overlap degree of the frustum. The formula is as follows:

[0111]

[0112] where V i is the frustum volume of camera i, and V j' is the frustum volume after the camera j is transformed to the coordinate system of camera i; When IoU ij ≥τ and θ min <α ij ≤θ max , then the current adjacent view pair meets the requirements;

[0113] Step 3.3) Multi-view loss calculation; In step 3.2), multiple camera views with overlapping frustums are obtained and used as the training camera view list. According to the camera views in the list, the current Gaussian distribution and rendering pipeline are used for image rendering to obtain the rendering results corresponding to each camera view; Calculate the loss for the corresponding view. The L1 loss calculation formula is as follows:

[0114]

[0115] where N is the total number of pixels in the image, I r (i) is the value of the rendered image at the i-th pixel, and I g (i) is the value of the real image at the i-th pixel;

[0116] The SSIM loss is used to measure the structural similarity between two images. Its calculation formula is:

[0117]

[0118] where x and y represent the pixel values of the rendered image and the real image respectively, u x and u y are the local averages of image x and image y respectively, and are the local variances of the image respectively, and σ xy is the local covariance of image x and image y; C1 and C2 are stable constants;

[0119] The loss calculation formula for a single camera view is:

[0120]

[0121] Add up the losses of all views in the training camera view list, and then perform backpropagation to update the parameters of the Gaussian ellipsoid;

[0122] Step 3.4) 3DGS three-dimensional scene reconstruction; According to steps 2) and 3), optimize spatial and visual consistency to achieve high-quality 3DGS scene reconstruction, generate the final three-dimensional scene model, ensure that the generated three-dimensional scene has high details and consistency in each view, meet the requirements of spatial and visual consistency, break through the limitations of single-view reconstruction, and improve the reconstruction quality of the entire scene;

[0123] Step 4) Understanding the reconstructed scene based on semantic segmentation:

[0124] Step 4.1) Semantic segmentation model training and application; collect and label a large number of aero-engine RGB image datasets with semantic category labels for training a deep learning semantic segmentation model. Each pixel is assigned a category label, such as blade, stator, rotor, bolt, etc.; adopt the U-Net deep learning model architecture, use the labeled dataset, and train the semantic segmentation model through supervised learning methods to optimize the model parameters to minimize the classification error;

[0125] After the model training is completed, use the pictures used to reconstruct the scene as input and input them into the trained U-Net model to obtain the category prediction results of each pixel:

[0126]

[0127] where I RGB is the input RGB image, is the predicted category label map output by the model;

[0128] Extract the category label of each pixel from the prediction results output by the model to form a semantic category label map;

[0129] Y pred (x, y) = argmax c P(c|I RGB (x, y))(26)

[0130] where Y pred (x, y) is the predicted category label at the coordinate (x, y), and P(c|I RGB (x, y)) is the predicted probability of category c;

[0131] Step 4.2) Project semantic labels into three-dimensional space;

[0132] According to the pose and projection relationship of the camera, map the semantic labels in the RGB image to or near the Gaussian points in three-dimensional space; K i is the camera pose, obtain the projection matrix P from Equation (14) i , according to the camera's internal and external parameters, back-project the two-dimensional coordinates of each pixel into three-dimensional space, and then transform it to the unified coordinate system according to the camera pose to obtain the three-dimensional coordinates of the corresponding Gaussian ellipsoid point;

[0133]

[0134] Map the category label Y pred (x, y) in the semantic category label map to the corresponding Gaussian ellipsoid point X to form the semantic category label C(X) of each Gaussian point;

[0135] C(X) = Y pred (x, y)(28) is each Gaussian ellipsoid point X j Store its corresponding semantic category label C j ;

[0136] Step 4.3) Semantic feature weighted fusion strategy:

[0137] For each Gaussian ellipsoid point X j , according to its semantic category label C mapped to each perspective j , generate the corresponding semantic feature vector

[0138]

[0139] where k represents the camera perspective number, is the semantic category label of Gaussian point j under perspective k, and OneHot(·) is the one-hot encoding function; normalize the semantic feature vector to ensure that the feature vectors under different perspectives have a consistent scale;

[0140]

[0141] Calculate the weights of the semantic features of each perspective according to the reliability w k ;

[0142]

[0143] where α k is the weight parameter of perspective k, and M is the total number of perspectives;

[0144] After obtaining the weights of the semantic features of each perspective, perform semantic feature weighted fusion, and weight-sum the semantic feature vectors of the Gaussian ellipsoids at the same position from different perspectives by weight w k to obtain the final semantic feature vector f j ;

[0145]

[0146] Apply the Softmax function to the weighted-fused semantic feature vector f j to calculate the probability distribution P j (c);

[0147]

[0148] where C is the total number of semantic categories;

[0149] Take the semantic category with the highest probability as the Gaussian point X jThe final semantic label C j ;

[0150] C j = argmax c P j (c)(34)

[0151] Step 4.4) Core component identification and marking; From the set S of Gaussian points of the entire scene obtained in step 3), combined with the final semantic label information obtained in step 4.3), filter out the Gaussian ellipsoid points belonging to predefined core component categories such as the rotor and stator, group them according to spatial proximity, and group the mutually close Gaussian ellipsoid points into one group through a preset spatial threshold to classify the Gaussian points that may not have been assigned semantic labels. The formula is as follows:

[0152]

[0153] where γ is the spatial proximity threshold, X j is the three-dimensional coordinate of the center of the Gaussian ellipsoid point X k is the reference point coordinate of group k;

[0154] In this way, a group of Gaussian ellipsoid points with the same label category represents a core component and can be marked as a core component instance, and a unique identifier ID k is assigned for subsequent virtual-real fusion;

[0155] Step 5) Reconstruct the scene for augmented reality virtual-real fusion visualization:

[0156] Step 5.1) Define the virtual model, display rules, and display conditions of the core components; Define the corresponding virtual object models and their attributes for each core component category of the aeroengine, including basic dimension information, uses, etc.; Set the display attributes for the virtual model, including transparency, movability, collision attributes, etc.; Set the display conditions for the virtual model, which can dynamically adjust the display state of the virtual object according to the user's current perspective and the distance to the core component, realizing display, hiding, size scaling, etc., and can also adjust the display content and method of the virtual object according to the current user's task requirements, such as highlighting, target flashing, etc.; Allow the user to move, rotate, etc. the object through gestures to adjust the position and attributes of the object, and support the user to control the display state of the virtual object or trigger specific operations through voice commands;

[0157] Step 5.2) Virtual-real fusion rendering and dynamic interactive display:

[0158] Using graphics rendering technology, coordinate the visual attributes such as color and lighting of virtual objects with the real scene to ensure natural visual integration; be able to respond to user interactions in real time, and according to the user's gesture or voice commands, adjust the position, direction and display attributes of virtual objects in real time to achieve interactive operations;

[0159]

[0160] Among them, f is an update function that updates the state of virtual objects according to the interaction method and instructions;

[0161] At the same time, it will dynamically adjust the display attributes of virtual objects according to the changes in the user's perspective or the scene environment to maintain coordination with the real scene.

[0162]

[0163] Among them, g is an adjustment function that updates the display rules according to the perspective and environment changes;

[0164] Step 5.3) Build an information database; establish training, assembly, and maintenance information databases, including training tutorials, operation steps, precautions, assembly processes, tool usage instructions, assembly step animations, maintenance manuals, repair steps, etc., and use a NoSQL database to store information to ensure efficient access and management of information;

[0165] Step 5.4) Augmented reality virtual-real fusion visualization; users wear HoloLens glasses, the system loads software modules, and visualizes the augmented reality virtual-real fusion of the aero-engine model and the operation interface. Multiple users can view and operate the aero-engine virtual model synchronously through their respective HoloLens devices to achieve collaboration among users. Users can operate the virtual model and database information through interaction methods such as gestures, voices, and menus, which is convenient for more intuitive training, assembly, maintenance, etc.; The visualization screen of the three-dimensional reconstruction scene in this embodiment is as Figure 4 shown, and it can be seen that there are no artifacts, and there is high reconstruction quality and geometric consistency.

[0166] Those skilled in the art of this technology can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used here have the same meaning as the general understanding of those of ordinary skill in the art to which this invention belongs. It should also be understood that terms defined in common dictionaries should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as here.

[0167] It should be understood that the above are only specific embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for virtual-real fusion of aircraft engine augmented reality based on deep prior scene reconstruction, characterized in that: The following steps are involved: Step 1) Target object surround data collection and camera pose solution: Use a depth camera to collect multi-view data of the target object, ensure that there is overlap between viewpoints, and obtain high-quality depth maps and RGB images; Use the SFM algorithm to process the collected RGB images, extract ORB key points and match them, and then calculate the camera frame pose to obtain the camera pose and sparse point cloud of each viewpoint; Use the ICP algorithm to accurately align the calculated initial camera frame pose with the depth map data, correct the pose error in the point cloud, ensure that data from different viewpoints can be accurately aligned, and generate aligned prior dense point cloud data; Step 2) Initialize and constrain Gaussian points based on deep prior point cloud and sparse point cloud: The prior dense point cloud obtained in step 1) is used as a benchmark to eliminate erroneous points in the sparse point cloud; determine the position of the initial Gaussian points and initialize the high-quality Gaussian ellipsoid according to the spatial distribution and scale characteristics of the optimized prior dense point cloud; use the boundary of the dense point cloud as a geometric constraint, introduce a constraint function, and limit the generated 3D Gaussian points to a preset geometric range, so that the geometric characteristics are always consistent during the training process, ensuring that the Gaussian ellipsoid accurately fits the engine surface, accurately reflects the geometric characteristics of the actual object, and is consistent with the real world; Step 3) 3D reconstruction based on adjacent multi-view enhancement strategy: In the 3D reconstruction training process, a deep learning method of adjacent multi-view enhancement strategy is introduced to create a pose graph to represent the pose information of all cameras and their mutual relationship. The nodes represent the poses at each moment, and the edges represent the relative position information. The pose graph is combined to calculate adjacent multi-view images with overlapping cones, learn the geometric structure relationship and texture consistency between adjacent views, enhance the model's ability to capture complex surface details, and ensure spatial consistency and visual consistency. Step 4) Reconstruct the target based on semantic segmentation understanding: Use the U-Net deep learning model to perform semantic segmentation on the RGB image, extract pixel category labels, map the semantic category labels to Gaussian ellipsoid points through camera pose and projection relationship, integrate the semantic features of different perspectives through weighted fusion strategy to form the final semantic label of Gaussian points, and identify and mark the core components of the stator and rotor; Step 5) Reconstruct the scene augmented reality virtual-reality fusion visualization: Based on the aircraft engine reconstructed in step 3) and the category information of the core components obtained in step 4), define the display rules, display conditions, and interaction methods of virtual objects for different components to ensure the geometric alignment and fusion of virtual information and the real scene, perform virtual-reality fusion rendering and dynamic display, and adjust the object display state or attributes according to the interaction instructions and scene environment changes; at the same time, build an information database for training, assembly, and maintenance to realize the virtual-reality fusion display of relevant information and guide users to perform related operations.

2. The method according to claim 1, characterized in that: Step 1) The target object surround data collection and camera pose solution are as follows: Step 1.1) Multi-view data collection of aircraft engines: Use the Realsense depth camera to collect surround data of the aircraft engine, obtain the corresponding high-quality depth map and RGB image, and extract frames of the obtained images to ensure that the overlap rate between adjacent viewpoints is no more than 80%; Step 1.2) SFM calculates the initial camera pose and generates a sparse point cloud; uses the SFM algorithm to process the collected RGB images, uses the ORB algorithm to extract key points, uses FAST to detect corner points in the image, calculates the rotation BRIEF descriptor for each key point, and then performs feature matching, performs descriptor matching between image pairs of adjacent perspectives, selects high-quality matching pairs, and uses the five-point method to calculate the initial camera pose; uses the initial pose to generate a sparse point cloud, and merges the point clouds from different perspectives to form an overall sparse point cloud model; During the merging process, duplicate points and noise points are removed to maintain the sparsity and accuracy of the point cloud; Step 1.3) Accurately align the camera poses; convert each aligned depth map into three-dimensional point cloud data, and use the camera intrinsic parameter matrix K to perform coordinate transformation; use the ICP algorithm, take the camera pose obtained in step 1.2) as the initial pose, perform ICP precise registration on the three-dimensional point clouds converted from adjacent depth maps, and use the precise pose result solved by the ICP algorithm to correct the initial pose calculated in step 1.2), and accurately align the camera poses; Step 1.4) generates dense point cloud data; using the precise camera pose obtained in step 1.3), the aligned depth point cloud data of each view are fused, and voxel grid filtering and weight fusion are used to process the overlapping areas of multiple views to make the point cloud density uniform; a weight is assigned to each point, and the weight is based on the confidence of the point. In the overlapping area, multiple points at the same spatial position are weighted averaged to reduce noise and duplicate points.

3. The method according to claim 1, characterized in that Step 2) Initialize Gaussian points based on depth prior data and sparse point cloud as follows: Step 2.1) The prior dense point cloud is used as a benchmark to remove erroneous points, and statistical outlier detection is used to screen out abnormal points in the sparse point cloud to ensure the high quality and geometric consistency of the point cloud data, and then the sparse point cloud is integrated into the prior dense point cloud; Step 2.2) According to the optimized spatial distribution and scale characteristics of the prior dense point cloud, the position of the initial Gaussian point is determined, the prior dense point cloud is clustered using the K-means algorithm, the center coordinates of the point cloud cluster are used as the initial position of the Gaussian point, and the covariance matrix of each cluster is calculated to define the shape and direction of the Gaussian ellipsoid; Step 2.3) Introduce geometric constraints to impose restrictions on the generation of Gaussian ellipsoids. Use the boundary of the prior dense point cloud as the geometric constraint, and use the constraint function to limit the generated 3D Gaussian points to the preset geometric range. When generating the Gaussian ellipsoid during training, only the ellipsoid center position u is allowed to i Within the geometric range defined by the boundary; for each Gaussian ellipsoid Only when μ i ∈V, generation and subsequent optimization are performed; Step 2.4) Geometric consistency maintenance and optimization during training: During the model training process, geometric constraints are continuously applied to ensure that the generated 3D Gaussian ellipsoid maintains geometric consistency.

4. The method according to claim 1, characterized in that: Step 3) The three-dimensional reconstruction based on the adjacent multi-view enhancement strategy is specifically as follows: Step 3.1) Pose graph construction: According to the camera pose, obtain the camera pose matrix T corresponding to each image i and R i , construct a pose graph; Step 3.2) Obtain adjacent viewpoints with overlapping view cones; obtain a pair of adjacent viewpoints from the pose graph, and use the Sutherland-Hodgman algorithm to calculate the intersection area of ​​the two view cones; increase the judgment of view direction consistency, when IoU ij ≥τ and θ min <α ij ≤θ max , then the current adjacent perspective pair meets the requirements; Step 3.3) Calculate multi-view loss; Step 3.2) Obtain multiple camera viewpoints with overlapping view cones, use them as a training camera viewpoint list, perform image rendering based on the camera viewpoints in the list, combined with the current Gaussian distribution and rendering pipeline, and obtain the rendering result corresponding to each camera viewpoint; Calculate the loss of the corresponding view, add up the losses of all the viewpoints in the training camera viewpoint list, and then perform backpropagation to update the parameters of the Gaussian ellipsoid; Step 3.4) 3DGS three-dimensional scene reconstruction; optimize spatial and visual consistency, achieve high-quality 3DGS scene reconstruction, generate the final three-dimensional scene model, ensure that the generated three-dimensional scene has high details and consistency under various viewing angles, and meet the spatial and visual consistency requirements.

5. The method according to claim 1, characterized in that Step 4) Reconstructing the scene based on semantic segmentation understanding is as follows: Step 4.1) Semantic segmentation model training and application; collect and annotate a dataset of aircraft engine RGB images with semantic category labels for training deep learning semantic segmentation models, with each pixel being assigned a category label; use the U-Net deep learning model architecture, use the annotated dataset, train the semantic segmentation model through supervised learning methods, and optimize the model parameters to minimize the classification error; after the model training is completed, use the image used to reconstruct the scene as input and input it into the trained U-Net model to obtain the category prediction result for each pixel; extract the category label of each pixel from the prediction result output by the model to form a semantic category label map; Step 4.2) The semantic label is projected into three-dimensional space; according to the camera’s position and projection relationship, the category label Y in the semantic category label map is projected into pred (x, y) is mapped to the corresponding Gaussian ellipsoid point X to form the semantic category label C(X) of each Gaussian point; Step 4.3) Semantic feature weighted fusion strategy: for each Gaussian ellipsoid point X j , according to its mapping to the semantic category label C of each perspective j , generate the corresponding semantic feature vector Normalize the semantic feature vectors to ensure that the feature vectors under different perspectives have consistent scales; according to the reliability of each perspective w k , calculate the weight of the semantic features of each perspective; After obtaining the weights of the semantic features of each perspective, the semantic features are weighted fused to combine the semantic feature vectors of the Gaussian ellipsoid at the same position from different perspectives. By weight w k Perform weighted summation to obtain the final semantic feature vector f j ; The semantic feature vector f after weighted fusion j Apply the Softmax function to calculate the probability distribution P of each semantic category j (c); Take the semantic category with the highest probability as the Gaussian point X j The final semantic label C j ; Step 4.4) Identification and labeling of core components; from the Gaussian point set S of the entire scene, combined with the final semantic label information obtained in step 4.3), filter out Gaussian ellipsoid points belonging to the predefined core component category, group them according to spatial proximity, and group the Gaussian ellipsoid points close to each other into a group through a preset spatial threshold, and classify the Gaussian points that may not be assigned semantic labels; a group of Gaussian ellipsoid points with the same label category represents a core component, is marked as a core component instance, and is assigned a unique identifier ID k , for subsequent virtual-real fusion use.

6. The method according to claim 1, characterized in that Step 5) Reconstructing the scene augmented reality virtual-real fusion visualization is as follows: Step 5.1) Define the virtual model, display rules and display conditions of the core components; define the corresponding virtual object model and its attributes for each type of core component of the aircraft engine, including basic size information and purpose; set display attributes for the virtual model, including transparency, movable and collision attributes; set the display conditions of the virtual model, and dynamically adjust the display state of the virtual object; or adjust the display content and mode of the virtual object according to the current user's task requirements, allowing the user to adjust the position and attributes of the object through gestures to move and rotate, and support the user to control the display state of the virtual object or trigger specific operations through voice commands; Step 5.2) Virtual-reality fusion rendering and dynamic interactive display; coordinate the visual attributes of the color and lighting of the virtual object with the real scene to ensure natural visual integration; be able to respond to user interaction in real time, adjust the position, direction and display attributes of the virtual object in real time according to the user's gestures or voice commands, and realize interactive operation; at the same time, dynamically adjust the display attributes of the virtual object according to changes in the user's perspective or scene environment to maintain coordination and consistency with the real scene; Step 5.3) Build an information database; establish a training, assembly, and maintenance information database, including training tutorials, operating procedures, precautions, assembly processes, tool instructions, assembly step animations, maintenance manuals, and repair steps, and use a NoSQL database to store information; Step 5.4) Augmented reality visualization: The user wears HoloLens glasses, the system loads the software module, and the augmented reality visualization of the aircraft engine model and operation interface is visualized by fusion of virtual and real. Multiple users can view and operate the aircraft engine virtual model synchronously through their respective HoloLens devices to achieve collaboration between users. Users can operate the virtual model and database information through gestures, voice, and menu interactions, which facilitates more intuitive training, assembly, and maintenance.

Images