An augmented reality-based multi-person cooperative guidance docking method for aircraft wings

Abstract

An augmented reality-based multi-person collaborative guidance and docking method for aircraft wings includes steps such as inter-frame transformation matrix calculation and 3D reconstruction, creation of a high-precision pose estimation dataset, construction of a deep learning framework, wing axis instance segmentation based on the FCN model, wing axis pose estimation method based on the DenseFusion network, multi-terminal world scene construction, wing axis synchronization method and access control, and visualization method for multi-person collaborative guidance and docking. This method applies visual and image processing methods to component pose estimation and proposes a multi-functional approach integrating wing axis pose estimation, 3D registration, multi-person collaboration, and visual guidance. Utilizing deep learning and image processing methods, the wing axis pose can be estimated, improving both efficiency and accuracy. The method is scalable and portable, easily adaptable to different component types. It also offers high visualization capabilities, improving assembly line efficiency and enhancing the overall intelligence of the production line.

Description

Technical Field

[0001] This invention relates to the field of aircraft wing collaborative assembly technology, specifically to an augmented reality-based multi-person collaborative guidance and docking method for aircraft wings. Background Technology

[0002] Advanced aviation equipment is a crucial component of modern national defense. Its assembly cycle accounts for approximately 60% of the total manufacturing cycle, and is currently a significant reason for insufficient production capacity. Shortening the assembly cycle of advanced aviation equipment has a vital impact on improving aircraft quality, reducing costs, and increasing production capacity. In the collaborative assembly of aircraft wings, untimely and insufficient information sharing between workstations and a lack of unified instructions lead to high demands on worker skill levels and low efficiency. Therefore, effectively improving the collaborative assembly efficiency of aircraft wings can boost the production capacity of advanced aviation equipment.

[0003] To improve the efficiency and quality of multi-person collaborative assembly of aircraft wings, it is necessary to combine methods such as spatial pose estimation, wing axis information synchronization, and visualization of multi-person collaborative assembly guidance information with augmented reality technology to solve the problems of difficult and untimely information sharing in multi-person collaborative assembly. For example, Chinese patent CN201910673931.0, entitled "A Virtual Assembly Method Based on Leap Motion and Local Area Network Collaboration", works by acquiring gesture data through Leap Motion sensors for interaction and inferring the virtual assembly scene and the state of the wing axis. The disadvantages are that it relies on external sensors and assembles in a virtual environment, which has significant limitations in the actual work site. Using augmented reality (AR) technology to guide aircraft wing assembly requires a visualization effect that blends the virtual and real worlds. This involves overlaying virtual information onto the physical object based on its pose information. For example, Chinese patent CN202110994137.3, entitled "A Multi-User Collaborative AR Assembly Method and System Based on Digital Twins," maps the physical assembly scene to a virtual one using digital twin data and displays the assembly information in augmented reality. However, it lacks a direct pose estimation method for specific assembly objects, offering only scene-level AR visualization and collaboration. It fails to provide object-level AR visualization and collaboration for on-site guidance. Therefore, a multi-user collaborative guidance and docking method for aircraft wings based on augmented reality is needed. This method should focus on wing axis pose estimation and multi-user collaborative assembly technology, overcoming the challenge of rapidly generating large datasets using deep learning-based pose estimation methods in real-world environments. It should also establish multi-terminal data synchronization channels to achieve timely and accurate prompts for multi-user collaborative assembly, thereby improving the efficiency of multi-user collaborative assembly. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, the present invention aims to provide an augmented reality-based multi-person collaborative guidance and docking method for aircraft wings, thereby improving the efficiency and accuracy of wing docking.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] An augmented reality-based multi-person collaborative guidance and docking method for aircraft wings includes a dataset creation method based on inter-frame matrix transformation, a pose estimation method based on instance segmentation, and a guidance and docking method for multi-person collaboration. Specifically, it includes the following steps:

[0007] Step 1), Calculation of inter-frame transformation matrix and 3D reconstruction: Use a depth camera to acquire point cloud data of multiple frames from different perspectives of the wing axis, and calculate all inter-frame transformation matrices of the wing axis point cloud data through coarse and fine registration; transform all the point cloud data of different perspectives of all frames to the point cloud coordinate system of the initial frame through the above transformation matrices, and stitch the point cloud data of different perspectives of the wing axis to obtain complete wing axis point cloud data and the pose of the initial frame.

[0008] Step 2), High-precision pose estimation dataset creation: Filter the stitched wing axis point cloud data from Step 1), align the wing axis theoretical model with the filtered stitched wing axis point cloud pose through point cloud registration; Based on the transformation matrix of the point cloud data between each frame in Step 1), sequentially transform the pose of the high-precision point cloud of the initial frame to obtain the high-precision wing axis theoretical model point cloud in all frames, calculate the bounding box of the model point cloud, and thus obtain the high-precision pose estimation dataset for each frame of the wing axis.

[0009] Step 3), Network Training: Combining the multi-view pose estimation dataset built in Step 1) and Step 2), extract data such as depth map, RGB image, mask image, label, and transformation matrix, build a deep learning framework, and train a fully convolutional network (FCN) for instance segmentation and a DenseFusion network for wing axis pose estimation.

[0010] Step 4), Wing axis instance segmentation based on FCN model: During the wing docking process, for the original images such as color images and depth images obtained from the depth camera, the target region in the original image is obtained based on the fully convolutional network (FCN), thereby realizing image segmentation of the region where the wing axis is located in the original image;

[0011] Step 5), Wing axis pose estimation method based on DenseFusion network: Based on the image segmentation in step 4), the result of instance segmentation is obtained as index data. Through data preprocessing, backbone network inference and iterative self-optimization process, the spatial pose data of the wing axis relative to the observer is obtained using DenseFusion network.

[0012] Step 6), Multi-terminal world scene construction: First, build a multi-terminal world scene framework based on Unity Mirror, specifying the server and client; second, realize the synchronization of the coordinate system of multiple terminals based on fixed identification codes; finally, obtain the coordinate information of the sensing device according to the multi-feature identification code, and combine it with the wing axis pose obtained in step 5) to realize the 3D registration of the wing axis in the world scene.

[0013] Step 7), Wing Axis Synchronization Method and Access Lock: First, render the virtual model of the wing axis and obtain the initial pose of the wing axis; second, collect the changes of the wing axis in space in real time and synchronize them to other terminals by the data synchronizer; set an access lock to allocate unique execution permissions, and call the wing axis synchronization method when the judgment function determines that the current terminal has execution permissions.

[0014] Step 8), Multi-person collaborative guided docking visualization: First, a standard wing shaft docking process database is constructed to enable the storage and retrieval of docking information, including the wing shaft 3D model, wing shaft docking steps, and wing shaft guided docking; second, based on the wing shaft pose information obtained in Step 5), the server is used for rendering to provide wing shaft guided docking information; finally, the guidance information is distributed to multiple terminals for collaborative guided docking, the wing shaft pose is estimated in real time, and the guidance information is updated until the docking task is completed.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0016] (1) This invention proposes a method for creating a pose estimation dataset based on inter-frame matrix transformation. Based on RGBD data and point cloud data, it consists of four technical stages, including inter-frame pose transformation matrix calculation, high-precision model conversion and multi-frame dataset generation. The pose data labeled by this method is accurate, the whole production process is relatively simple, and the generated dataset can contain multiple perspectives, different backgrounds and lighting conditions, which can meet the needs of subsequent pose estimation methods.

[0017] (2) Based on the real-time physical environment, this invention proposes a new method for wing axis pose estimation using FCN and DenseFusion deep learning networks. It consists of three technical stages, including network training, instance segmentation and pose estimation. It can accurately predict the wing axis position and attitude without relying on the structural features of the wing axis, has good transferability, and can meet the real-time requirements of multi-person collaboration.

[0018] (3) This invention utilizes augmented reality technology to achieve three-dimensional registration of the wing shaft in the world scene, providing real, intuitive and sufficient assembly information sharing. Combined with the proposed multi-person collaborative guided docking visualization method, it provides wing shaft guided docking information, completes docking tasks efficiently and with high quality, and improves docking efficiency and quality. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the process of the present invention.

[0020] Figure 2 This is a schematic diagram illustrating the process of creating the pose estimation dataset based on inter-frame matrix transformation according to the present invention.

[0021] Figure 3 A schematic diagram of a multi-person collaborative assembly guidance process for wing shaft guided docking.

[0022] Figure 4 A schematic diagram of the scene setup for a depth camera to acquire point cloud data from multiple frames of different perspectives along the wing axis.

[0023] Figure 5 The final result of creating a pose estimation dataset based on inter-frame matrix transformation.

[0024] Figure 6 The images show the results of the instance segmentation and pose estimation method for the wing shaft docking scenario, where (a) is the segmentation result of the FCN network and (b) is the pose estimation result of the DenseFusion network.

[0025] Figure 7 The following are demonstration images of a multi-person collaborative assembly guidance system for wing shaft docking scenarios; (a) is the scene information in HoloLens2 A, and (b) is the scene information in HoloLens2 B. Detailed Implementation

[0026] The present invention will now be described in detail with reference to embodiments and accompanying drawings. The present invention can be implemented in many different forms and should not be considered limited to the embodiments described herein. Rather, these embodiments are provided so that the present invention will be thoroughly and completely disclosed and will fully express the scope of the invention to those skilled in the art.

[0027] like Figure 1 As shown, an augmented reality-based multi-person collaborative guidance and docking method for aircraft wings includes a dataset creation method based on inter-frame matrix transformation, a pose estimation method based on instance segmentation, and an auxiliary guidance method for multi-person collaboration. The dataset creation method based on inter-frame matrix transformation includes step 1) calculating the inter-frame transformation matrix and step 2) creating a 3D reconstruction and high-precision pose estimation dataset. The specific flowchart is shown below. Figure 2As shown; the instance segmentation-based pose estimation method includes step 3) network training, step 4) wing axis instance segmentation based on the FCN model, and step 5) wing axis pose estimation method based on the DenseFusion network; the multi-user collaborative assisted guidance docking method includes step 6) multi-terminal world scene construction, step 7) wing axis synchronization method and permission lock, and step 8) multi-user collaborative guidance docking visualization, the specific flowchart of which is shown in the figure. Figure 3 As shown;

[0028] A multi-person collaborative guidance and docking method for aircraft wings based on augmented reality, specifically including the following steps:

[0029] Step 1), Inter-frame transformation matrix calculation and 3D reconstruction: Multiple frames of point cloud data from different viewpoints along the wing axis are acquired using a depth camera. All inter-frame transformation matrices for the wing axis point cloud data are calculated through coarse and fine registration. All frame point cloud data from different viewpoints are transformed to the point cloud coordinate system of the initial frame using the aforementioned transformation matrices. The point cloud data from different viewpoints along the wing axis are then stitched together to obtain complete wing axis point cloud data and the pose of the initial frame. The specific steps are as follows:

[0030] Step 1.1), as Figure 4 As shown, the depth camera acquires point cloud data from multiple frames of different perspectives on the wing axis. Before acquisition, a certain number of Aruco codes are placed around the wing axis as initial positioning references, and it is ensured that there are about 3 QR codes in the field of view during recording.

[0031] Step 1.2): Based on the positions of the Aruco codes between frames, calculate the pose of the depth camera relative to the Aruco codes in each group of images as follows:

[0032]

[0033] In the formula, A represents the position of the Aruco code in the world coordinate system; R n Represents the rotation matrix of the Aruco code relative to the depth camera in the nth data set; A n t represents the position of the depth camera in the world coordinate system in the nth data set; n Let R0, A0, and t0 represent the translation matrix of the Aruco code relative to the depth camera in the nth data set; R0, A0, and t0 represent the matrices corresponding to the initial data sets, respectively; the Aruco codes in the same original data set have the same coordinates in the world coordinate system, therefore the pose relationship between adjacent frames is:

[0034]

[0035] At this point, the coarse pose transformation matrices of the wing axis point clouds between each frame can be calculated, which are respectively the rotation transformation matrices R. n and translation transformation matrix T nHowever, due to the limitations of the depth camera's accuracy, the inter-frame transformation matrices calculated in this study are used as coarse registration results and as the initial transformation matrices in the subsequent fine registration method.

[0036] Step 1.3): Based on the coarse registration transformation matrix obtained in Step 1.2), perform pose transformation using the point cloud of the previous frame as the target point cloud and the point cloud of the next frame as the source point cloud to complete the coarse registration. Then, use the Intermediate Closest Point (ICP) method as the means for fine registration, taking the two coarsely registered point clouds as input, to solve for the pose transformation matrix of the fine registration. The formula is as follows:

[0037]

[0038] In the formula, R * Represents the optimal rotation matrix, t * Represents the optimal translation matrix, and N represents the number of point clouds. Representing the source cloud, Represents the target point cloud, R t Represents the rotation matrix, t t Represents the translation matrix;

[0039] At this point, the coarse and fine registration pose transformation matrices of the wing axis point clouds for all frames can be obtained;

[0040] Step 1.4): Based on the coarse and fine registration pose transformation matrices of the wing axis point clouds between frames obtained in Step 1.2) and Step 1.3), transform all the wing point clouds of all frames to the initial frame to complete the stitching of point clouds from different perspectives of the wing axis, and realize the three-dimensional reconstruction of the wing axis point cloud in the initial frame.

[0041] Step 2), High-precision pose estimation dataset creation: The stitched wing axis point cloud data from Step 1) is filtered, and the wing axis theoretical model is aligned with the filtered stitched wing axis point cloud pose through point cloud registration to improve dataset accuracy. Based on the transformation matrix of the point cloud data between frames in Step 1), the high-precision point cloud of the initial frame is sequentially pose-transformed to obtain the high-precision wing axis theoretical model point cloud in all frames. The bounding box of the model point cloud is calculated, thus obtaining the high-precision pose estimation dataset for the wing axis between frames. The specific steps are as follows:

[0042] Step 2.1) Filter the initial frame wing axis point cloud reconstructed in Step 1). Combining the characteristics of different filtering methods, select pass-through filtering, guided image filtering and radius filtering to process the stitched point cloud information and remove background noise in the reconstructed wing axis point cloud data.

[0043] Step 2.2): Due to the limitations of depth camera accuracy, the wing axis point cloud data filtered in Step 2.1) generally has poor quality and some missing details. Therefore, the theoretical model point cloud of the wing axis is used to replace it to improve the accuracy of the data. To ensure that the spatial poses of the two are consistent after replacement, it is necessary to calculate the spatial pose transformation matrix of the theoretical model point cloud of the wing axis and the point cloud reconstructed by the depth camera. The specific calculation steps are as follows:

[0044] ① The FPFH feature descriptors of the point cloud based on the computer wing axis theory model of the PCL open-source function library and the point cloud reconstructed by the depth camera are used to prepare for subsequent coarse registration.

[0045] ② Based on the Sample Consistency Initial Registration (SAC-IA) method in the PCL open-source function library, the FPFH feature descriptor in step ① is used as the basis for matching. The pose transformation matrix is ​​calculated using the point cloud of the wing axis theoretical model as the source point cloud and the point cloud reconstructed by the depth camera as the target point cloud to achieve coarse registration of the two point clouds.

[0046] ③ The pose transformation matrix of the two wing shaft point cloud data after coarse registration in step ② is calculated using the ICP method to achieve fine registration of the two point clouds;

[0047] ④ Use the polygon_obb method in the Trimesh module to perform polygon convex hull detection, sequentially enclosing the entire point cloud to generate the outer bounding box of the point cloud model;

[0048] This completes the replacement of the wing axis point cloud reconstructed in the initial frame of the depth camera with the high-precision theoretical model point cloud of the wing axis, resulting in the high-precision model point cloud of the wing axis initial frame and the initial position target pose;

[0049] Step 2.3): Combining the inter-frame transformation matrix obtained in Step 1), the high-precision model of the wing axis in the initial frame with the outer bounding box is transformed by the inverse transformation of each inter-frame transformation matrix to obtain the high-precision theoretical model point cloud of the wing axis in all frames, thus obtaining a dataset of high-precision pose estimation of the wing axis in each frame and at each viewpoint; such as Figure 5 As shown, the pose estimation dataset is displayed in different scenes and from different perspectives after it is completed. The red and blue 3D bounding boxes intuitively show the pose estimation results, and the red end indicates the direction of the wing axis.

[0050] Step 3), Network Training: Combining the multi-view pose estimation dataset built in Steps 1) and 2), extract data such as depth maps, RGB images, mask images, labels, and transformation matrices to build a deep learning framework. Train a fully convolutional network (FCN) for instance segmentation and a DenseFusion network for wing axis pose estimation; the specific steps are as follows:

[0051] Step 3.1) The training process for the FCN network includes: constructing an instance segmentation dataset based on the datasets obtained in steps 1) and 2), and using single-channel P-mode images from the PIL library as annotation maps for instance segmentation. Specific steps include:

[0052] First, convert the color image data into grayscale image data:

[0053] y=0.299·R+0.587·G+0.114·B(4)

[0054] In the formula, y represents the converted grayscale value; R, G, and B represent the pixel values ​​of the three channels in the color image;

[0055] The formula for converting a grayscale image to a P-image is as follows:

[0056]

[0057] In the formula, p represents the pixel value in the P-mode image; a color palette containing multiple colors is predefined, and m represents the number of colors in the color palette; the pixel values ​​in the image are changed according to the type of mask image obtained in steps 1) and 2), the mask boundary is defined, and they are combined into a P-mode image through the PIL library. Thus, a dataset for instance segmentation is obtained. After entering the training phase, the loss during the training process is monitored. Training ends when the network loss reaches below 1mm, and the generalization ability is tested to ensure good model quality.

[0058] Step 3.2) The training process for the DenseFusion network includes: performing computation based on the PyTorch framework, selecting the multi-view pose estimation RGBD dataset from Step 1) and Step 2) as the original data, and dividing it into training and validation sets. First, train the backbone network, set the pose estimation threshold, and when the loss value is lower than the threshold, switch to the iterative self-optimization network, and retain the optimal model when the ADD value (average distance between the predicted pose and the true value) reaches the minimum in the above process.

[0059] Step 4), Wing Axis Instance Segmentation Based on FCN Model: During the wing docking process, for the original images such as color images and depth images obtained from the depth camera, the target region in the original image is obtained based on a fully convolutional network (FCN), thereby achieving image segmentation of the region where the wing axis is located in the original image; the specific steps are as follows:

[0060] The input point cloud and RGB image data are processed through a 5-layer convolutional and 5-layer pooling network structure to obtain instance segmentation feature maps from the original image. Simultaneously, 8x up-pooling and deconvolution operations are performed to fuse feature maps from different layers, thereby ensuring the accuracy of instance segmentation. Figure 6(a) shows the segmentation result of the FCN network, indicating that the segmentation of the target region where the axis is located is accurate;

[0061] Step 5), Wing Axis Pose Estimation Method Based on DenseFusion Network: Based on the image segmentation in Step 4), the results of instance segmentation are obtained as index data. Through data preprocessing, backbone network inference, and iterative self-optimization, the spatial pose data of the wing axis relative to the observer is obtained using the DenseFusion network. The specific steps are as follows:

[0062] Step 5.1) For the input RGBD data, the depth map is converted into a 3D point cloud. The point cloud and the aligned depth and RGB image information are used to normalize the imaging plane. The feature information of 500 points is selected as the input of the pose estimation backbone network.

[0063] Step 5.2) Downsample the input color features using ResNet18 and upsample them using four PSPNet pyramid structures to extract color feature information;

[0064] Step 5.3) The input depth features are converted into point cloud information, and PointNet is used to extract geometric features, outputting geometric features with the same dimension as the color information;

[0065] Step 5.4) Input color features and geometric features into the network, perform multiple convolutions and activation function operations, and concatenate to obtain pixel information fused from color and geometric features; perform global pooling to fuse pixel, local, and global information of each pixel into a feature vector, calculate pixel rotation, translation, and confidence parameters, and obtain the pose parameters of the pixel.

[0066] Step 5.5) Based on the rotation and translation parameters output by the backbone network, the pose estimation effect is optimized by iteratively optimizing the PoseRefineNet network. Figure 6 (b) shows the pose estimation results of the DenseFusion network. The 3D bounding box visually verifies the pose estimation results.

[0067] Step 6), Multi-terminal world scene construction: First, build a multi-terminal world scene framework based on Unity Mirror, specifying the server and client; second, synchronize the coordinate systems of multiple terminals based on fixed identifier codes; finally, obtain the coordinate information of the sensing device based on the multi-feature identifier codes, and combine it with the wing axis pose obtained in Step 5) to achieve 3D registration of the wing axis in the world scene; the specific steps are as follows:

[0068] Step 6.1) Use the Unity Mirror framework to build a multi-terminal world scene framework, and establish a network manager based on the NetworkManager component to manage the data stream transmission of multiple terminals; set up the desktop computer with high computing power as the server to comprehensively process scene information, client generation and scene information synchronization; the other terminals are all clients, and render the current client scene according to the information sent by the server when they join the server.

[0069] Step 6.2) Place and fix the identification code at a certain position, and then transmit the color image of the client camera to the Python program on the desktop computer in real time. Use the OpenCV library to identify the coordinates of the identification code in the color image. When the first client identifies the identification code, use its fixed coordinate system as the world coordinate system for multi-client synchronization. When subsequent clients identify the identification code, determine the relative position between the client and the fixed identification code based on the position of the origin of the moving coordinate system in the fixed coordinate system and the identification result, and obtain the offset of the client relative to the world coordinate system, thereby realizing the fixed coordinate system synchronization of the client.

[0070] Step 6.3) The sensing device is fixed with a multi-feature identification code. After the client camera recognizes the identification code, it obtains the spatial coordinates of the sensing device relative to the client. Then, combined with the spatial pose of the wing axis obtained in Step 5), it is transformed to realize the 3D registration of the wing axis in the world scene.

[0071] Step 7), Wing Axis Synchronization Method and Access Lock: First, render the virtual model of the wing axis to obtain its initial pose; second, collect the changes in the wing axis in space in real time, and synchronize them to other terminals via a data synchronizer; set an access lock to assign unique execution permissions, and call the wing axis synchronization method when the current terminal has the necessary execution permissions to avoid scene chaos; the specific steps are as follows:

[0072] Step 7.1) Render the scene and virtual model using the HoloLens terminal to obtain the initial pose of the wing axis model. Use the MRTK Mixed Reality Toolkit to build a spatial interaction module and UI interface. After the client operates on the wing axis virtual model, the solver tracks the movement of the wing axis, obtains the rotation and translation matrices in real time, and synchronously transmits the transformation matrix to the server. Specifically, the operation is sent to the server through the function corresponding to the [Command] identifier. The server activates the callback function on other clients to complete the synchronization operation and synchronizes the transformation matrix to the client as the offset of the local fixed coordinate system. When rendering the wing axis on the local end, the offset is added to all wing axis positions before rendering to achieve spatial consistency of the wing axis across multiple clients.

[0073] Step 7.2) Set a permission lock to assign unique execution permissions, synchronize permission control information synchronization direction, and only after obtaining permissions can it be broadcast to other clients through the server side, and relinquish ownership after synchronization is completed;

[0074] In the Unity Mirror framework, control of the wing axis can only belong to one terminal at a time. The judgment function calls the wing axis synchronization method when it determines that the current terminal has execution permission, so as to avoid scene chaos caused by multiple clients operating at the same time.

[0075] Step 8), Multi-person Collaborative Guided Docking Visualization: First, a standard wing shaft docking process database is constructed to store and retrieve docking information, including the wing shaft 3D model, docking steps, and wing shaft guided docking. Second, based on the wing shaft pose information obtained in Step 5), rendering processing is performed on the server side to provide guided docking information. Finally, the guidance information is distributed to multiple terminals for collaborative guided docking, wing shaft pose is estimated in real time, and guidance information is updated until the docking task is completed. The specific steps are as follows:

[0076] Step 8.1) Construct a standard docking process database to enable the storage and retrieval of docking information, including wing shaft 3D models, docking steps, and wing shaft guided docking.

[0077] Step 8.2) Based on Step 5), the pose information of the wing axis in the docking scene is obtained and rendered on the server side. Based on Step 6), the fixed coordinate system synchronization of multiple terminals is realized based on the fixed identification code and multi-feature identification code in the scene, the three-dimensional coordinate information of the sensing device is obtained, and the pose information of the wing axis in the world coordinate system is obtained.

[0078] Step 8.3) Based on the wing shaft pose information, a three-dimensional bounding box and docking guidance information are generated at the target location, displaying docking direction, docking steps, and other information, specifically including:

[0079] ① The steps for connecting the wing shaft and the objects involved are shown in the interface near the target location. The holes, interfaces, shafts and other related objects are highlighted and distinguished.

[0080] ② Wing shaft guidance and docking information prompts: Add a virtual arrow at the center of the docking object to indicate the general direction of docking, and add small arrows, curves and other prompts and guidance information around it to make adjustments and corrections during the docking process;

[0081] Step 8.4) During the wing shaft docking process, the wing shaft attitude is estimated in real time, the guidance information is updated based on the attitude change information, and the information is transmitted to each client until the docking task is completed; specifically including:

[0082] ① The accuracy of the wing axis position and orientation estimation result should be around 1cm, which can be considered as valid for guiding docking information;

[0083] ② The total time for estimating the wing axis attitude and updating the guidance information in real time, and finally distributing the information to the terminal, does not exceed 1 second;

[0084] ③ During the wing shaft docking process, if the docking is incorrect, the outline of the incorrect part and text prompts will be displayed, and then the correct docking method will be guided.

[0085] ④ If the error between the center of the wing shaft and the theoretical position does not exceed 3cm and the error of the axis does not exceed 2°, the docking is considered complete.

[0086] In summary, this invention discloses a multi-person collaborative guidance and docking method for aircraft wings based on augmented reality, including a dataset creation method based on inter-frame matrix transformation, a pose estimation method based on instance segmentation, and an auxiliary guidance method for multi-person collaboration. Specifically, it comprises eight steps: inter-frame transformation matrix calculation and 3D reconstruction, creation of a high-precision pose estimation dataset, construction of a deep learning framework, wing axis instance segmentation based on the FCN model, wing axis pose estimation method based on the DenseFusion network, multi-terminal world scene construction, wing axis synchronization method and access control, and a visualization method for multi-person collaborative guidance and docking. This invention applies visual and image methods to the field of aircraft wing axis pose estimation and proposes a method integrating wing axis pose estimation, 3D registration, multi-person collaboration, and visual guidance. First, deep learning and image processing methods can be used to estimate wing axis pose, improving the efficiency and accuracy of pose estimation. Second, this method has scalability and scene portability, making it relatively easy to expand the types of components. Third, this method has a high level of visualization, which can improve the docking efficiency of the production line and enhance the intelligence level of the production line.

[0087] like Figure 6 As shown, Figure 6 This paper describes a method for segmenting and estimating the wing axis using a custom dataset based on the FCN and DenseFusion deep learning networks. The goal is to segment the region containing the wing axis from the scene. Figure 6 As shown in the blue part of (a), its three-dimensional spatial pose is calculated, and the visualization effect is as follows. Figure 6 As shown in the bounding box in (b), the average error distance of the pose estimation method does not exceed 10% of the minimum circumscribed cylinder diameter of the wing shaft.

[0088] like Figure 7 As shown, Figure 7This is a demonstration diagram of a multi-person collaborative assembly guidance system for wing shaft docking scenarios. The orange box indicates the real-time estimated wing shaft pose, and the green arrow's tail points from the center of the wing shaft pose to the center of the theoretical docking position. The arrow direction is updated in real time according to the wing shaft pose changes, and the guidance information is distributed to multiple terminals for sharing until the docking task is completed, so as to realize multi-person collaborative guidance docking of aircraft wings.

Claims

1. An enhanced reality based multi-person cooperative guidance docking method for an aircraft wing, characterized in that, Includes the following steps: Step 1), Calculation of inter-frame transformation matrix and 3D reconstruction: Use a depth camera to acquire point cloud data of multiple frames from different perspectives of the wing axis. Calculate all inter-frame transformation matrices of the wing axis point cloud data through coarse and fine registration. Transform all frames of point cloud data from different perspectives to the point cloud coordinate system of the initial frame using the above transformation matrices. Stitch together the point cloud data from different perspectives of the wing axis to obtain complete wing axis point cloud data and the pose of the initial frame. Step 2), High-precision pose estimation dataset creation: Filter the stitched wing axis point cloud data from Step 1), and align the wing axis theoretical model with the filtered stitched wing axis point cloud pose through point cloud registration; Based on the transformation matrix of the point cloud data between each frame in Step 1), sequentially transform the pose of the high-precision point cloud of the initial frame to obtain the bounding boxes of the high-precision wing axis theoretical model point cloud and the computer wing axis model point cloud in all frames, thus obtaining the high-precision pose estimation dataset between each frame of the wing axis; Step 3), Network Training: Combining the multi-view pose estimation dataset built in Step 1) and Step 2), extract depth maps, RGB images, mask images, labels, and transformation matrix data, build a deep learning framework, and train a fully convolutional network (FCN) for instance segmentation and a DenseFusion network for wing axis pose estimation. Step 4), Wing Axis Instance Segmentation Based on FCN Model: During the wing docking process, for the original color image and the original depth image obtained from the depth camera, the target region in the original image is obtained based on the Fully Convolutional Network (FCN), thereby realizing image segmentation of the region where the wing axis is located in the original image; Step 5), Wing Axis Pose Estimation Method Based on DenseFusion Network: Based on the image segmentation in Step 4), the results of instance segmentation are obtained as index data. Through data preprocessing, backbone network inference and iterative self-optimization process, the spatial pose data of the wing axis relative to the observer is obtained using the DenseFusion network. Step 6), Multi-terminal world scene construction: First, build a multi-terminal world scene framework based on Unity Mirror, specifying the server and client; second, realize the synchronization of the coordinate system of multiple terminals based on fixed identification codes; finally, obtain the coordinate information of the sensing device according to the multi-feature identification code, and combine it with the wing axis pose obtained in step 5) to realize the 3D registration of the wing axis in the world scene. Step 7), Wing Axis Synchronization Method and Access Lock: First, render the virtual model of the wing axis to obtain the initial pose of the wing axis; second, collect the changes of the wing axis in space in real time and synchronize them to other terminals by the data synchronizer. A permission lock is set to assign unique execution permissions, and the wing axis synchronization method is called when the current terminal determines that it has execution permissions. Step 8), Multi-person collaborative guided docking visualization: First, construct a standard wing shaft docking process database to enable the storage and retrieval of docking information, including wing shaft 3D models, docking steps, and wing shaft guided docking; Second, based on the wing shaft pose information obtained in Step 5), perform rendering processing on the server side to provide wing shaft guided docking information. Finally, the guidance information is distributed to multiple terminals for collaborative guidance and docking, the wing axis position and attitude are estimated in real time, the guidance information is updated, and the docking task is completed.

2. The method of claim 1, wherein, Step 1) Specifically: Step 1.1) The depth camera acquires point cloud data from multiple frames of different perspectives on the wing axis. Before acquisition, Aruco codes are placed around the wing axis as initial positioning references, and it is ensured that there are 3 QR codes in the field of view during recording. Step 1.2), using the positions of the Aruco codes between frames, calculate the camera pose relative to the Aruco codes in each group of images as follows: (1) In the formula, A Indicates the position of the Aruco code in the world coordinate system; R n Indicates the first n The rotation matrix of the Aruco code relative to the camera in the dataset; A n Indicates the first n The position of the camera in the world coordinate system within the dataset; t n Indicates the first n The translation matrix of the Aruco code relative to the camera in the dataset; R 0、 A 0 and t 0 represents the matrix corresponding to the initial group of data; the pose relationship between adjacent frames is: (2) Calculate the pose coarse transformation matrix of the wing axis point cloud between each frame, which is the rotation transformation matrix. and translation transformation matrix ; Step 1.3): Based on the coarse registration transformation matrix obtained in Step 1.2), the point cloud of the previous frame is used as the target point cloud, and the point cloud of the next frame is used as the source point cloud for pose transformation to complete the coarse registration. The nearest point iteration (ICP) method is used as the means for fine registration. The two point clouds after coarse registration are used as input to solve for the pose transformation matrix of fine registration. The formula is as follows: (3) In the formula, Represents the optimal rotation matrix. Represents the optimal translation matrix, and N represents the number of point clouds. Representing the source cloud, Represents the target point cloud, Represents a rotation matrix. Represents the translation matrix; Step 1.4) Based on the coarse and fine registration pose transformation matrices of the wing axis point clouds between frames obtained in Step 1.2) and Step 1.3), transform all the wing point clouds of all frames to the initial frame to complete the stitching of point clouds from different perspectives of the wing axis, and realize the three-dimensional reconstruction of the wing axis point cloud in the initial frame.

3. The method according to claim 1, characterized in that, Step 2) specifically involves: Step 2.1) Filter the initial frame wing axis point cloud reconstructed in Step 1) by selecting pass-through filtering, guided image filtering and radius filtering to process the stitched point cloud information; Step 2.2) Calculate the spatial pose transformation matrix for the point cloud of the wing axis theoretical model and the point cloud reconstructed by the depth camera. The specific calculation steps are as follows: ① FPFH feature descriptors for point clouds based on computer wing axis theory model and point clouds reconstructed by depth camera, using the PCL open-source function library; ② Based on the Sample Consistency Initial Registration (SAC-IA) method in the PCL open-source function library, the FPFH feature descriptor in step ① is used as the basis for matching. The pose transformation matrix is ​​calculated using the point cloud of the wing axis theoretical model as the source point cloud and the point cloud reconstructed by the depth camera as the target point cloud to achieve coarse registration of the two point clouds. ③ The pose transformation matrix of the two wing shaft point cloud data after coarse registration in step ② is calculated using the ICP method to achieve fine registration of the two point clouds; ④ Use the polygon_obb method in the Trimesh module to perform polygon convex hull detection, and enclose the entire point cloud in sequence to generate the outer bounding box of the wing axis point cloud model; Step 2.3), combined with the inter-frame pose transformation matrix obtained in Step 1), the high-precision model of the wing axis in the initial frame with the outer bounding box is transformed by the inverse transformation of the inter-frame pose transformation matrix to obtain the high-precision theoretical model point cloud of the wing axis in all frames, thus obtaining the dataset of high-precision pose estimation of the wing axis in each frame and at each viewpoint.

4. The method according to claim 1, characterized in that: Step 3) specifically refers to: Step 3.1) The training process for the FCN network includes: constructing an instance segmentation dataset based on the datasets obtained in Steps 1) and 2), using single-channel P-mode images from the PIL library as annotation maps for instance segmentation. Specific steps include: First, convert the color image data into grayscale image data: (4) In the formula, y represents the converted grayscale value; R, G, and B represent the pixel values ​​of the three channels in the color image; The formula for converting a grayscale image to a P-image is as follows: (5) In the formula, p represents the pixel value in the P-mode image; a color palette containing multiple colors is predefined, and m represents the number of colors in the color palette; the pixel values ​​in the image are changed according to the type of mask image obtained in steps 1) and 2), the mask boundaries are defined, and they are combined into a P-mode image through the PIL library to obtain a dataset for instance segmentation. After entering the training phase, the loss during the training process is monitored, and the training ends when the network loss reaches less than 1mm. Step 3.2) The training process for the DenseFusion network includes: performing computation based on the PyTorch framework, selecting the multi-view pose estimation RGBD dataset from Step 1) and Step 2) as the original data, and dividing it into training and validation sets. First, train the backbone network, set the pose estimation threshold, and switch to an iterative self-optimizing network when the loss value is lower than the threshold. The optimal model when the ADD value reaches the lowest value in the above process is retained. The ADD value is the average distance between the predicted pose and the true value.

5. The method according to claim 1, characterized in that: Step 4) specifically involves: processing the input point cloud and RGB image data through a 5-layer convolutional and 5-layer pooling network structure to obtain the instance segmentation feature map in the original data. Simultaneously, through 8x upper pooling and deconvolution operations, feature maps from different layers are fused.

6. The method according to claim 1, characterized in that: Step 5) specifically involves: Step 5.1) For the input RGBD data, the depth map is converted into a 3D point cloud. The imaging plane is normalized using the point cloud and the aligned depth and RGB image information. The feature information of multiple points is selected as the input of the pose estimation backbone network. Step 5.2) Downsample the input color features using ResNet18 and upsample using 4 PSPNet pyramid structures to extract color feature information; Step 5.3) Convert the input depth features into point cloud information, use PointNet to extract geometric features, and output geometric features with the same dimension as the color information; Step 5.4) Input color features and geometric features into the network, perform multiple convolutions and activation function operations, and concatenate to obtain pixel information fused from color and geometric features; perform global pooling to fuse pixel, local, and global information of each pixel into a feature vector, calculate pixel rotation, translation, and confidence parameters, and obtain the pose parameters of the pixel. Step 5.5) Based on the rotation and translation parameters output by the backbone network, the pose estimation effect is optimized by iteratively optimizing the PoseRefineNet network.

7. The method according to claim 1, characterized in that: Step 6) specifically refers to: Step 6.1) Use the Unity Mirror framework to build a multi-terminal world scene framework, establish a network manager based on the Network Manager component to manage the data stream transmission of multiple terminals; set the desktop computer as the server to comprehensively process scene information, client generation and scene information synchronization; the other terminals are all clients, and render the current client scene according to the information sent by the server when they join the server. Step 6.2) Place and fix the identification code at a certain position, and then transmit the color image from the client camera to the Python program on the desktop computer in real time. Use the OpenCV library to identify the coordinates of the identification code in the color image. When the first client identifies the identification code, use its fixed coordinate system as the world coordinate system for multi-client synchronization. When subsequent clients identify the identification code, determine the relative position between the client and the fixed identification code based on the position of the origin of the moving coordinate system in the fixed coordinate system and the identification result, and obtain the offset of the client relative to the world coordinate system, thereby realizing the synchronization of the fixed coordinate system of each client. Step 6.3) A multi-feature identification code is fixed on the sensing device. After the client camera recognizes the identification code, it obtains the spatial coordinates of the sensing device relative to the client. Then, combined with the spatial pose of the wing axis obtained in Step 5), it is transformed to realize the 3D registration of the wing axis in the world scene.

8. The method according to claim 1, characterized in that, Step 7) specifically refers to: Step 7.1) Render the scene and model using the HoloLens terminal to obtain the initial pose of the wing axis model; Using the MRTK Mixed Reality Toolkit, a spatial interaction module and UI interface are built. After the client operates on the wing axis model, the solver tracks the movement of the wing axis, obtains the rotation and translation matrices in real time, and synchronously transforms the matrices to the server. The specific operation involves sending the corresponding function to the server via the [Command] identifier. The server then activates the callback function on the other clients to complete the synchronization operation. The transformation matrix is ​​synchronized to the client as the offset of the local fixed coordinate system. When rendering the wing axis on the local side, the offset is added to all wing axis positions before rendering, thus achieving spatial consistency of wing axis rendering across multiple clients. Step 7.2) Set a permission lock to assign unique execution permissions, synchronize the direction of permission control information synchronization, broadcast the permissions to other clients through the server after obtaining permissions, and relinquish ownership after synchronization is complete; In the Unity Mirror framework, the wing axis synchronization method is called when the current terminal determines that it has execution permission.

9. The method according to claim 1, characterized in that, Step 8) specifically refers to: Step 8.1) Construct a standard wing shaft docking process database to enable the storage and retrieval of docking information, including wing shaft 3D models, docking steps, and wing shaft guided docking. Step 8.2) Based on Step 5), the pose information of the wing axis in the docking scene is obtained and rendered on the server side. Based on Step 6), the fixed coordinate system synchronization of multiple terminals is realized based on the fixed identification code and multi-feature identification code in the scene, the three-dimensional coordinate information of the sensing device is obtained, and the pose information of the wing axis in the world coordinate system is obtained. Step 8.3) Based on the wing shaft's pose information, a three-dimensional bounding box is formed at the wing shaft location, and the guided docking, docking direction, and docking steps are displayed, specifically including: ① The steps for connecting the wing shaft and the objects involved are shown in the interface near the target location. The holes, interfaces and shafts involved are highlighted and distinguished. ② Wing shaft guidance and docking information prompts: A virtual arrow is added to the center of the wing shaft to indicate the general direction of guidance and docking, and small arrows and curves are added around it to provide various prompts and guidance information for adjustment and correction during the docking process; Step 8.4) During the docking process, the wing axis pose is estimated in real time, the guidance information is updated based on the pose change information, and the information is transmitted to each client until the assembly task is completed; specifically including: ① The accuracy of the wing axis pose estimation result is 1cm, indicating that the guidance docking information is valid; ② The total time for estimating the wing axis attitude and updating the guidance information in real time, and finally distributing the information to the terminal, does not exceed 1 second; ③ During the docking process, if an error occurs, the outline of the incorrect part will be displayed along with text prompts, and then the correct docking method will be guided. ④ If the error between the center of the wing shaft and the theoretical position does not exceed 3cm and the error of the axis does not exceed 2°, the docking is considered complete.

Images