An augmented reality assembly aid for a blind spot in a field of view of an aircraft equipment bay

Abstract

An aircraft equipment cabin visual field blind area augmented reality auxiliary assembly method comprises a depth camera self-positioning based on point cloud registration, an assembly object pose estimation based on deep learning, an assembly object pose estimation based on an edge region, a pose estimation method adaptive switching and an auxiliary assembly guide visualization; the adaptive switching method is used to fuse the pose estimation method based on deep learning and the pose estimation method of traditional image processing, so that the robustness, real-time performance and accuracy of the pose estimation are balanced; meanwhile, based on the pose estimation result, the visual field blind area of the assembly process can be visualized by using the augmented reality technology, and the assembly result is detected, which is beneficial to the user to quickly locate the assembly object and complete the assembly task, so as to improve the consistency, real-time performance and immersion of the aircraft assembly operation guide, and further improve the artificial assembly efficiency and quality.

Description

Technical Field

[0001] This invention relates to the field of aircraft assembly technology, and more specifically to an augmented reality-assisted assembly method for blind spots in aircraft equipment bays. Background Technology

[0002] Advanced aviation equipment such as aircraft are characterized by complex product structures, a huge number of parts, and numerous coordination relationships, resulting in challenging assembly tasks, highly complex operations, and lengthy workflows. Due to the flexibility and adaptability requirements of the assembly process, a large amount of assembly work is still dominated by manual assembly. The characteristics of aircraft assembly operations require operators to be familiar with and memorize complex operating instructions, and to consult a large number of process manuals and production operation documents. This leads to weak process guidance, is time-consuming and labor-intensive, and is prone to errors in interpreting drawings, seriously restricting assembly efficiency and quality. This problem is particularly prominent in assembly operations in the blind spots of the aircraft equipment compartment.

[0003] To assist assembly workers in improving the efficiency and quality of advanced aviation equipment assembly, methods such as 3D assembly instructions and visualization of guidance information, as well as augmented reality technology, are gradually being applied to aircraft assembly sites. These methods provide assembly workers with a visual and interactive description of the work process. For example, Chinese patent CN202110337423.2, entitled "An Assembly Guidance Method, System and Application Based on Augmented Reality," works by identifying parts through the contour information of part models and marking them in an envelope manner to guide assembly. However, its drawback is that it cannot accurately locate parts in slightly complex real-world scenarios by relying on position and contour information, thus affecting the virtual-real fusion effect. Furthermore, it cannot be used in blind spots of the aircraft equipment bay by judging the assembly status solely based on position and contour information. To achieve augmented reality visualization that blends the virtual and real worlds, the pose information of the assembly object is needed to overlay virtual information onto the physical object. For example, Chinese patent CN202110892450.6, entitled "An Assembly Guidance Method and System Based on HoloLens Depth Data," works by using point cloud data combined with an improved Votenet network and ICP calibration to obtain the part pose and guide assembly. However, this method, based on the Votenet network, is slow and suffers from lag in pose updates. Therefore, an augmented reality-assisted assembly method is needed for blind spots in aircraft equipment bays to address the problems of inaccurate and non-real-time pose estimation of assembly objects, and the difficulty in automatically detecting assembly results and guiding assembly in blind spot scenarios within aircraft equipment bays. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, the present invention aims to provide an augmented reality-assisted assembly method for blind spots in aircraft equipment bays. This method uses an adaptive switching approach to integrate deep learning-based pose estimation and traditional image processing pose estimation methods, balancing the robustness, real-time performance, and accuracy of pose estimation. Simultaneously, based on the pose estimation results, augmented reality technology can be used to visualize the blind spots in the assembly process and detect the assembly results. This facilitates users in quickly locating assembly objects and completing assembly tasks, thereby improving the consistency, real-time performance, and immersive experience of aircraft assembly operation guidance, further enhancing the efficiency and quality of manual assembly.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] An augmented reality-assisted assembly method for blind spots in aircraft equipment bays includes the following steps:

[0007] Step 1), Depth camera self-localization based on point cloud registration: First, install a depth camera in a designated area of ​​the assembly scene; second, register the point cloud data of the scene model with the point cloud data collected by the depth camera; finally, calculate the pose information of the depth camera in the assembly scene based on the point cloud registration result.

[0008] Step 2), Deep learning-based pose estimation of assembly objects (assemblies include avionics equipment, pipe joints, and bolts): First, collect the pose dataset of the assembly objects and enrich the dataset using data augmentation methods such as color space enhancement and six-degree-of-freedom transformation enhancement; second, build a deep learning network, using a model pre-trained on a public dataset as the initial model; third, train and test the initial model on the pose dataset of the assembly objects, and fine-tune it to obtain a transfer model; finally, use the transfer model online to output the pose of the assembly objects from the collected RGB images through the deep learning network.

[0009] Step 3), Estimation of the pose of the assembly object based on the edge region: First, render the spatial viewpoint model of the CAD model of the assembly object to obtain the edge contour points and normal vectors of the spatial viewpoint model; second, statistically analyze the segmentation probability of the corresponding line segments in the direction of the normal vector of the edge contour points of the assembly object in the acquired RGBD image as belonging to the foreground or background in an online manner to complete the uncertainty modeling; finally, optimize the uncertainty model and calculate the corresponding contour and pose of the assembly object that best fits the image segmentation.

[0010] Step 4), Adaptive switching of pose estimation method: Use the method described in step 2) to perform online pose estimation on the assembly object to obtain the initial pose, and use the method described in step 3) to iteratively perform subsequent pose estimation based on the initial pose; use an evaluation function to judge the stability of the pose estimation of the current assembly object, and call the method described in step 2) once when the calculated value of the evaluation function is greater than the set threshold to correct and stabilize the pose estimation result;

[0011] Step 5), Visualized Assembly Guidance: First, an assembly process database is constructed to store and retrieve assembly information, including 3D models of assembly parts, assembly tools, assembly steps, process indicators, and assembly guidance information. Second, based on the pose information of the depth camera in the assembly scene obtained in Step 1), combined with the pose estimation results of the assembly object obtained in Step 4), the pose information of the assembly object in the assembly scene is obtained, and the assembly information, such as the 3D models of assembly parts, is rendered on the display device. Finally, the pose information of the assembly object is used to determine whether the current assembly step is completed, thereby guiding the next assembly step until the assembly task is completed.

[0012] The beneficial effects of this invention are as follows:

[0013] (1) This invention uses augmented reality technology to visualize blind spots in the assembly process, which helps users quickly locate assembly objects and complete assembly tasks.

[0014] (2) This invention proposes an adaptive switching method that adaptively switches between edge region-based and deep learning-based assembly object pose estimation methods to balance the robustness, real-time performance and accuracy of assembly object localization, thereby improving the versatility and portability of the method of this invention.

[0015] (3) The present invention can automatically detect assembly results and use a variety of auxiliary assembly guidance visualization methods to ensure the accuracy and diversity of assembly guidance information and improve assembly efficiency and quality. Attached Figure Description

[0016] Figure 1 This is an overall flowchart of the method of the present invention.

[0017] Figure 2 This is a flowchart of the assembly object pose estimation method based on edge regions according to the present invention.

[0018] Figure 3 This is the input and output diagram of the pose estimation method of the present invention. Detailed Implementation

[0019] The technical solution of the present invention will be further described in detail below with reference to the embodiments and accompanying drawings.

[0020] like Figure 1As shown, an augmented reality-assisted assembly method for blind spots in aircraft equipment bays includes the following steps:

[0021] Step 1), Depth camera self-localization based on point cloud registration: First, install a depth camera in a designated area of ​​the assembly scene; second, register the point cloud data of the scene model with the point cloud data acquired by the depth camera; finally, calculate the pose information of the depth camera in the assembly scene based on the point cloud registration result; specifically:

[0022] Step 1.1) Install a depth camera in a designated area of ​​the assembly scene to acquire more RGBD images of the operator's blind spots. Select the center position of this area and the orientation of the depth camera as the initial pose of the depth camera. Use the PCA algorithm to coarsely register the point cloud data of the scene model with the point cloud data acquired by the depth camera, and then use the GICP algorithm to complete the fine registration. The pose matrix M of the depth camera in the assembly scene is... C for:

[0023]

[0024] In the formula: M is the transformation matrix obtained by fine registration. C' This is the initial pose matrix of the depth camera;

[0025] Step 2), assembly object pose estimation based on deep learning (assemblies include avionics equipment, pipe joints, and bolts): First, collect the assembly object pose dataset and enrich it using data augmentation methods such as color space enhancement and six-degree-of-freedom transformation enhancement; second, build a deep learning network, using a model pre-trained on a publicly available dataset as the initial model; third, train and test the initial model on the assembly object pose dataset, and fine-tune it to obtain a transfer model; finally, use the transfer model online to output the assembly object pose from the collected RGB images through the deep learning network; specifically:

[0026] Step 2.1) Use the ObjectDatasetTools open-source tool to construct an assembly object pose dataset for the aircraft equipment bay scene. Collect RGBD images of the assembly objects under different angles, distances, and lighting conditions and annotate them. The annotation information includes the assembly object name and pose information. The pose information is annotated by manually registering and annotating the pose information of the assembly objects in the first frame of the RGBD image. The pose information of the assembly objects in other frames is calculated based on the pose of the QR code next to the assembly object, realizing automatic inference and annotation of pose information. Based on the assembly object pose dataset, expand the assembly object pose dataset by using data augmentation methods such as color space enhancement and six-degree-of-freedom transformation enhancement.

[0027] Step 2.2) Build the EfficientPose deep learning network. Pre-train the initial model based on the Linemod public dataset. Use the initial model to train and test on the assembly object pose dataset. Fine-tune the weight parameters of the initial model to obtain the transfer model. The transfer model is used to output the predicted assembly object pose information of the online acquired RGB images in the EfficientPose deep learning network.

[0028] Step 3), based on the pose estimation of the assembled object in the edge region, such as... Figure 2 As shown, firstly, the spatial viewpoint model of the assembly object's CAD model is rendered to obtain the edge contour points and normal vectors of the spatial viewpoint model; secondly, the segmentation probability of the corresponding line segments along the normal vector direction of the assembly object's edge contour points in the acquired RGBD image belonging to the foreground or background is statistically analyzed online to complete uncertainty modeling; finally, the uncertainty model is optimized to calculate the corresponding contour and assembly object pose that best fit the image segmentation; specifically:

[0029] Step 3.1): To obtain a dense and uniform spatial viewpoint model, the triangles on each face of the icosahedron are divided into four sub-triangles on an average basis. This process is repeated four times to obtain 256 equilateral triangles on each face, for a total of 5120 equilateral triangles. The virtual camera sampling viewpoint is located at the vertex of the triangle, 1 meter away from the center of the icosahedron (this distance depends on the size of the assembly object). The optical axis points to the center of the icosahedron. The center of the CAD model of the assembly object is set to the center of the icosahedron. The spatial viewpoint model of the CAD model is rendered at 2562 sampling viewpoints. 200 edge contour points and their normal vectors of the CAD model in the spatial viewpoint model are collected.

[0030] Step 3.2): Select the CAD model corresponding to the pose estimation result of the previous frame projected onto the imaging plane as a mask. Using the edge contour points of the CAD model sampled from the mask contour as the midpoint, extend equal distances to both sides along their respective normal vector directions (this distance ranges from 5 to 30, decreasing with the number of iterations). The resulting line segments are the corresponding lines. Eliminate the corresponding lines where the difference between the CAD model depth value of the edge contour point and the actual depth value at that point (taking the minimum depth value of 8 pixels around the edge contour point in the RGBD image) is greater than 3 cm (at this point, it is considered that occlusion has occurred). At the same time, eliminate... Except for sampling points where the absolute value of the deviation between the actual depth value of each pixel within the mask and the depth value of the CAD model is greater than 5 cm (at which point it is considered that occlusion has occurred or that the sampling point is the background); the corresponding line is projected onto the imaging plane to form the projection correspondence line, and the probability of each pixel on the projection correspondence line belonging to the foreground and background is calculated: based on Bayesian theory and the mask contour of the previous frame, the posterior probability of each pixel being located in the current frame mask contour and the posterior probability of the depth information satisfying the current frame mask are calculated. The joint probability of the two posterior probabilities is maximized using Newton's method and Tikhonov regularization method to predict the pose change vector θ. Iterative optimization is used to update the pose estimation results:

[0031]

[0032] in,

[0033]

[0034]

[0035] In the formula, g is the gradient vector (the two addends consider the corresponding line and the sampling point respectively), H is the 6×6 Hessian matrix (the two addends consider the corresponding line and the sampling point respectively), and λ r and λ t The rotation and translation regularization parameters (corresponding to the prior probabilities) control the confidence level of the pose estimation results from the previous frame, thereby stabilizing the optimization and preventing it from progressing in the wrong direction. Let λ be the value of these parameters. r =1000, λ t =20000), I3 is the identity matrix; n is the number of corresponding lines remaining after removal, d i Let l be the distance from the edge contour point of the i-th corresponding line to the midpoint of the corresponding line in the previous frame. i For the i-th corresponding line, L i This refers to the domain formed by the corresponding lines. Let be the i-th CAD model point in the camera coordinate system, and n' be the number of sampling points remaining after removal. and d represents the CAD model points, normal vectors, and the nearest sampled point obtained by the depth camera in the model coordinate system. ziσ is the depth value of the most recent sampling point. d The standard deviation of depth information (assuming that the quality deteriorates as the sampling point is farther away, thus controlling the weight of the sampling points, is σ). d =20);

[0036] The formula for calculating the relevant derivative is as follows:

[0037] In the formula, μ i and σ i Let be the mean and standard deviation of the posterior probability distribution of the corresponding line, which follow a normal distribution. Let be the rotation matrix from the camera coordinate system to the model coordinate system. This is the antisymmetric matrix form of CAD model points in the model coordinate system, such as for point X = [xyz]. T ,have |n xi |and|n yi | represents the length of the corresponding line projected onto the imaging plane in the image coordinate system along the horizontal and vertical axes, f. x and f y This refers to the camera's focal length parameter;

[0038] Step 3.3) Predict the pose change vector after iterative optimization. Update pose estimation results:

[0039]

[0040] In the formula, This is the transformation matrix from the model coordinate system to the camera coordinate system. and for The rotational and translational components;

[0041] Step 4), Adaptive switching of pose estimation method: The method described in Step 2) is used to perform online pose estimation on the assembly object to obtain an initial pose. Based on the initial pose, the method described in Step 3) is used iteratively to perform subsequent pose estimations. An evaluation function is used to determine the stability of the current assembly object's pose estimation. When the calculated value of the evaluation function is greater than a set threshold, the method described in Step 2) is called once to correct and stabilize the pose estimation result. Specifically:

[0042] Step 4.1): In the initial stage, the assembly objects in the aircraft equipment bay scene are online pose estimated according to the method described in step 2) to obtain the initial pose. The initial pose is used as the first frame. The pose estimation of the assembly objects in subsequent image frames is completed iteratively according to the method described in step 3).

[0043] Step 4.2), use the F norm of equation (4) as the evaluation function F;

[0044] F = ||H|| F +α n (6)

[0045] In the formula, α n For noise parameters, ||H|| F Let H be the F-norm;

[0046] The pose estimation results in the iterative cycle are evaluated. When F is greater than the set threshold (the threshold for avionics is 4, and α is taken for avionics A, B, and C in ascending order of size), the result is evaluated. n If the results are 2.4, 2.8, or 2.9, and the current pose estimation result is considered to have low reliability, then the method described in step 3) is inserted once to perform pose estimation, thereby correcting the erroneous pose estimation result and improving the robustness of pose estimation.

[0047] Step 5), Visualized Assembly Guidance: First, an assembly process database is constructed to store and retrieve assembly information, including 3D models of assembly components, assembly tools, assembly steps, process parameters, and assembly guidance information. Second, based on the pose information of the depth camera in the assembly scene obtained in Step 1), combined with the pose estimation results of the assembly object obtained in Step 4), the pose information of the assembly object in the assembly scene is obtained, and the assembly information, such as the 3D models of assembly components, is rendered on the display device. Finally, the pose information of the assembly object is used to determine whether the current assembly step is complete, thereby guiding the next assembly step until the assembly task is completed. Specifically:

[0048] Step 5.1) Construct an assembly process database to enable the storage and retrieval of assembly information, including 3D models of assembly parts, assembly tools, assembly steps, process parameters, and assembly guidance information.

[0049] Step 5.2) According to the method described in Step 1), locate the pose of the depth camera in the aircraft equipment cabin scene. Based on the QR code and SLAM algorithm in the scene, locate the pose of the user's display device with camera function (the display device can be AR glasses, such as HoloLens, or a flat panel display, such as iPad). Combined with the pose estimation result of the assembly object in the camera coordinate system calculated in Step 4), the pose of the assembly object in the display device coordinate system can be calculated. As a result, the assembly information is displayed at the corresponding position, specifically including: (1) The assembly objects involved in the assembly steps being executed are rendered in a semi-transparent manner at the corresponding pose position and distinguished by different colors, and the assembly parts (holes, shafts, interfaces, joints) are highlighted in orange; (2) A three-dimensional assembly simulation animation (including assembly process, tools, actions) is played at the corresponding position of the assembly object, and the assembly guidance information such as arrows, curves, circles, exclamation marks, and voice is used to provide prompts; (3) The process indicators (such as torque, number of winding turns, measurement data) that are easy to browse are displayed in the attachment of the assembly object, and the textual descriptions of other process indicators and assembly steps are displayed in the upper right corner of the device.

[0050] Step 5.3): In each assembly step, the pose estimation result of the assembly object is used to determine whether the assembly is complete and whether the assembly result is correct, and the next assembly step is automatically guided. Specifically, this includes: (1) If the pose estimation result of the assembly object changes by no more than 2 cm / 20 degrees within one second, the assembly object is considered to be stationary, and the assembly step is considered complete; (2) If the pose estimation result of the assembly object deviates from the correct assembly pose of the assembly object in the scene by no more than 3 cm / 10 degrees, the assembly result is considered correct; (3) If the assembly is not complete, the assembly is guided to continue; if the assembly is completed but the result is incorrect, the incorrect assembly part and the outline of the assembly object are highlighted in red, and the correct assembly position is indicated in yellow and the assembly is guided to continue; if the assembly step is completed and the result is correct, the next assembly step is automatically entered until the assembly task is completed, the assembly process log is recorded and evidence is saved.

[0051] In summary, this invention discloses an augmented reality-assisted assembly method for blind spots in aircraft equipment bays. Firstly, it proposes an edge-region-based assembly object pose estimation method to address the slow speed and low accuracy issues of deep learning-based pose estimation methods. Secondly, it proposes an adaptive switching method that combines edge-region-based and deep learning-based assembly object pose estimation methods, adaptively switching when necessary to balance the robustness, real-time performance, and accuracy of assembly object localization. Finally, the assembly object pose estimation results are used for augmented reality-assisted assembly and assembly result detection, enabling visualization and guided assembly of assembly objects in blind spots of aircraft equipment bay assembly operations, and automatically detecting assembly results, thereby improving assembly efficiency and quality.

[0052] Figure 3The input-output diagram for the pose estimation method shows the pose estimation result of the assembly object calculated by the EfficientPose deep learning network from the input image. This is used to initialize or correct the pose estimation result. Then, through iterative optimization of the assembly object pose estimation method based on edge regions (the sampling points in the diagram correspond to the 1st, 3rd, and 6th iterations of the line), the pose estimation result of the next frame is calculated. This method can reduce the interference caused by incorrect corresponding lines and sampling points in real-world scenarios, thereby improving the robustness, speed, and accuracy of the algorithm.

[0053] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0054] The embodiments described above are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An augmented reality-assisted assembly method for blind spots in aircraft equipment bays, characterized in that, Includes the following steps: Step 1), Depth camera self-localization based on point cloud registration: First, install a depth camera in a designated area of ​​the assembly scene; Secondly, the point cloud data of the scene model is registered with the point cloud data collected by the depth camera; finally, the pose information of the depth camera in the assembly scene is calculated based on the point cloud registration result. Step 2), deep learning-based pose estimation of assembly objects, including avionics equipment, pipe joints, and bolts: First, an assembly object pose dataset is collected and enriched using data augmentation methods such as color space enhancement and six-degree-of-freedom transformation enhancement; second, an EfficientPose deep learning network is built, using a model pre-trained on a publicly available dataset as the initial model; third, the initial model is trained and tested on the assembly object pose dataset, and fine-tuned to obtain a transfer model; finally, the transfer model is used online to output the assembly object pose from the collected RGB images through the deep learning network. Step 3), Assembly object pose estimation based on edge regions: First, render the spatial viewpoint model of the assembly object CAD model to obtain the edge contour points and normal vectors of the spatial viewpoint model; second, statistically analyze the segmentation probability of the corresponding line segments in the normal direction of the edge contour points of the assembly object in the acquired RGBD image as belonging to the foreground or background, and complete the uncertainty modeling; finally, use Newton's method and Tikhonov regularization method to maximize and optimize the uncertainty model, and calculate the corresponding contour and assembly object pose that best fits the image segmentation. Step 4), Adaptive switching of pose estimation method: Use the method described in Step 2) to perform online pose estimation on the assembly object to obtain the initial pose, and use the method described in Step 3) to iteratively perform subsequent pose estimation based on the initial pose; use an evaluation function to judge the stability of the pose estimation of the current assembly object, and call the method described in Step 2) once when the calculated value of the evaluation function is greater than the set threshold to correct and stabilize the pose estimation result; Step 5), Visualized Assembly Guidance: First, construct an assembly process database to enable access to assembly information, including 3D models of assembly parts, assembly tools, assembly steps, process parameters, and assembly guidance information. Secondly, based on the pose information of the depth camera in the assembly scene obtained in step 1), and combined with the pose estimation result of the assembly object obtained in step 4), the pose information of the assembly object in the assembly scene is obtained, and the assembly information of the 3D model of the assembly parts is rendered on the display device. Finally, the pose information of the assembly object is used to determine whether the assembly step is completed, and then guides the next assembly step until the assembly task is completed.

2. The method according to claim 1, characterized in that, Step 1) Specifically: A depth camera is installed in a designated area of ​​the assembly scene to acquire RGBD images of more blind spots in the operator's field of vision. The center position of this area and the orientation of the depth camera are selected as the initial pose of the depth camera. The point cloud data of the scene model and the point cloud data acquired by the depth camera are coarsely registered using the PCA algorithm, and then finely registered using the GICP algorithm. The pose matrix of the depth camera in the assembly scene is then determined. for: （1） In the formula: The transformation matrix obtained by fine registration, This is the initial pose matrix of the depth camera.

3. The method according to claim 2, characterized in that, Step 2) specifically involves: Step 2.1) Use the ObjectDatasetTools open-source tool to build an assembly object pose dataset in the aircraft equipment bay scene. Collect RGBD images of the assembly objects under different angles, distances, and lighting conditions and annotate them. The annotation information includes the assembly object name and pose information. The pose information is annotated by manually registering and annotating the pose information of the assembly objects in the first frame of the RGBD image. The pose information of the assembly objects in other frames is calculated based on the pose of the QR code next to the assembly object, so as to realize the automatic inference and annotation of pose information. Based on the assembly object pose dataset, the data augmentation method of color space enhancement and six-degree-of-freedom transformation enhancement is used to expand the assembly object pose dataset; Step 2.2) Build the EfficientPose deep learning network. Pre-train the initial model based on the Linemod public dataset. Use the initial model to train and test on the assembly object pose dataset. Fine-tune the weight parameters of the initial model to obtain the transfer model. The transfer model is used to output the predicted assembly object pose information of the online acquired RGB images in the EfficientPose deep learning network.

4. The method according to claim 3, characterized in that, Step 3) specifically involves: Step 3.1) To obtain a dense and uniform spatial viewpoint model, the triangles on each face of the icosahedron are divided into four sub-triangles on an average basis. This process is repeated four times to obtain 256 equilateral triangles on each face, for a total of 5120 equilateral triangles. The virtual camera sampling viewpoint is located at the vertex of the triangle, 1 meter away from the center of the icosahedron, with the optical axis pointing towards the center of the icosahedron. The center of the CAD model of the assembly object is set to the center of the icosahedron. The spatial viewpoint model of the CAD model is rendered at 2562 sampling viewpoints. 200 edge contour points and their normal vectors of the CAD model in the spatial viewpoint model are collected. Step 3.2) Select the graphic of the CAD model projected onto the imaging plane corresponding to the pose estimation result of the previous frame as the mask. Take the edge contour point of the CAD model sampled on the mask contour as the midpoint, and extend it to both sides by the same distance along the direction of its respective normal vector. The distance is 5 to 30, and decreases as the number of iterations increases. The line segment formed in this way is the corresponding line. Remove the corresponding line where the difference between the depth value of the edge contour point CAD model and the actual depth value at that point is greater than 3 cm. At this time, it is considered that occlusion has occurred. The actual depth value is the minimum depth value of 8 pixels around the edge contour point in the RGBD image. Simultaneously, sampling points where the absolute value of the deviation between the actual depth value of each pixel within the mask and the depth value of the CAD model is greater than 5 cm are removed. These points are considered to be either occluded or background. The corresponding lines are projected onto the imaging plane to form projection lines. The probability of each pixel on the projection lines belonging to the foreground and background is calculated. Based on Bayesian theory and the mask contour of the previous frame, the posterior probability of each pixel being located within the current frame's mask contour and the posterior probability of the depth information satisfying the current frame's mask are calculated. The joint probability of the two posterior probabilities is maximized using Newton's method and Tikhonov regularization for the pose change vector. Predicted value Iterative optimization is used to update the pose estimation results: （2） in, （3） （4） In the formula, The gradient vector is represented by two addends that account for the corresponding line and the sampling point, respectively. The matrix is ​​a 6×6 Hessian matrix, with the two addends taking into account the corresponding lines and sampling points, respectively. and The rotation and translation regularization parameters correspond to the prior probabilities and control the confidence level of the pose estimation results from the previous frame. This stabilizes the optimization and prevents it from progressing in the wrong direction. , ; It is the identity matrix; This represents the number of corresponding lines remaining after removal. Let be the distance from the edge contour point of the i-th corresponding line to the midpoint of the corresponding line in the previous frame. For the i-th corresponding line, This refers to the domain formed by the corresponding lines. Let i be the i-th CAD model point in the camera coordinate system. This represents the number of sampling points remaining after removal. , and These are the CAD model points, normal vectors, and the nearest sampled point obtained by the depth camera in the model coordinate system. The depth value of the most recent sampling point. The standard deviation of depth information is used, and it is assumed that the quality deteriorates as the sampling point becomes farther away. Therefore, the weights of the sampling points are controlled accordingly. ; The formula for calculating the relevant derivative is as follows: , , , ; In the formula, and Let be the mean and standard deviation of the posterior probability distribution of the corresponding line, which follow a normal distribution. Let be the rotation matrix from the camera coordinate system to the model coordinate system. This is the antisymmetric matrix form of CAD model points in the model coordinate system, such as for points. ,have , and This represents the length of the corresponding line projected onto the imaging plane in the image coordinate system along the horizontal and vertical axes. and This refers to the camera's focal length parameter; Step 3.3), predict the pose change vector after iterative optimization. Update pose estimation results: （5） In the formula, This is the transformation matrix from the model coordinate system to the camera coordinate system. and for The rotational and translational components.

5. The method according to claim 4, characterized in that, Step 4) specifically involves: Step 4.1) In the initial stage, the assembly objects in the aircraft equipment compartment scene are online pose estimated according to the method described in Step 2) to obtain the initial pose. The initial pose is used as the first frame. The pose estimation of the assembly objects in subsequent image frames is completed iteratively according to the method described in Step 3). Step 4.2), using the F-norm of equation (4) as the evaluation function. ; （6） In the formula, For noise parameters, for The F-norm; The pose estimation results in the iterative process are evaluated. The threshold for avionics is set to 4. For avionics devices A, B, and C, whose sizes are arranged from smallest to largest, the thresholds are set to 4 respectively. The values ​​are 2.4, 2.8, and 2.

9. When the value exceeds the set threshold, the current pose estimation result is considered to have low reliability. Then, the method described in step 3) is inserted to perform pose estimation again, thereby correcting the erroneous pose estimation result and improving the robustness of pose estimation.

6. The method according to claim 5, characterized in that, Step 5) specifically involves: Step 5.1) Construct an assembly process database to enable the storage and retrieval of assembly information, including 3D models of assembly parts, assembly tools, assembly steps, process parameters, and assembly guidance information. Step 5.2) According to the method described in Step 1), locate the pose of the depth camera in the aircraft equipment cabin scene. Based on the QR code and SLAM algorithm in the scene, locate the pose of the user's display device with camera function. Combine the pose estimation result of the assembly object in the camera coordinate system obtained in Step 4) to calculate the pose of the assembly object in the display device coordinate system. Then display the assembly information at the corresponding position. Specifically, it includes: (1) The assembly objects involved in the assembly steps being executed are rendered in a semi-transparent manner at the corresponding pose position and distinguished by different colors. The holes, shafts, interfaces and joints in the assembly parts are highlighted in orange; (2) Play the three-dimensional assembly simulation animation at the corresponding position of the assembly object, including the assembly process, tools and actions, and provide prompts with assembly guidance information such as arrows, curves, circles, exclamation marks and voice; (3) Display the process indicators including torque, number of winding turns and measurement data in an easy-to-browse manner on the side of the assembly object. Other process indicators and textual descriptions of assembly steps are displayed in the upper right corner of the device. Step 5.3) In each assembly step, the pose estimation result of the assembly object is used to determine whether the assembly is complete and whether the assembly result is correct, and the next assembly step is automatically guided. Specifically, this includes: (1) If the pose estimation result of the assembly object changes by no more than 2 cm / 20 degrees within one second, the assembly object is considered to be stationary, that is, the assembly step is completed; (2) If the pose estimation result of the assembly object deviates from the correct assembly pose of the assembly object in the scene by no more than 3 cm / 10 degrees, the assembly result is considered to be correct; (3) If the assembly is not completed, the assembly is guided to continue; if the assembly is completed but the result is wrong, the incorrect assembly part and the outline of the assembly object are highlighted in red, and the correct assembly position is indicated in yellow and the assembly is guided to continue; if the assembly step is completed and the result is correct, the next assembly step is automatically entered until the assembly task is completed, the assembly process log is recorded and the evidence is saved.

Images