A Spatial Positioning and Tracking Registration Method for Augmented Reality Assembly Guidance of Aero-engine Components

Abstract

This invention provides a spatial localization and tracking registration method for augmented reality assembly guidance of aero-engine parts, solving the problems of difficulty in small target detection, low pose estimation accuracy, and poor real-time performance in complex scenarios in existing technologies. This method improves the YOLOv8 target detection module by introducing an adaptive weight adjustment mechanism to accurately identify parts in aero-engines; it uses a SAM-6D model to perform fine segmentation, extracting the precise contours and six-DOF attitude information of the parts, comparing and registering them with predefined CAD models to ensure the accuracy of part installation and maintenance positions; furthermore, it introduces confidence scoring, adjustment error, and optimization evaluation criteria, combined with the L-M optimization algorithm, to track the attitude changes of the parts in real time and provide dynamic adjustment feedback. The method proposed in this invention is suitable for the precise localization and efficient tracking of complex parts in aero-engine assembly and maintenance scenarios, improving the real-time performance and accuracy of detection and pose estimation.

Description

Technical Field

[0001] This invention relates to the field of precision inspection technology for aero-engine manufacturing and maintenance, and in particular to a spatial positioning and tracking registration method for aero-engine parts with augmented reality assembly guidance. Background Technology

[0002] The assembly and maintenance of aero-engines requires precise positioning and attitude tracking of various components, representing a complex application scenario based on computer vision. Existing methods typically integrate traditional computer vision and sensor technologies to achieve component positioning and tracking accuracy. However, due to the high complexity of the scenario, the variety of components, and their significant size differences, existing models struggle to accurately handle these variations, resulting in performance bottlenecks in small target detection, pose estimation, and real-time tracking. Furthermore, the real-time performance and accuracy of existing technologies are insufficient to meet the high demands of assembly and maintenance tasks when faced with dynamic changes in the size, shape, and position of different components. Therefore, to address these challenges in the spatial positioning and attitude tracking of aero-engine components, this invention proposes a novel method based on an improved YOLOv8 and SAM-6D model, which can efficiently and accurately solve the problems of difficult small target detection and inaccurate attitude tracking. Summary of the Invention

[0003] To address the problems existing in the prior art, this invention proposes a spatial positioning and tracking registration method for augmented reality assembly guidance of aero-engine parts. This method solves the problems of difficult target detection, low pose estimation accuracy, and poor real-time performance in complex scenarios, and achieves efficient detection, accurate segmentation, spatial position and attitude tracking of engine parts under multiple degrees of freedom.

[0004] To achieve the above-mentioned technical objectives, the present invention provides the following technical solution:

[0005] A spatial positioning and tracking registration method for augmented reality assembly guidance of aero-engine parts includes the following steps:

[0006] S1. Acquire images of aircraft engine parts; use cameras to capture multi-view, multi-angle image data to obtain comprehensive part information;

[0007] S2. Using the part images collected in step S1 as input, perform target detection on each part based on the YOLOv8 model and output a two-dimensional bounding box.

[0008] S3. Based on the two-dimensional bounding box output in step S2, extract the corresponding region of the detected part and input it into the SAM-6D model for pixel segmentation to extract the precise contour information of the part; analyze the segmentation results and perform part pose estimation, that is, calculate the actual pose of the part including three-dimensional position and rotation angle, and output six-degree-of-freedom 6-DoF pose information.

[0009] S4. Compare the pose information obtained in step S3 with the preset CAD model of the part, and use the LM optimization algorithm to calculate the spatial error between the part pose and the CAD model; adjust the position and attitude according to the calculation results to accurately align with the CAD model and complete the 3D registration.

[0010] S5. Continuously acquire multiple frames of images; and repeat steps S2-S3 to obtain the part pose information of two consecutive frames; and perform smoothing processing, compare the position and rotation changes between consecutive frames in real time, track and monitor the spatial position and attitude of the part in real time; if a pose deviation is detected, generate a feedback signal.

[0011] S6. Based on the tracking results of step S5, establish a real-time feedback mechanism. When feedback information is detected, i.e., when a pose deviation occurs, an adjustment command is issued to control the system to make timely pose adjustments. At the same time, record various parameters during the adjustment process for subsequent optimization and verification, and periodically evaluate system performance, optimize the parameter configuration of each module, and improve the accuracy and stability of the overall system's spatial positioning and tracking registration.

[0012] Furthermore, in step S1, the camera is a high-resolution camera; and the position and angle of the camera are set to cover all key components to ensure that there are no shadows or reflections in the lighting conditions.

[0013] Furthermore, in step S2, the target detection described in step S2 utilizes the feature pyramid network structure of the YOLOv8 model, specifically as follows:

[0014] S21. Load the trained weights, configure the model parameters, input the acquired part images into the YOLOv8 model and perform forward propagation. The formula is as follows:

[0015] f i,j,k =sigmoid(W k ·X i,j +b k );

[0016] Among them, f i,j,k W represents the activation value at coordinates i,j on the feature map. k Let X be the weight of the k-th convolutional kernel. i,j b is the pixel value of a local part of the input image. kThe bias term is used, and the sigmoid function is used as the activation function; the detection results for each part are generated through forward propagation.

[0017] S22. Perform threshold filtering on the detection results, remove detection boxes with confidence scores below the set threshold, generate a two-dimensional bounding box for each detected part, and record the part's category, location coordinates, and overall confidence score.

[0018] More specifically, in the object detection process of step S2, an adaptive weight adjustment mechanism is introduced to dynamically adjust the weights of the feature pyramid in the YOLOv8 model, specifically as follows:

[0019] Let F be the output of each level of the feature pyramid. n It contains multiple feature maps, and the formula is expressed as:

[0020]

[0021] Where n represents the nth layer, and K represents the number of convolutional kernels used in this layer;

[0022] Assign adaptive weights α to different layers n The adaptive weight α n Adjustments are made based on the part's dimensions and shape information, expressed by the formula:

[0023]

[0024] Where, r n and r shape,n μ represents the size ratio and shape ratio of the nth layer, respectively; r μ shape It is the corresponding mean, σ r σ shape is the corresponding standard deviation, β is the balance factor, and Z is the normalization constant;

[0025] The formulas for calculating the size ratio and shape ratio are as follows:

[0026]

[0027] Among them, A bbox A represents the area of ​​the detection box. img w represents the total area of ​​the image. bbox h represents the width of the detection box. bbox Indicates the height of the detection frame;

[0028] Based on the adaptive weights assigned above, the importance of each feature map is dynamically adjusted, and the final feature pyramid network output is:

[0029] Furthermore, the specific formula for calculating the comprehensive confidence level in step S22 is as follows:

[0030] Confidence=P(Object)×P(Class);

[0031] Here, Confidence represents the overall confidence level; P(Object) is the probability that the target exists, with a value between 0 and 1. The closer it is to 1, the more likely the target is to exist; P(Class) is the probability that the target belongs to a specific class. A probability value is generated for each class. The higher the class confidence, the more confident the model is that the target belongs to that class.

[0032] Further, in step S3, the target part is first segmented using the ISM instance segmentation model, and then the actual pose of the part is estimated using the pose estimation model PEM through a two-stage part-to-part point matching process, specifically including:

[0033] S31. For each candidate object m that matches the preset CAD model of the part, PEM uses a point matching strategy to predict its 6D pose relative to the CAD model, and represents the segmented candidate object m and the sampling point set of the CAD model as follows: and Where N m and No represent the number of points in the point sets, respectively; and the features of the two point sets are represented as and C represents the number of channels for the feature;

[0034] S32 and ISM equip the point set with a background token, denoted as... and Construct attention matrix A and assignment matrix The formula is expressed as:

[0035] Attention Matrix A:

[0036]

[0037] Allocation matrix

[0038]

[0039] Among them, Softmax row and Softmax col These represent normalization operations along the rows and columns of the matrix, respectively; τ represents a constant. The value of each row in the table, except for the first row, represents the point set P. m Each point P in m With background and Po The midpoint matching score, by locating the index of the maximum matching score, can be used to find the match with P. m Matching points; constructing the assignment matrix Then all matching point pairs {(P m ,P o The actual pose of the part is obtained by weighted SVD, along with the corresponding matching score.

[0040] S33. For each pose estimation result, the SAM-6D model generates a confidence score C. pose This is to evaluate the reliability of the pose estimation results.

[0041] More specifically, step S33 is as follows:

[0042] The confidence score C pose Expressed as:

[0043] C pose =α·C bbox +β·C seg +γ·C feat ;

[0044] Among them, C bbox C represents the confidence score of the YOLOv8 bounding box, reflecting the model's confidence in the part detection; seg It is a score based on segmentation accuracy, measuring the difference between the segmented contour and the actual contour; C feat It is a score based on the feature point matching degree during the pose estimation process; α, β, and γ are weighting coefficients used to balance the contributions of various factors;

[0045] When outputting pose estimation results, the system will simultaneously provide pose information and the corresponding confidence score C. pose The confidence score is used to evaluate the reliability of the pose estimation results; the confidence score is between 0 and 1, where 1 indicates the most reliable pose estimation result and 0 indicates unreliable.

[0046] Furthermore, step S4 specifically includes:

[0047] S41. The pose of the part is represented by a homogeneous transformation matrix, wherein the matrix contains three-dimensional translation and rotation information; specifically, it is represented as follows:

[0048]

[0049] Among them, T actual This represents the actual pose of the part, i.e., the pose estimation result; T CAD Represents the desired pose of the part, i.e., the preset pose of the CAD model; R actual and R CAD These are the rotation matrices of the corresponding matrices, pactual and p CAD It is the translation vector of the corresponding matrix;

[0050] S42, using the rotation error E between the actual pose and the desired pose. rotation Translation error E translation To define the overall error, that is:

[0051] Rotation error is calculated by the difference in the rotation matrices, and the formula is as follows:

[0052] E rotation =||R actual -R CAD || F ;

[0053] Translation error is calculated using Euclidean distance, expressed by the following formula:

[0054] E translation =||p actual -p CAD ||2;

[0055] The total error is the sum of the two errors mentioned above, expressed by the formula:

[0056] E total =λ1E rotation +λ2E translation ;

[0057] Wherein, λ1 and λ2 are weighting coefficients used to balance the effects of rotation error and translation error;

[0058] S43. Minimize the total error E using the LM optimization algorithm. total The goal is to find the optimal pose adjustment amount ΔT = (ΔR, Δp), where ΔR is the rotation adjustment amount and Δp is the pose adjustment amount, so that the actual pose gradually approaches the desired pose.

[0059] More specifically, step S43 involves using the LM optimization algorithm to minimize the total error E. total Specifically, it includes:

[0060] S431. Calculate the Jacobian matrices for rotation error and translation error respectively:

[0061]

[0062] The Jacobian matrix of the total error is then:

[0063] J = [J rotation J translation ];

[0064] S432. In each iteration, the LM algorithm tracks and adjusts the pose based on the error and the Jacobian matrix. The formula for calculating the adjustment amount ΔT is:

[0065] ΔT=-(J T J+λI) -1 J T r;

[0066] Where r = (T) actual -T CAD ), where I is the identity matrix and λ is the damping coefficient in the LM algorithm;

[0067] The adjustment amount ΔT includes corrections for rotation and translation, thus yielding a new pose matrix:

[0068]

[0069] Among them, R new =R actual +ΔR,p new =p actual +Δp;

[0070] S433. After each pose adjustment, recalculate the new pose matrix T. new With CAD model pose T CAD If the error obtained is still greater than the preset tolerance, the LM optimization algorithm is used to continue iterating until the error is reduced to within the tolerance range, and the pose adjustment is completed.

[0071] Furthermore, in step S6,

[0072] The performance evaluation system specifically includes the following indicators:

[0073] (1) Error convergence: If the total error gradually decreases and tends to converge after multiple adjustments, it indicates that the system performance is good; the formula for evaluating error convergence is:

[0074] E current <E previous ;

[0075] Where E current E represents the total error after adjustment. previous This indicates the total error of the previous adjustment; if the error gradually decreases and the convergence condition is met, it means that the system adjustment command is effective.

[0076] (2) Number of adjustments: If the number of adjustments exceeds the set threshold, the system performance is considered poor and the adjustment parameters need to be optimized.

[0077] (3) Adjustment time: The shorter the time from detecting pose deviation to completing the adjustment, the faster the system response and the better the performance; this is determined by recording the response time T for each adjustment. adjust To conduct an evaluation;

[0078] The optimization of parameter configurations for each module includes:

[0079] The damping coefficient λ in the LM optimization algorithm;

[0080] The formula for adjusting the damping coefficient is as follows:

[0081]

[0082] The weighting coefficients μ and ν are used to adjust the relative sensitivity to rotation and translation errors, and the adjustment relationship is μ rotation =ν·μ translation .

[0083] Based on the above technical solution, the method proposed in this invention has the following beneficial effects:

[0084] The method proposed in this invention achieves spatial localization, tracking, and registration of aero-engine parts based on YOLOv8 and SAM-6D models. By improving the YOLOv8 module and introducing an adaptive weight adjustment mechanism, the detection accuracy of small targets and complex parts is enhanced. The SAM-6D module enables precise segmentation and pose estimation, and a confidence scoring system is introduced. Real-time tracking and spatial registration ensure high-precision positioning and attitude adjustment of parts during assembly and maintenance. During attitude adjustment, the LM optimization algorithm and performance evaluation indicators are combined to enhance the overall system reliability and response efficiency, providing an efficient and automated solution for assembly and maintenance in complex industrial scenarios. Attached Figure Description

[0085] Figure 1 This is a flowchart illustrating the overall implementation of a spatial positioning and tracking registration method for augmented reality assembly guidance of aero-engine parts proposed in this invention. Detailed Implementation

[0086] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0087] Although the steps in this invention are arranged by reference numerals, this is not intended to limit the order of the steps. Unless the order of the steps is explicitly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0088] like Figure 1 As shown, a spatial positioning and tracking registration method for augmented reality assembly guidance of aero-engine parts proposed in this invention is presented, specifically including:

[0089] S1. Acquire images of aircraft engine parts; use cameras to capture multi-view, multi-angle image data to obtain comprehensive part information;

[0090] In this embodiment, a high-resolution camera is used; appropriate exposure time, aperture, and focal length are set to optimize image quality; during the assembly or maintenance of aero-engines, the camera position and angle are set to ensure coverage of all critical components; suitable lighting conditions are adopted to eliminate shadows and reflections, thereby improving image visibility and contrast; multi-view and multi-angle image data are captured to obtain comprehensive component information and enhance the robustness of subsequent processing.

[0091] S2. Using the part images collected in step S1 as input, perform target detection on each part based on the YOLOv8 model and output a two-dimensional bounding box.

[0092] In a preferred embodiment, step S2 specifically includes:

[0093] S21. Load the trained weights, configure the model parameters, input the acquired part images into the YOLOv8 model and perform forward propagation. The formula is as follows:

[0094] f i,j,k =σ(W k ·X i,j +b k );

[0095] Among them, f i,j,k W represents the activation value at coordinates i,j on the feature map. k Let X be the weight of the k-th convolutional kernel. i,j b is the pixel value of a local part of the input image. k σ is the bias term, and σ is the activation function; through forward propagation, the detection results for each part are generated;

[0096] S22. Perform threshold filtering on the detection results, remove detection boxes with confidence scores below the set threshold, generate a two-dimensional bounding box for each detected part, and record the part's category, location coordinates, and overall confidence score.

[0097] In this embodiment, the YOLOv8 model first performs forward propagation on the input image through a feature extraction network to generate feature maps at multiple levels. These feature maps are high-level feature representations extracted from the input image by a convolutional neural network, used to capture information from different parts of the image.

[0098] After generating feature maps, the YOLOv8 model calculates the detection results for each candidate target based on the feature maps. These results include the probability of the target's existence, the target category, and the overall confidence score. Each detection result corresponds to a bounding box, which is then adjusted through regression to generate a two-dimensional bounding box around the target. After filtering, detection boxes with confidence scores below a set threshold are removed, and the remaining boxes are used as the final two-dimensional bounding boxes. The model also records the part's category, location coordinates, and overall confidence score. The overall confidence score is obtained by multiplying the probability of the target's existence by the probability of the target's category; a higher confidence score indicates a more reliable detection.

[0099] The specific formula for calculating the overall confidence level is as follows:

[0100] Confidence=P(Object)×P(Class);

[0101] Here, Confidence represents the confidence level information; P(Object) is the probability that the target exists, with a value between 0 and 1. The closer it is to 1, the more likely the target is to exist; P(Class) is the probability that the target belongs to a specific class. A probability value is generated for each class. The higher the class confidence, the more confident the model is that the target belongs to that class.

[0102] More specifically, in the object detection process of step S2, an adaptive weight adjustment mechanism is introduced to dynamically adjust the weights of the feature pyramid in the YOLOv8 model, specifically as follows:

[0103] Let F be the output of each level of the feature pyramid. n It contains multiple feature maps, and the formula is expressed as:

[0104]

[0105] Where n represents the nth layer, and K represents the number of convolutional kernels used in that layer.

[0106] Assign adaptive weights α to different layers n The adaptive weight α nAdjustments are made based on the part's dimensions and shape information, expressed by the formula:

[0107]

[0108] Where, r n and r shape,n μ represents the size ratio and shape ratio of the nth layer, respectively; r μs shape It is the corresponding mean, σ r σ shape is the corresponding standard deviation, β is the balance factor, and Z is the normalization constant;

[0109] The formulas for calculating the size ratio and shape ratio are as follows:

[0110]

[0111] Among them, A bbox A represents the area of ​​the detection box. img w represents the total area of ​​the image. bbox h represents the width of the detection box. bbox This indicates the height of the detection frame.

[0112] By introducing an adaptive weighting mechanism, multi-scale feature extraction can be optimized, further improving the accuracy of small target detection and the pose estimation accuracy of complex parts.

[0113] S3. Based on the two-dimensional bounding box output in step S2, extract the corresponding region and input it into the SAM-6D model for pixel segmentation to extract the precise contour information of the part; analyze the segmentation results and perform part pose estimation, that is, calculate the actual pose of the part including three-dimensional position and rotation angle, and output six-degree-of-freedom 6-DoF pose information.

[0114] In a preferred embodiment, step S3 first involves segmenting the target part using an ISM instance segmentation model, and then estimating the actual pose of the part using a pose estimation model PEM through a two-stage part-to-part point matching process. Specifically, this includes:

[0115] S31. For each candidate object m that matches the preset CAD model of the part, PEM uses a point matching strategy to predict its 6D pose relative to the CAD model, and represents the segmented candidate object m and the sampling point set of the CAD model as follows: and Where N m and No represent the number of points in the point sets, respectively; and the features of the two point sets are represented as and C represents the number of channels for the feature;

[0116] S32 and ISM equip the point set with a background token, denoted as... and Construct attention matrix A and assignment matrix The formula is expressed as:

[0117] Attention Matrix A:

[0118]

[0119] Allocation matrix

[0120]

[0121] Among them, Softmax row and Softmax col These represent normalization operations along the rows and columns of the matrix, respectively; τ represents a constant. The value of each row in the table, except for the first row, represents the point set P. m Each point P in m With background and P o The midpoint matching score, by locating the index of the maximum matching score, can be used to find the match with P. m Matching points; constructing the assignment matrix Then all matching point pairs {(P m ,P o The actual pose of the part is obtained by weighted SVD, along with the corresponding matching score.

[0122] S33. For each pose estimation result, the SAM-6D model generates a confidence score C. pose This is to evaluate the reliability of the pose estimation results.

[0123] In this embodiment, the confidence score C pose This is mainly based on the following factors:

[0124] The accuracy of the model in segmenting the edges of the part is considered; the smaller the difference between the segmented contour and the actual contour, the higher the confidence level. The degree of matching between the feature points extracted by the model during pose estimation and the actual feature points is also considered; the higher the matching degree, the higher the confidence level. The confidence level C of the bounding box generated by the YOLOv8 model is also considered. bbox This confidence level directly affects the initial input for pose estimation.

[0125] Taking all these factors into account, the confidence score C pose Represented as:

[0126] C pose =α·C bbox+β·C seg +γ·C feat ;

[0127] Among them, C bbox C represents the confidence score of the YOLOv8 bounding box, reflecting the model's confidence in the part detection; seg It is a score based on segmentation accuracy, measuring the difference between the segmented contour and the actual contour; C feat It is a score based on the feature point matching degree during the pose estimation process; α, β, and γ are weighting coefficients used to balance the contributions of various factors;

[0128] When outputting pose estimation results, the system will simultaneously provide pose information and the corresponding confidence score C. pose The confidence score is used to evaluate the reliability of the pose estimation results; the confidence score is between 0 and 1, where 1 indicates the most reliable pose estimation result and 0 indicates unreliable.

[0129] S4. Compare the pose information obtained in step S3 with the preset CAD model of the part, and use the LM optimization algorithm to calculate the spatial error between the part pose and the CAD model; adjust the position and attitude according to the calculation results to accurately align with the CAD model and complete the 3D registration.

[0130] In a preferred embodiment, step S4 includes:

[0131] S41. The pose of the part is represented by a homogeneous transformation matrix, wherein the matrix contains three-dimensional translation and rotation information; specifically, it is represented as follows:

[0132]

[0133] Among them, T actual This represents the actual pose of the part, i.e., the pose estimation result; T CAD Represents the desired pose of the part, i.e., the preset pose of the CAD model; R actual and R CAD These are the rotation matrices of the corresponding matrices, p actual and p CAD It is the translation vector of the corresponding matrix;

[0134] S42, using the rotation error E between the actual pose and the desired pose. rotation Translation error E translation To define the overall error, that is:

[0135] Rotation error is calculated by the difference in the rotation matrices, and the formula is as follows:

[0136] E rotation =||R actual -R CAD ||F ;

[0137] Translation error is calculated using Euclidean distance, expressed by the following formula:

[0138] E translation =||p actual -p CAD ||2;

[0139] The total error is the sum of the two errors mentioned above, expressed by the formula:

[0140] E total =λ1E rotation +λ2E translation ;

[0141] Wherein, λ1 and λ2 are weighting coefficients used to balance the effects of rotation error and translation error;

[0142] S43. Minimize the total error E using the LM optimization algorithm. total The goal is to find the optimal pose adjustment amount ΔT = (ΔR, Δp), where ΔR is the rotation adjustment amount and Δp is the pose adjustment amount, so that the actual pose gradually approaches the desired pose.

[0143] More specifically, step S43 includes:

[0144] S431. Calculate the Jacobian matrices for rotation error and translation error respectively:

[0145]

[0146] The Jacobian matrix of the total error is then:

[0147] J = [J rotation J translation ];

[0148] S432. In each iteration, the LM algorithm tracks and adjusts the pose based on the error and the Jacobian matrix. The formula for calculating the adjustment amount ΔT is:

[0149] ΔT=-(J T J+λI) -1 J T r;

[0150] Where r = (T) actual -T CAD ), where I is the identity matrix and λ is the damping coefficient in the LM algorithm;

[0151] The adjustment amount ΔT includes corrections for rotation and translation, thus yielding a new pose matrix:

[0152]

[0153] Among them, R new =R actual +ΔR,p new =p actual +Δp;

[0154] S433. After each pose adjustment, recalculate the new pose matrix T. new With CAD model pose T CAD If the error obtained is still greater than the preset tolerance, the LM optimization algorithm is used to continue iterating until the error is reduced to within the tolerance range, and the pose adjustment is completed.

[0155] In this embodiment, tolerance ε is defined. rotation With ε translation These are the error tolerances for rotation and translation, respectively. The tolerance standard is set according to requirements. The rotation error tolerance is that the attitude jitter between adjacent frames is less than 0.3°, i.e., ε. rotation <0.3°, the tolerance for translation error is that the inter-frame displacement jitter is less than 0.1%, i.e., ε translation <0.1%. When satisfied

[0156] E new,rotation <ε rotation, E new,translation <ε translation ;

[0157] The system determines that pose alignment is complete and stops iterating.

[0158] S5. Continuously acquire multiple frames of images; and repeat steps S2-S3 to obtain the part pose information of two consecutive frames; and perform smoothing processing, compare the position and rotation changes between consecutive frames in real time, track and monitor the spatial position and attitude of the part in real time; if a pose deviation is detected, generate a feedback signal.

[0159] In this embodiment, step S4 tends to focus on pose estimation and 3D alignment in static scenes, while step S5 mainly handles pose tracking and real-time feedback in dynamic scenes to ensure that parts maintain the correct pose and position during assembly.

[0160] S6. Based on the tracking results of step S5, establish a real-time feedback mechanism. When feedback information is detected, i.e., when a pose deviation occurs, an adjustment command is issued to control the system to make timely pose adjustments. At the same time, record various parameters during the adjustment process for subsequent optimization and verification, and periodically evaluate system performance, optimize the parameter configuration of each module, and improve the accuracy and stability of the overall system's spatial positioning and tracking registration.

[0161] In a preferred embodiment, the criterion for judging the pose deviation in step S6 is determined by comparing the current pose of the part with the preset target pose (i.e., the pose of the preset CAD model); when ΔR>ε rotation And Δp>ε translation The system determines that a pose deviation has occurred.

[0162] When the system detects a rotational error ΔR, it adjusts the pose according to the following formula:

[0163] R new =R actual +ΔR;

[0164] The system will use the newly calculated R new Used to reset the rotational orientation of the part, ensuring that the part's angle is aligned with the target orientation.

[0165] When the system detects a translational deviation, it adjusts according to the following formula:

[0166] p new =p actual +Δp;

[0167] After pose adjustment, the system needs to be evaluated for performance and whether further optimization of system parameters is required. The evaluation criteria mainly include the following aspects:

[0168] (1) Error convergence: If the total error gradually decreases and tends to converge after multiple adjustments, it indicates that the system performance is good; the formula for evaluating error convergence is:

[0169] E current <E previous ;

[0170] Where E current E represents the total error after adjustment. previous This indicates the total error of the previous adjustment; if the error gradually decreases and the convergence condition is met, it means that the system adjustment command is effective.

[0171] (2) Number of adjustments: If the number of adjustments exceeds the set threshold, which is set to 5 in this embodiment, the system performance is considered to be poor and the adjustment parameters need to be optimized.

[0172] (3) Adjustment time: The shorter the time from detecting pose deviation to completing the adjustment, the faster the system response and the better the performance; this is determined by recording the response time T for each adjustment. adjust To conduct the evaluation; in this embodiment, T adjust A time response time of <20ms is considered good performance.

[0173] The optimization of parameter configurations for each module includes:

[0174] The damping coefficient λ in the LM optimization algorithm;

[0175] The formula for adjusting the damping coefficient is as follows:

[0176]

[0177] The weighting coefficients μ and ν are used to adjust the relative sensitivity to rotation and translation errors, and the adjustment relationship is μ rotation =ν·μ translation .

[0178] After parameter adjustment, the system's performance was verified again through experiments. The adjustment time, number of adjustments, and error convergence were recorded to evaluate the optimization effect until the system converged.

[0179] In summary, the method proposed in this invention, by improving the YOLOv8 and SAM-6D models and combining confidence scoring and a real-time evaluation optimization system, solves the problems of difficult target detection, low pose estimation accuracy, and poor real-time performance of existing methods in complex scenes, and achieves efficient detection, accurate segmentation, spatial position and attitude tracking of engine parts under multiple degrees of freedom.

[0180] It should be noted that, for those skilled in the art, it is obvious that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0181] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A spatial positioning and tracking registration method for augmented reality assembly guidance of aero-engine parts, characterized in that, Specifically, the following steps are included: S1. Acquire images of aircraft engine parts; use cameras to capture multi-view, multi-angle image data to obtain comprehensive part information; S2. Using the part images collected in step S1 as input, perform target detection on each part based on the YOLOv8 model and output a two-dimensional bounding box. The target detection described in step S2 utilizes the feature pyramid network structure of the YOLOv8 model, specifically as follows: S21. Load the trained weights, configure the model parameters, input the acquired part images into the YOLOv8 model and perform forward propagation. The formula is as follows: f i,j,k =sigmoid(W k ·X i,j +b k ); Among them, f i,j,k W represents the activation value at coordinates i,j on the feature map. k Let X be the weight of the k-th convolutional kernel. i,j b is the pixel value of a local part of the input image. k The bias term is used, and the sigmoid function is used as the activation function; the detection results for each part are generated through forward propagation; the weights of the convolution kernel are learned through the training process. S22. Perform threshold filtering on the detection results, remove detection boxes with confidence scores below the set threshold, generate a two-dimensional bounding box for each detected part, and record the part's category, location coordinates, and overall confidence score. In step S2, during object detection, an adaptive weight adjustment mechanism is also introduced to dynamically adjust the weights of the feature pyramid in the YOLOv8 model. Specifically: Let F be the output of each level of the feature pyramid. n It contains multiple feature maps, and the formula is expressed as: Where n represents the nth layer, and K represents the number of convolutional kernels used in that layer; Assign adaptive weights α to different layers n The adaptive weight α n Adjustments are made based on the part's dimensions and shape information, expressed by the formula: Where, r n and r shape,n These represent the size ratio and shape ratio of the nth layer, respectively; μ r μ shape It is the corresponding mean, σ r σ shape is the corresponding standard deviation, β is the balance factor, and Z is the normalization constant; The formulas for calculating the size ratio and shape ratio are as follows: Among them, A bbox A represents the area of ​​the detection box. img w represents the total area of ​​the image. bbox h represents the width of the detection box. bbox Indicates the height of the detection frame; Based on the adaptive weights assigned above, the importance of each feature map is dynamically adjusted, and the final feature pyramid network output is: S3. Based on the two-dimensional bounding box output in step S2, extract the corresponding region of the detected part and input it into the SAM-6D model for pixel segmentation to extract the precise contour information of the part; analyze the segmentation results and perform part pose estimation, that is, calculate the actual pose of the part including three-dimensional position and rotation angle, and output six-degree-of-freedom 6-DoF pose information. S4. Compare the pose information obtained in step S3 with the preset CAD model of the part, and use the LM optimization algorithm to calculate the spatial error between the part pose and the CAD model; adjust the position and attitude according to the calculation results to accurately align with the CAD model and complete the 3D registration. S5. Continuously acquire multiple frames of images; and repeat steps S2-S3 to obtain the part pose information of two consecutive frames; and perform smoothing processing, compare the position and rotation changes between consecutive frames in real time, track and monitor the spatial position and attitude of the part in real time; if a pose deviation is detected, generate a feedback signal. S6. Based on the tracking results of step S5, establish a real-time feedback mechanism. When feedback information is detected, i.e., when a pose deviation occurs, an adjustment command is issued to control the system to make timely pose adjustments. At the same time, record various parameters during the adjustment process for subsequent optimization and verification, and periodically evaluate system performance, optimize the parameter configuration of each module, and improve the accuracy and stability of the overall system's spatial positioning and tracking registration.

2. The spatial positioning and tracking registration method for augmented reality assembly guidance of aero-engine parts according to claim 1, characterized in that, Step S1 specifically includes: The camera is a high-resolution camera; and the camera's position and angle are set to cover all critical components, ensuring that there are no shadows or reflections under the lighting conditions.

3. The spatial positioning and tracking registration method for augmented reality assembly guidance of aero-engine parts according to claim 1, characterized in that, The specific formula for calculating the comprehensive confidence level in step S22 is as follows: Confidence=P(Object)×P(Class); Here, Confidence represents the overall confidence level; P(Object) is the probability that the target exists, with a value between 0 and 1. The closer it is to 1, the more likely the target is to exist; P(Class) is the probability that the target belongs to a specific class. A probability value is generated for each class. The higher the class confidence, the more confident the model is that the target belongs to that class.

4. The spatial positioning and tracking registration method for augmented reality assembly guidance of aero-engine parts according to claim 1, characterized in that, In step S3, the target part is first segmented using the ISM instance segmentation model, and then the actual pose of the part is estimated using the Pose Estimation Model (PEM) through a two-stage part-to-part point matching process. Specifically, this includes: S31. For each candidate object m that matches the preset CAD model of the part, PEM uses a point matching strategy to predict its 6D pose relative to the CAD model, and represents the segmented candidate object m and the sampling point set of the CAD model as follows: and Where N m and N o Let each be the number of points in the point set; and let the characteristics of the two point sets be represented as... and C represents the number of channels for the feature; S32 and ISM equip the point set with a background token, denoted as... and Construct attention matrix A and assignment matrix The formula is expressed as: Attention Matrix A: Allocation matrix Among them, Softmax row and Softmax col These represent normalization operations along the rows and columns of the matrix, respectively; τ represents a constant. The value of each row in the table, except for the first row, represents the point set P. m Each point in the set of points P o The midpoint matching score, by locating the index of the maximum matching score, can be used to find the match with P. m Matching points; constructing the assignment matrix Then all matching point pairs {(P m ,P o The actual pose of the part is obtained by weighted SVD, along with the corresponding matching score. S33. For each pose estimation result, the SAM-6D model generates a confidence score C. pose This is to evaluate the reliability of the pose estimation results.

5. A spatial positioning and tracking registration method for augmented reality assembly guidance of aero-engine parts according to claim 4, characterized in that, Step S33 is as follows: The confidence score C pose Represented as: C pose =α·C bbox +β·C seg +γ·C feat ; Among them, C bbox C represents the confidence score of the YOLOv8 bounding box, reflecting the model's confidence in the part detection; seg It is a score based on segmentation accuracy, measuring the difference between the segmented contour and the actual contour; C feat It is a score based on the feature point matching degree during the pose estimation process; α, β, and γ are weighting coefficients used to balance the contributions of various factors; When outputting pose estimation results, the system will simultaneously provide pose information and the corresponding confidence score C. pose The confidence score is used to evaluate the reliability of the pose estimation results; the confidence score is between 0 and 1, where 1 indicates the most reliable pose estimation result and 0 indicates unreliable.

6. The spatial positioning and tracking registration method for augmented reality assembly guidance of aero-engine parts according to claim 1, characterized in that, Step S4 specifically includes: S41. The pose of the part is represented by a homogeneous transformation matrix, wherein the matrix contains three-dimensional translation and rotation information; specifically, it is represented as follows: Among them, T actual This represents the actual pose of the part, i.e., the pose estimation result; T CAD Represents the desired pose of the part, i.e., the preset pose of the CAD model; R actual and R CAD These are the rotation matrices of the corresponding matrices, p actual and p CAD It is the translation vector of the corresponding matrix; S42, using the rotation error E between the actual pose and the desired pose. rotation Translation error E translation To define the overall error, that is: Rotation error is calculated by the difference in the rotation matrices, and the formula is as follows: E rotation =||R actual -R CAD || F ; Translation error is calculated using Euclidean distance, expressed by the following formula: E translation =||p actual -p CAD ||2; The total error is the sum of the two errors mentioned above, expressed by the formula: AND total =λ1E rotation +λ2E translation ; Wherein, λ1 and λ2 are weighting coefficients used to balance the effects of rotation error and translation error; S43. Minimize the total error E using the LM optimization algorithm. total The goal is to find the optimal pose adjustment amount ΔT = (ΔR, Δp), where ΔR is the rotation adjustment amount and Δp is the pose adjustment amount, so that the actual pose gradually approaches the desired pose.

7. A spatial positioning and tracking registration method for augmented reality assembly guidance of aero-engine parts according to claim 6, characterized in that, Step S43 describes minimizing the total error E using the LM optimization algorithm. total Specifically, it includes: S431. Calculate the Jacobian matrices for rotation error and translation error respectively: The Jacobian matrix of the total error is then: J=[J rotation ,J translation ]; S432. In each iteration, the LM algorithm tracks and adjusts the pose based on the error and the Jacobian matrix. The formula for calculating the adjustment amount ΔT is: ΔT=-(J T J+lI) -1 J T r; Where r = (T) actual -T CAD ), where I is the identity matrix and λ is the damping coefficient in the LM algorithm; The adjustment amount ΔT includes corrections for rotation and translation, thus yielding a new pose matrix: Among them, R new =R actual +ΔR,p new =p actual +Δp; S433. After each pose adjustment, recalculate the new pose matrix T. new With CAD model pose T CAD If the error obtained is still greater than the preset tolerance, the LM optimization algorithm is used to continue iterating until the error is reduced to within the tolerance range, and the pose adjustment is completed.

8. A spatial positioning and tracking registration method for augmented reality assembly guidance of aero-engine parts according to claim 6, characterized in that, In step S6, The performance evaluation system specifically includes the following indicators: (1) Error convergence: If the total error gradually decreases and tends to converge after multiple adjustments, it indicates that the system performance is good; the formula for evaluating error convergence is: AND current <And previous ; Where E current E represents the total error after adjustment. previous This indicates the total error of the previous adjustment; if the error gradually decreases and the convergence condition is met, it means that the system adjustment command is effective. (2) Number of adjustments: If the number of adjustments exceeds the set threshold, the system performance is considered poor and the adjustment parameters need to be optimized. (3) Adjustment time: The shorter the time from detecting pose deviation to completing the adjustment, the faster the system response and the better the performance; this is determined by recording the response time T for each adjustment. adjust To conduct an evaluation; The optimization of parameter configurations for each module includes: The damping coefficient λ in the LM optimization algorithm; The formula for adjusting the damping coefficient is as follows: The weighting coefficients μ and v are used to adjust the relative sensitivity to rotation and translation errors, and the adjustment relationship is μ rotation =v·μ translation .

Images