Space positioning and tracking registration method oriented to aero-engine part augmented reality assembly guidance

Abstract

The invention provides a space positioning and tracking registration method oriented to aero-engine part augmented reality assembly guidance, and solves the problems of difficulty in small target detection, low pose estimation precision and poor real-time performance when facing a complex scene in the prior art. According to the method, a YOLOv8 target detection module is improved, a self-adaptive weight adjustment mechanism is introduced, and parts in an aero-engine are accurately recognized; performing fine segmentation by using an SAM-6D model, extracting the precise contour and six-degree-of-freedom attitude information of the part, and comparing and registering the precise contour and six-degree-of-freedom attitude information with a predefined CAD model to ensure the mounting and maintenance position precision of the part; in addition, confidence score, adjustment error and optimization evaluation criteria are introduced, and an L-M optimization algorithm is combined, so that the posture change of the part is tracked in real time, and dynamic adjustment feedback is provided. The method provided by the invention is suitable for accurate positioning and efficient tracking of complex parts in an aero-engine assembly and maintenance scene, and the real-time performance and precision of detection and pose estimation are improved.

Description

Technical Field

[0001] The present invention relates to the field of precision detection technology for aircraft engine manufacturing and maintenance, and in particular to a spatial positioning and tracking registration method for augmented reality assembly guidance of aircraft engine parts. Background Art

[0002] The assembly and maintenance process of aircraft engines requires accurate positioning and posture tracking of a variety of different parts, which is a complex application scenario based on computer vision. Existing methods usually achieve the positioning and tracking of parts by integrating traditional computer vision and sensor technology in order to ensure the accuracy of detection and tracking. However, due to the high complexity of the scene, the wide variety of parts and the large size differences, it is difficult for existing models to accurately respond to these changes, resulting in performance bottlenecks in small target detection, pose estimation and real-time tracking. In addition, in the face of dynamic changes in the size, shape and position of different parts, the real-time and accuracy of the existing technology are difficult to meet the high-demand assembly and maintenance tasks. Therefore, in order to cope with these challenges in the spatial positioning and posture tracking of aircraft engine parts, the present invention proposes a novel method based on improved YOLOv8 and SAM-6D models, which can efficiently and accurately solve the problems of difficult small target detection and inaccurate posture tracking. Summary of the invention

[0003] In view of the problems existing in the above-mentioned prior art, the present invention proposes a spatial positioning and tracking registration method for augmented reality assembly guidance of aviation engine parts, so as to solve the problems of difficult target detection, low pose estimation accuracy and poor real-time performance of the existing methods in complex scenes, and realize efficient detection, precise segmentation, spatial position and posture tracking of engine parts in multiple degrees of freedom.

[0004] In order to achieve the above technical objectives, the present invention provides the following technical solutions:

[0005] A spatial positioning and tracking registration method for augmented reality assembly guidance of aircraft engine parts, specifically comprising the following steps:

[0006] S1. Collect images of aircraft engine parts; use cameras to capture multi-view and multi-angle image data to obtain comprehensive part information;

[0007] S2, taking the part image collected in step S1 as input, performing target detection on each part based on the YOLOv8 model, and outputting a two-dimensional bounding box;

[0008] S3, based on the two-dimensional bounding box output in step S2, extract the corresponding area of ​​the detected part and input it into the SAM-6D model for pixel segmentation to extract the precise contour information of the part; analyze the segmentation results and perform part pose estimation, that is, calculate the actual pose of the part including the three-dimensional position and rotation angle, and output the six-degree-of-freedom 6-DoF pose information;

[0009] S4, roughly comparing the posture information obtained in step S3 with the preset CAD model of the part, and using the LM optimization algorithm to calculate the spatial error between the posture of the part and the CAD model; adjusting the position and posture according to the calculation results to accurately align with the CAD model to complete the three-dimensional registration;

[0010] S5, continuously collect multiple frames of images; and repeat steps S2-S3 to obtain the part posture information of the two consecutive frames; and perform smoothing, compare the position and rotation changes between consecutive frames in real time, and track and monitor the spatial position and posture of the part in real time; if a posture deviation is detected, a feedback signal is generated;

[0011] S6. According to the tracking results of step S5, a real-time feedback mechanism is established. When feedback information is detected, that is, when posture deviation occurs, an adjustment instruction is issued to control the control system to make timely posture adjustments. At the same time, various parameters in the adjustment process are recorded for subsequent optimization and verification, and the system performance is regularly evaluated to optimize the parameter configuration of each module and improve the accuracy and stability of the overall system spatial positioning and tracking registration.

[0012] Furthermore, in step S1, the camera is a high-resolution camera; and the position and angle of the camera are set to cover all key parts to ensure that there are no shadows and reflections in the lighting conditions.

[0013] Further, in step S2, the target detection in step S2 utilizes the feature pyramid network structure of the YOLOv8 model, specifically:

[0014] S21. Load the trained weights, configure the model parameters, input the collected part images into the YOLOv8 model and forward propagate. The formula is expressed as:

[0015] f i,j,k =sigmoid(W k ·X i,j +b k );

[0016] Among them, f i,j,k represents the activation value at the i,j coordinate on the feature map, W k is the weight of the kth convolution kernel, X i,j is the local pixel value of the input image, b kis the bias term, sigmoid is the activation function; through forward propagation, the detection results of each part are generated;

[0017] S22, performing threshold filtering on the detection results, eliminating detection frames with confidence levels lower than a set threshold, generating a two-dimensional bounding box for each detected part, and recording the part's category, location coordinates, and comprehensive confidence level.

[0018] More specifically, in the process of target detection in step S2, an adaptive weight adjustment mechanism is introduced to dynamically adjust the weights of the feature pyramid in the YOLOv8 model, specifically:

[0019] Let the output of each layer of the feature pyramid be F n , which contains multiple feature maps, the formula is expressed as:

[0020]

[0021] Where n represents the nth layer, and K represents the number of convolution kernels used in this layer;

[0022] Assign adaptive weights α to different layers n ; The adaptive weight α n According to the size and shape information of the parts, adjustments are made. The formula is expressed as:

[0023]

[0024] Among them, r n and r shape,n Respectively represent the size ratio and shape ratio of the nth layer; μ r , μ shape is the corresponding mean, σ r , σ shape is the corresponding standard deviation, β is the balance factor, and Z is the normalization constant;

[0025] The calculation formula of the size ratio and shape ratio is:

[0026]

[0027] Among them, A bbox Represents the area of ​​the detection box, A img represents the total area of ​​the image, w bbox Indicates the width of the detection box, h bbox Indicates the height of the detection box;

[0028] Based on the adaptive weights assigned above, the importance of each layer of feature maps is dynamically adjusted, and the final feature pyramid network output is:

[0029] In addition, the specific calculation process formula of the comprehensive confidence in step S22 is:

[0030] Confidence=P(Object)×P(Class);

[0031] Among them, Confidence represents the comprehensive confidence; P(Object) is the probability of the existence of the target, which takes a value between 0 and 1. The closer it is to 1, the more likely it is that the target exists; P(Class) is the probability that the target belongs to a specific category. A probability value is generated for each category. The higher the category confidence, the more confident the model is that the target belongs to that category.

[0032] Furthermore, in step S3, the target part is first segmented using the ISM instance segmentation model, and then the pose estimation model PEM is used to estimate the actual pose of the part through a two-stage part-to-part point matching process, specifically including:

[0033] S31. For each candidate object m that matches the preset CAD model of the part, PEM uses a point matching strategy to predict its 6D pose relative to the CAD model, and represents the segmented candidate object m and the sampling point set of the CAD model as and Where N m and No are the number of points in the point set respectively; and the characteristics of the two point sets are expressed as and C represents the number of channels of the feature;

[0034] S32, ISM equips the point set with a background token, denoted as and Construct the attention matrix A and the allocation matrix The formula is:

[0035] Attention matrix A:

[0036]

[0037] Allocation Matrix

[0038]

[0039] Among them, Softmax row and Softmax col Respectively represent normalization operations along the rows and columns of the matrix; τ represents a constant; The value of each row in, except the first row, represents the point set P m Each point P m With background and Po The matching score of the midpoint can be found by locating the index of the maximum matching score. m Matched points; constructing the assignment matrix Then we can gather all matching point pairs {(P m ,P o )}, and the corresponding matching score, and finally use weighted SVD to get the actual pose of the part;

[0040] S33. For each pose estimation result, the SAM-6D model generates a confidence score C pose , in order to evaluate the reliability of the pose estimation results.

[0041] More specifically, step S33 is as follows:

[0042] The confidence score C pose It is expressed as:

[0043] C pose =α·C bbox +β·C seg +γ·C feat ;

[0044] Among them, C bbox is the confidence of the YOLOv8 bounding box, reflecting the model’s confidence in part detection; C seg It is a score based on segmentation accuracy, which measures the difference between the segmented contour and the actual contour; C feat It is a score based on the matching degree of feature points in the pose estimation process; α, β, and γ are weight coefficients used to balance the contribution of various factors;

[0045] When the pose estimation result is output, the system will provide both the pose information and the corresponding confidence score C pose , to evaluate the reliability of the pose estimation results; the confidence score ranges from 0 to 1, where 1 represents the most reliable pose estimation result and 0 represents unreliable.

[0046] Furthermore, step S4 specifically includes:

[0047] S41, expressing the position and posture of the part by a homogeneous transformation matrix, wherein the matrix contains three-dimensional translation and rotation information; specifically expressed as:

[0048]

[0049] Among them, T actual represents the actual pose of the part, that is, the pose estimation result; T CAD Represents the expected pose of the part, that is, the preset CAD model pose; R actual and R CAD are the rotation matrices of the corresponding matrices, pactual and p CAD is the translation vector of the corresponding matrix;

[0050] S42, through the rotation error E between the actual posture and the expected posture rotation and translation error E translation To define the overall error, that is:

[0051] The rotation error is calculated by the difference of the rotation matrix, which is expressed as:

[0052] E rotation =||R actual -R CAD || F ;

[0053] The translation error is calculated by Euclidean distance, which is expressed as:

[0054] E translation =||p actual -p CAD ||2;

[0055] The total error is the sum of the above two errors, and the formula is expressed as:

[0056] E total =λ1E rotation +λ2E translation ;

[0057] Among them, λ1 and λ2 are weight coefficients used to balance the influence of rotation error and translation error;

[0058] S43, using LM optimization algorithm to minimize the total error E total , to find the optimal posture adjustment ΔT = (ΔR, Δp), where ΔR is the rotation adjustment and Δp is the posture adjustment; so that the actual posture gradually approaches the desired posture.

[0059] More specifically, in step S43, the total error E is minimized by using the LM optimization algorithm. total Specifically include:

[0060] S431, calculate the Jacobian matrices of the rotation error and the translation error respectively:

[0061]

[0062] The Jacobian matrix of the total error is:

[0063] J=[J rotation ,J translation ];

[0064] S432. At each iteration, the LM algorithm tracks the posture and makes adjustments based on the error and Jacobian matrix. The calculation formula for the adjustment amount ΔT is:

[0065] ΔT=-(J T J+λI) -1 J T r;

[0066] Where r = (T actual -T CAD ), I is the identity matrix, λ is the damping coefficient in the LM algorithm;

[0067] The adjustment ΔT includes the correction of rotation and translation, and the new pose matrix can be obtained:

[0068]

[0069] Among them, R new =R actual +ΔR,p new =p actual +Δp;

[0070] S433, after each posture adjustment, recalculate the new posture matrix T new The pose T of the CAD model CAD If the error is still greater than the preset tolerance, the LM optimization algorithm is used to iterate until the error is reduced to within the tolerance range, completing the posture adjustment.

[0071] Further, in step S6,

[0072] The evaluation system performance specifically includes the following indicators:

[0073] (1) Error convergence: After multiple adjustments, if the total error gradually decreases and tends to converge, it means that the system performance is good. The formula for evaluating error convergence is:

[0074] E current <E previous ;

[0075] Where E current Represents the total error after current adjustment, E previous Indicates the total error of the last adjustment; if the error gradually decreases and meets the convergence condition, it means that the system adjustment instruction is effective;

[0076] (2) Adjustment times: If the number of adjustments exceeds the set threshold, the system performance is considered to be poor and the adjustment parameters need to be optimized;

[0077] (3) Adjustment time: The shorter the time from detecting posture deviation to completing adjustment, the faster the system response and the better the performance. The response time T of each adjustment is recorded. adjust to conduct an assessment;

[0078] The parameter configuration of each module is optimized as follows:

[0079] Damping coefficient λ in LM optimization algorithm;

[0080] The formula for adjusting the damping coefficient is as follows:

[0081]

[0082] Among them, the weight coefficients μ and ν are used to adjust the relative sensitivity of rotation and translation errors, and the adjustment relationship is μ rotation =ν·μ translation .

[0083] Based on the above technical solution, the method proposed by the present invention has the following beneficial effects:

[0084] The method proposed in the present invention realizes spatial positioning and tracking registration of aircraft engine parts based on YOLOv8 and SAM-6D models. By improving the YOLOv8 module and introducing an adaptive weight adjustment mechanism, the detection accuracy of small targets and complex parts is improved; accurate segmentation and pose estimation are achieved through the SAM-6D module, and a confidence scoring system is introduced. Through real-time tracking and spatial registration functions, high-precision positioning and posture adjustment of parts during assembly and maintenance are ensured; in the process of posture adjustment, the LM optimization algorithm and performance evaluation indicators are combined to enhance the reliability and response efficiency of the overall system, providing an efficient and automated solution for assembly and maintenance in complex industrial scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 The present invention provides an overall flow chart for implementing a spatial positioning and tracking registration method for augmented reality assembly guidance of aircraft engine parts. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0087] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.

[0088] like Figure 1 As shown, a spatial positioning and tracking registration method for augmented reality assembly guidance of aircraft engine parts proposed by the present invention is given, which specifically includes:

[0089] S1. Collect images of aircraft engine parts; use cameras to capture multi-view and multi-angle image data to obtain comprehensive part information;

[0090] In this embodiment, a high-resolution camera is used; appropriate exposure time, aperture and focal length are set to optimize image quality; during the assembly or maintenance of aircraft engines, the camera position and angle are set to ensure that all key parts are covered; appropriate lighting conditions are used to eliminate shadows and reflections to improve image visibility and contrast; and multi-view and multi-angle image data are captured to obtain comprehensive part information and enhance the robustness of subsequent processing.

[0091] S2, taking the part image collected in step S1 as input, performing target detection on each part based on the YOLOv8 model, and outputting a two-dimensional bounding box;

[0092] As a preferred implementation, step S2 specifically includes:

[0093] S21. Load the trained weights, configure the model parameters, input the collected part images into the YOLOv8 model and forward propagate. The formula is expressed as:

[0094] f i,j,k =σ(W k ·X i,j +b k );

[0095] Among them, f i,j,k represents the activation value at the i,j coordinate on the feature map, W k is the weight of the kth convolution kernel, X i,j is the local pixel value of the input image, b k is the bias term, σ is the activation function; through forward propagation, the detection results of each part are generated;

[0096] S22, performing threshold filtering on the detection results, eliminating detection frames with confidence levels lower than a set threshold, generating a two-dimensional bounding box for each detected part, and recording the part's category, location coordinates, and comprehensive confidence level.

[0097] In this embodiment, the YOLOv8 model first forward-propagates the input image through the feature extraction network to generate multiple levels of feature maps. The feature map is a high-level feature representation extracted from the input image by a convolutional neural network, which is used to capture information from different parts of the image.

[0098] After generating the feature map, the YOLOv8 model calculates the detection results of each candidate target based on the feature map. These detection results include the probability of target existence, target category, and comprehensive confidence. Each detection result corresponds to a detection box, which is adjusted by regression to generate a two-dimensional bounding box around the target. After the detection box is filtered, the detection boxes with confidence lower than the set threshold are removed, and the retained detection boxes are used as the final two-dimensional bounding boxes. The model also records the category, position coordinates, and comprehensive confidence of the part. The comprehensive confidence is obtained by multiplying the probability of target existence and the probability of target category. The higher the confidence, the more reliable the detection.

[0099] The specific calculation formula of the comprehensive confidence is:

[0100] Confidence=P(Object)×P(Class);

[0101] Among them, Confidence represents confidence information; P(Object) is the probability of the existence of the target, and its value ranges from 0 to 1. The closer it is to 1, the more likely it is that the target exists; P(Class) is the probability that the target belongs to a specific category. A probability value is generated for each category. The higher the category confidence, the more confident the model is that the target belongs to that category.

[0102] More specifically, in the process of target detection in step S2, an adaptive weight adjustment mechanism is introduced to dynamically adjust the weights of the feature pyramid in the YOLOv8 model, specifically:

[0103] Let the output of each layer of the feature pyramid be F n , which contains multiple feature maps, the formula is expressed as:

[0104]

[0105] Among them, n represents the nth layer, and K represents the number of convolution kernels used for this layer.

[0106] Assign adaptive weights α to different layers n ; The adaptive weight α nAccording to the size and shape information of the parts, adjustments are made. The formula is expressed as:

[0107]

[0108] Among them, r n and r shape,n Respectively represent the size ratio and shape ratio of the nth layer; μ r , μs shape is the corresponding mean, σ r , σ shape is the corresponding standard deviation, β is the balance factor, and Z is the normalization constant;

[0109] The calculation formula of the size ratio and shape ratio is:

[0110]

[0111] Among them, A bbox Represents the area of ​​the detection box, A img represents the total area of ​​the image, w bbox Indicates the width of the detection box, h bbox Indicates the height of the detection box.

[0112] By introducing an adaptive weight mechanism, multi-scale feature extraction can be optimized, further improving the accuracy of small target detection and the pose estimation accuracy of complex parts.

[0113] S3, based on the two-dimensional bounding box output in step S2, extract the corresponding area and input it into the SAM-6D model for pixel segmentation to extract the precise contour information of the part; analyze the segmentation results and perform part pose estimation, that is, calculate the actual pose of the part including the three-dimensional position and rotation angle, and output the six-degree-of-freedom 6-DoF pose information;

[0114] As a preferred implementation mode, in step S3, the target part is first segmented using the ISM instance segmentation model, and then the pose estimation model PEM is used to estimate the actual pose of the part through a two-stage part-to-part point matching process, specifically including:

[0115] S31. For each candidate object m that matches the preset CAD model of the part, PEM uses a point matching strategy to predict its 6D pose relative to the CAD model, and represents the segmented candidate object m and the sampling point set of the CAD model as and Where N m and No are the number of points in the point set respectively; and the characteristics of the two point sets are expressed as and C represents the number of channels of the feature;

[0116] S32, ISM equips the point set with a background token, denoted as and Construct the attention matrix A and the allocation matrix The formula is:

[0117] Attention Matrix A:

[0118]

[0119] Allocation Matrix

[0120]

[0121] Among them, Softmax row and Softmax col Respectively represent normalization operations along the rows and columns of the matrix; τ represents a constant; The value of each row in, except the first row, represents the point set P m Each point P m With background and P o The matching score of the midpoint can be found by locating the index of the maximum matching score. m Matched points; constructing the assignment matrix Then we can gather all matching point pairs {(P m ,P o )}, and the corresponding matching score, and finally use weighted SVD to get the actual pose of the part;

[0122] S33. For each pose estimation result, the SAM-6D model generates a confidence score C pose , in order to evaluate the reliability of the pose estimation results.

[0123] In this embodiment, the confidence score C pose Mainly based on the following factors:

[0124] The accuracy of the model in the segmentation process of the part edge, the smaller the difference between the segmented contour and the actual contour, the higher the confidence; the matching degree between the feature points extracted by the model and the actual feature points during the pose estimation process, the higher the matching degree of the feature points, the higher the confidence; the confidence C of the bounding box generated by the YOLOv8 model bbox , which directly affects the initial input of pose estimation.

[0125] Taking these factors into consideration, the confidence score C pose It is expressed as:

[0126] C pose =α·C bbox+β·C seg +γ·C feat ;

[0127] Among them, C bbox is the confidence of the YOLOv8 bounding box, reflecting the model’s confidence in part detection; C seg It is a score based on segmentation accuracy, which measures the difference between the segmented contour and the actual contour; C feat It is a score based on the matching degree of feature points in the pose estimation process; α, β, and γ are weight coefficients used to balance the contribution of various factors;

[0128] When the pose estimation result is output, the system will provide both the pose information and the corresponding confidence score C pose , to evaluate the reliability of the pose estimation results; the confidence score ranges from 0 to 1, where 1 represents the most reliable pose estimation result and 0 represents unreliable.

[0129] S4, roughly comparing the posture information obtained in step S3 with the preset CAD model of the part, and using the LM optimization algorithm to calculate the spatial error between the posture of the part and the CAD model; adjusting the position and posture according to the calculation results to accurately align with the CAD model to complete the three-dimensional registration;

[0130] As a preferred embodiment, step S4 includes:

[0131] S41, expressing the position and posture of the part by a homogeneous transformation matrix, wherein the matrix contains three-dimensional translation and rotation information; specifically expressed as:

[0132]

[0133] Among them, T actual represents the actual pose of the part, that is, the pose estimation result; T CAD Represents the expected pose of the part, that is, the preset CAD model pose; R actual and R CAD are the rotation matrices of the corresponding matrices, p actual and p CAD is the translation vector of the corresponding matrix;

[0134] S42, through the rotation error E between the actual posture and the expected posture rotation and translation error E translation To define the overall error, that is:

[0135] The rotation error is calculated by the difference of the rotation matrix, which is expressed as:

[0136] E rotation =||R actual -R CAD ||F ;

[0137] The translation error is calculated by Euclidean distance, which is expressed as:

[0138] E translation =||p actual -p CAD ||2;

[0139] The total error is the sum of the above two errors, and the formula is expressed as:

[0140] E total =λ1E rotation +λ2E translation ;

[0141] Among them, λ1 and λ2 are weight coefficients used to balance the influence of rotation error and translation error;

[0142] S43, using LM optimization algorithm to minimize the total error E total , to find the optimal posture adjustment ΔT = (ΔR, Δp), where ΔR is the rotation adjustment and Δp is the posture adjustment; so that the actual posture gradually approaches the desired posture.

[0143] More specifically, step S43 includes:

[0144] S431, calculate the Jacobian matrices of the rotation error and the translation error respectively:

[0145]

[0146] The Jacobian matrix of the total error is:

[0147] J=[J rotation ,J translation ];

[0148] S432. At each iteration, the LM algorithm tracks the posture and makes adjustments based on the error and Jacobian matrix. The calculation formula for the adjustment amount ΔT is:

[0149] ΔT=-(J T J+λI) -1 J T r;

[0150] Where r = (T actual -T CAD ), I is the identity matrix, λ is the damping coefficient in the LM algorithm;

[0151] The adjustment ΔT includes the corrections for rotation and translation, and the new pose matrix can be obtained:

[0152]

[0153] Among them, R new =R actual +ΔR,p new =p actual +Δp;

[0154] S433, after each posture adjustment, recalculate the new posture matrix T new The pose T of the CAD model CAD If the error is still greater than the preset tolerance, the LM optimization algorithm is used to iterate until the error is reduced to within the tolerance range, completing the posture adjustment.

[0155] In this embodiment, the tolerance ε is defined as rotation With ε translation They are the error tolerances for rotation and translation respectively. The error tolerance standard is set according to the requirements. The tolerance for rotation error is that the posture jitter between adjacent frames is less than 0.3°, that is, ε rotation <0.3°, the tolerance of translation error is that the displacement jitter between adjacent frames is less than 0.1%, that is, ε translation <0.1%. When satisfied

[0156] E new,rotation <ε rotation, E new,translation <ε translation ;

[0157] The system determines that the pose alignment is complete and stops iterating.

[0158] S5, continuously collect multiple frames of images; and repeat steps S2-S3 to obtain the part posture information of the two consecutive frames; and perform smoothing, compare the position and rotation changes between consecutive frames in real time, and track and monitor the spatial position and posture of the part in real time; if a posture deviation is detected, a feedback signal is generated;

[0159] In this embodiment, step S4 tends to handle posture estimation and three-dimensional alignment in static scenes, and step S5 mainly handles posture tracking and real-time feedback in dynamic scenes to ensure that the parts maintain the correct posture and position during the assembly process.

[0160] S6. According to the tracking results of step S5, a real-time feedback mechanism is established. When feedback information is detected, that is, when posture deviation occurs, an adjustment instruction is issued to control the control system to make timely posture adjustments. At the same time, various parameters in the adjustment process are recorded for subsequent optimization and verification, and the system performance is regularly evaluated to optimize the parameter configuration of each module and improve the accuracy and stability of the overall system spatial positioning and tracking registration.

[0161] As a preferred embodiment, the judgment criterion of the posture deviation in step S6 is determined by comparing the current posture of the part with the preset target posture (i.e., the posture of the preset CAD model); when ΔR>ε rotation And Δp>ε translation , determine the posture deviation;

[0162] When the system detects a rotation error ΔR, it adjusts the posture according to the following formula:

[0163] R new =R actual +ΔR;

[0164] The system will calculate the new R new Used to reset the rotational posture of a part to ensure that the angle of the part is aligned with the target posture.

[0165] When the system detects translation deviation, it adjusts according to the following formula:

[0166] p new =p actual +Δp;

[0167] After the posture adjustment is completed, the system needs to evaluate the performance and whether the system parameters need to be further optimized. The evaluation criteria mainly include the following aspects:

[0168] (1) Error convergence: After multiple adjustments, if the total error gradually decreases and tends to converge, it means that the system performance is good. The formula for evaluating error convergence is:

[0169] E current <E previous ;

[0170] Where E current Represents the total error after current adjustment, E previous Indicates the total error of the last adjustment; if the error gradually decreases and meets the convergence condition, it means that the system adjustment instruction is effective;

[0171] (2) Adjustment times: If the adjustment times exceed a set threshold, which is set to 5 in this embodiment, the system performance is considered to be poor and the adjustment parameters need to be optimized;

[0172] (3) Adjustment time: The shorter the time from detecting posture deviation to completing adjustment, the faster the system response and the better the performance. The response time T of each adjustment is recorded. adjust To evaluate; in this embodiment, T adjust <20ms is considered good performance.

[0173] The parameter configuration of each module is optimized as follows:

[0174] Damping coefficient λ in LM optimization algorithm;

[0175] The formula for adjusting the damping coefficient is as follows:

[0176]

[0177] Among them, the weight coefficients μ and ν are used to adjust the relative sensitivity of rotation and translation errors, and the adjustment relationship is μ rotation =ν·μ translation .

[0178] After the parameters are adjusted, the system will verify the adjusted performance again through experiments. The adjustment time, number of adjustments and error convergence will also be recorded to evaluate the optimization effect until the system converges.

[0179] In summary, the method proposed in the present invention solves the problems of difficult target detection, low pose estimation accuracy and poor real-time performance in complex scenes in existing methods by improving the YOLOv8 and SAM-6D models and combining confidence scoring and real-time evaluation optimization systems, and realizes efficient detection, precise segmentation, spatial position and posture tracking of engine parts under multiple degrees of freedom.

[0180] It should be noted that it is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be regarded as exemplary and non-restrictive from any point of view, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any reference numerals in the claims should not be regarded as limiting the claims involved.

[0181] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A spatial positioning and tracking registration method for augmented reality assembly guidance of aircraft engine parts, characterized in that: The specific steps include: S1. Collect images of aircraft engine parts; use cameras to capture multi-view and multi-angle image data to obtain comprehensive part information; S2, taking the part image collected in step S1 as input, performing target detection on each part based on the YOLOv8 model, and outputting a two-dimensional bounding box; S3, based on the two-dimensional bounding box output in step S2, extract the corresponding area of ​​the detected part and input it into the SAM-6D model for pixel segmentation to extract the precise contour information of the part; analyze the segmentation results and perform part pose estimation, that is, calculate the actual pose of the part including the three-dimensional position and rotation angle, and output the six-degree-of-freedom 6-DoF pose information; S4, roughly comparing the posture information obtained in step S3 with the preset CAD model of the part, and using the LM optimization algorithm to calculate the spatial error between the posture of the part and the CAD model; adjusting the position and posture according to the calculation results to accurately align with the CAD model to complete the three-dimensional registration; S5, continuously collect multiple frames of images; and repeat steps S2-S3 to obtain the part posture information of the two consecutive frames; and perform smoothing, compare the position and rotation changes between consecutive frames in real time, and track and monitor the spatial position and posture of the part in real time; if a posture deviation is detected, a feedback signal is generated; S6. According to the tracking results of step S5, a real-time feedback mechanism is established. When feedback information is detected, that is, when posture deviation occurs, an adjustment instruction is issued to control the control system to make timely posture adjustments. At the same time, various parameters in the adjustment process are recorded for subsequent optimization and verification, and the system performance is regularly evaluated to optimize the parameter configuration of each module and improve the accuracy and stability of the overall system spatial positioning and tracking registration.

2. The spatial positioning and tracking registration method for augmented reality assembly guidance of aircraft engine parts according to claim 1 is characterized in that: Step S1 specifically includes: The camera is a high-resolution camera; and the position and angle of the camera are set to cover all key parts, ensuring that there are no shadows and reflections in the lighting conditions.

3. The spatial positioning and tracking registration method for augmented reality assembly guidance of aircraft engine parts according to claim 1 is characterized in that: The target detection in step S2 uses the feature pyramid network structure of the YOLOv8 model, specifically: S21. Load the trained weights, configure the model parameters, input the collected part images into the YOLOv8 model and forward propagate. The formula is expressed as: f i,j,k =sigmoid(W k ·X i,j +b k ); Among them, f i,j,k represents the activation value at the i,j coordinate on the feature map, W k is the weight of the kth convolution kernel, X i,j is the local pixel value of the input image, b k is a bias term, sigmoid is an activation function; through forward propagation, the detection result of each part is generated; the weight of the convolution kernel is learned through the training process; S22, performing threshold filtering on the detection results, eliminating detection frames with confidence levels lower than a set threshold, generating a two-dimensional bounding box for each detected part, and recording the part's category, location coordinates, and comprehensive confidence level.

4. The spatial positioning and tracking registration method for augmented reality assembly guidance of aircraft engine parts according to claim 3 is characterized in that: In the process of target detection in step S2, an adaptive weight adjustment mechanism is also introduced to dynamically adjust the weights of the feature pyramid in the YOLOv8 model, specifically: Let the output of each layer of the feature pyramid be F n , which contains multiple feature maps, the formula is expressed as: Among them, n represents the nth layer, and K represents the number of convolution kernels used in this layer; Assign adaptive weights α to different layers n ; The adaptive weight α n According to the size and shape information of the parts, adjustments are made. The formula is expressed as: Among them, r n and r shape,n Respectively represent the size ratio and shape ratio of the nth layer; μ r , μ shape is the corresponding mean, σ r , σ shape is the corresponding standard deviation, β is the balance factor, and Z is the normalization constant; The calculation formula of the size ratio and shape ratio is: Among them, A bbox Represents the area of ​​the detection box, A img represents the total area of ​​the image, w bbox Indicates the width of the detection box, h bbox Indicates the height of the detection box; Based on the adaptive weights assigned above, the importance of each layer of feature maps is dynamically adjusted, and the final feature pyramid network output is:

5. The spatial positioning and tracking registration method for augmented reality assembly guidance of aircraft engine parts according to claim 3 is characterized in that: The specific calculation formula of the comprehensive confidence in step S22 is: Confidence=P(Object)×P(Class); Among them, Confidence represents the comprehensive confidence; P(Object) is the probability of the existence of the target, which takes a value between 0 and 1. The closer it is to 1, the more likely it is that the target exists; P(Class) is the probability that the target belongs to a specific category. A probability value is generated for each category. The higher the category confidence, the more confident the model is that the target belongs to that category.

6. The spatial positioning and tracking registration method for augmented reality assembly guidance of aircraft engine parts according to claim 1, characterized in that: In step S3, the target part is first segmented using the ISM instance segmentation model, and then the pose estimation model PEM is used to estimate the actual pose of the part through a two-stage part-to-part point matching process, which specifically includes: S31. For each candidate object m that matches the preset CAD model of the part, PEM uses a point matching strategy to predict its 6D pose relative to the CAD model, and represents the segmented candidate object m and the sampling point set of the CAD model as and Where N m and No are the number of points in the point set respectively; and the characteristics of the two point sets are expressed as and C represents the number of channels of the feature; S32, ISM equips the point set with a background token, denoted as and Construct the attention matrix A and the allocation matrix The formula is: Attention matrix A: Allocation Matrix Among them, Softmax row and Softmax col Respectively represent normalization operations along the rows and columns of the matrix; t represents a constant; The value of each row in, except the first row, represents the point set P m Each point P m With background and P o The matching score of the midpoint can be found by locating the index of the maximum matching score. m Matched points; constructing the assignment matrix Then we can gather all matching point pairs {(P m ,P o )}, and the corresponding matching score, and finally use weighted SVD to get the actual pose of the part; S33. For each pose estimation result, the SAM-6D model generates a confidence score C pose , in order to evaluate the reliability of the pose estimation results.

7. The spatial positioning and tracking registration method for augmented reality assembly guidance of aircraft engine parts according to claim 6, characterized in that: Step S33 is specifically as follows: The confidence score C pose It is expressed as: C pose =α·C bbox +β·C seg +γ·C feat ; Among them, C bbox is the confidence of the YOLOv8 bounding box, reflecting the model’s confidence in part detection; C seg It is a score based on segmentation accuracy, which measures the difference between the segmented contour and the actual contour; C feat It is a score based on the matching degree of feature points in the pose estimation process; α, β, and γ are weight coefficients used to balance the contribution of various factors; When the pose estimation result is output, the system will provide both the pose information and the corresponding confidence score C pose , to evaluate the reliability of the pose estimation results; the confidence score ranges from 0 to 1, where 1 represents the most reliable pose estimation result and 0 represents unreliable.

8. The spatial positioning and tracking registration method for augmented reality assembly guidance of aircraft engine parts according to claim 1, characterized in that: Step S4 specifically includes: S41, expressing the position and posture of the part by a homogeneous transformation matrix, wherein the matrix contains three-dimensional translation and rotation information; specifically expressed as: Among them, T actual represents the actual pose of the part, that is, the pose estimation result; T CAD Represents the expected pose of the part, that is, the preset CAD model pose; R actual and R CAD are the rotation matrices of the corresponding matrices, p actual and p CAD is the translation vector of the corresponding matrix; S42, through the rotation error E between the actual posture and the expected posture rotation and translation error E translation To define the overall error, that is: The rotation error is calculated by the difference of the rotation matrix, which is expressed as: E rotation =‖R actual -R CAD ‖ F ; The translation error is calculated by Euclidean distance, which is expressed as: E translation =||p actual -p CAD ||2; The total error is the sum of the above two errors, and the formula is expressed as: AND total =l1E rotation +l2E translation ; Among them, λ1 and λ2 are weight coefficients used to balance the influence of rotation error and translation error; S43, using LM optimization algorithm to minimize the total error E total , to find the optimal posture adjustment ΔT = (ΔR, Δp), where ΔR is the rotation adjustment and Δp is the posture adjustment; so that the actual posture gradually approaches the desired posture.

9. The spatial positioning and tracking registration method for augmented reality assembly guidance of aircraft engine parts according to claim 8, characterized in that: Step S43 uses the LM optimization algorithm to minimize the total error E total Specifically include: S431, calculate the Jacobian matrices of the rotation error and the translation error respectively: The Jacobian matrix of the total error is: J=[J rotation ,J translation ]; S432. At each iteration, the LM algorithm tracks the posture and makes adjustments based on the error and Jacobian matrix. The calculation formula for the adjustment amount ΔT is: ΔT=-(J T J+lI) -1 J T r; Where r = (T actual -T CAD ), I is the identity matrix, λ is the damping coefficient in the LM algorithm; The adjustment ΔT includes the correction of rotation and translation, and the new pose matrix can be obtained: Among them, R new =R actual +ΔR,p new =p actual +Δp; S433, after each posture adjustment, recalculate the new posture matrix T new The pose T of the CAD model CAD If the error is still greater than the preset tolerance, the LM optimization algorithm is used to iterate until the error is reduced to within the tolerance range, completing the posture adjustment.

10. The spatial positioning and tracking registration method for augmented reality assembly guidance of aircraft engine parts according to claim 8, characterized in that: In step S6, The evaluation system performance specifically includes the following indicators: (1) Error convergence: After multiple adjustments, if the total error gradually decreases and tends to converge, it means that the system performance is good. The formula for evaluating error convergence is: AND current <And previous ; Where E current Represents the total error after current adjustment, E previous Indicates the total error of the last adjustment; if the error gradually decreases and meets the convergence condition, it means that the system adjustment instruction is effective; (2) Adjustment times: If the number of adjustments exceeds the set threshold, the system performance is considered to be poor and the adjustment parameters need to be optimized; (3) Adjustment time: The shorter the time from detecting posture deviation to completing adjustment, the faster the system response and the better the performance. The response time T of each adjustment is recorded. adjust to conduct an assessment; The parameter configuration of each module is optimized as follows: Damping coefficient λ in LM optimization algorithm; The formula for adjusting the damping coefficient is as follows: Among them, the weight coefficients μ and v are used to adjust the relative sensitivity of rotation and translation errors, and the adjustment relationship is μ rotation =v·μ translation .

Images