Mobile assembly guiding and detecting system based on optimized ORB algorithm

Abstract

The invention discloses a mobile assembly guide and detection system based on an optimized ORB algorithm. The system comprises an image acquisition and preprocessing module, an optimized ORB feature detection module, a multi-level feature matching module, an AR guide module, a knowledge graph construction and application module, a predictive maintenance module and a system integration and data management module. According to the system, feature detection is optimized through adaptive FAST corner detection and an improved rotating BRIEF descriptor algorithm, the accuracy is improved by adopting a multi-level matching strategy, and stable AR guidance is realized by utilizing a vision-inertia fusion technology. And in combination with a knowledge graph and a machine learning technology, intelligent assembly process management and equipment health monitoring are supported. The method is suitable for precise assembly and equipment maintenance in a complex industrial environment, and the assembly efficiency and precision are improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing, and in particular to a mobile assembly guidance and detection system based on an optimized ORB (Oriented FAST and Rotated BRIEF) algorithm. Background Art

[0002] With the development of Industry 4.0 and intelligent manufacturing, the assembly and maintenance of complex mechanical systems have increasingly higher requirements for accuracy, efficiency, and intelligence. Traditional manual assembly methods have problems such as low efficiency, prone to errors, and difficulty in standardization. Existing assembly guidance systems based on computer vision have the following main problems:

[0003] The computational complexity is high and it is difficult to run in real time on mobile devices;

[0004] Poor adaptability to complex industrial environments and susceptible to changes in lighting and shading;

[0005] Lack of intuitive and efficient augmented reality (AR) guidance interface;

[0006] The integration with the enterprise's existing information system is low, making it difficult to achieve real-time synchronization and analysis of data.

[0007] Therefore, there is an urgent need for an intelligent assembly system that can run efficiently on mobile devices, accurately identify complex scenes, provide real-time AR guidance, and seamlessly integrate with enterprise information systems. Summary of the invention

[0008] To solve the above problems, the present invention proposes a mobile assembly guidance and detection system based on an optimized ORB algorithm, comprising the following modules:

[0009] Image acquisition and preprocessing module: This module uses adaptive exposure control and high dynamic range (HDR) imaging technology to achieve high-quality image acquisition in complex lighting environments; convolutional neural network (DnCNN) is used for denoising to ensure the accuracy of subsequent image processing and improve the system's adaptability in complex environments.

[0010] Optimized ORB feature detection module: Adaptive FAST corner detection and scale space construction are used to improve the efficiency and accuracy of feature point detection; the improved rotation BRIEF descriptor algorithm enhances the robustness of matching, enabling the module to run efficiently on mobile devices with limited resources.

[0011] Multi-level feature matching module: This module achieves efficient coarse matching through locality sensitive hashing (LSH), and uses PROSAC geometric verification and Lucas-Kanade optical flow algorithm for precise adjustment; combined with graph cut optimization technology, it ensures global consistency and adapts to the precise matching requirements under multi-view and complex scenes.

[0012] Augmented reality (AR) guidance module: It uses visual-inertial fusion technology for posture tracking, and uses spherical harmonic functions to perform lighting estimation and depth map occlusion processing to achieve virtual and real AR content superposition; the adaptive transparency adjustment function enhances the user's visual experience and provides more intuitive real-time assembly guidance.

[0013] Knowledge graph construction and application module: Through OWL ontology modeling and Markov logic network reasoning, the system can automatically identify decision points and potential faults in the assembly process; this module supports the automated management and optimization of intelligent assembly processes, improving assembly efficiency and accuracy.

[0014] Predictive maintenance module: This module uses the long short-term memory network (LSTM) to monitor equipment health and predict remaining service life, and combines the isolation forest algorithm to accurately detect equipment anomalies; through the fusion analysis of historical data and real-time data, early warning of equipment maintenance is achieved to reduce unplanned downtime.

[0015] System Integration and Data Management Module: Data is exchanged with the enterprise's MES and ERP systems through standardized RESTful API interfaces; real-time data processing and synchronization are achieved through the Apache Spark-based data analysis pipeline; Redis distributed cache is used to optimize data access performance and ensure system response efficiency under large-scale data; at the same time, data security is ensured through OAuth 2.0 authentication and HTTPS encrypted communication.

[0016] Specifically, the system's workflow is as follows:

[0017] S1. Image acquisition and preprocessing: The system uses adaptive exposure technology to dynamically adjust exposure parameters according to scene brightness to ensure that the image is neither overexposed nor underexposed. In scenes with complex lighting conditions, HDR images are synthesized through multiple exposures to increase the dynamic range of the image. Image denoising uses a denoising algorithm based on a convolutional neural network (DnCNN) to remove noise from the image, and then uses an adaptive histogram equalization (CLAHE) algorithm to improve the image contrast to enhance the image.

[0018] Furthermore, the denoising function is:

[0019]

[0020] Where y is the input noise image, is the denoised image, F(y,Θ) represents the residual learning function, Θ is the network parameter;

[0021] Further, the image enhancement method is:

[0022]

[0023] Among them, h i is the i-th interval value of the histogram, N is the total number of intervals, and α is the adjustment factor, which is usually between 3 and 4.

[0024] S2. Feature detection: The optimized ORB algorithm detects feature points in the image and calculates the rotated BRIEF descriptor.

[0025] Further, the calculation process of the ORB algorithm is:

[0026] A). Adaptive FAST corner detection

[0027] a). Calculate the local contrast C(p):

[0028] C(p)=(max(I N (p))-min(I N (p))) / (max(I N (p)))+min(I N (p)))+ε)

[0029] Where:

[0030] I N (p) represents the N×N region of pixel p (typical value N=9), and ε is a small positive number (typical value is 10 -6 ), used to avoid the denominator being zero;

[0031] b). Calculate the adaptive threshold T(p)

[0032] T(p)=T base ·(1+α·(C(p)-μ c )) / σ c

[0033] Where:

[0034] T base is the basic threshold (default value is 16), α is the adjustment factor, μ c , σ c are the mean and standard deviation of the contrast of the entire image, respectively.

[0035] B) Scale space construction: Gaussian pyramid is used to construct the scale space of the image, with a total of L layers (typical value L = 8). The scale σ of the mth layerm Defined as:

[0036] l / S

[0037] σ l =σ0·2

[0038] Among them, σ0 is the basic scale (taken as 1.0), and S is the number of layers in each group (usually 3).

[0039] C). Feature point direction allocation:

[0040] The orientation of the feature point is assigned by calculating the main orientation θ of the feature point:

[0041]

[0042] Where:

[0043] mx is the x-direction gradient of the pixels in the feature point area;

[0044] my is the y-direction gradient of the pixels in the feature point area;

[0045] D). rBRIEF descriptor calculation: Based on the main direction θ rotation sampling pattern, calculate the binary descriptor. The i-th bit of the descriptor D is defined as:

[0046]

[0047] Among them, (p', q') is the sampling point pair after rotation, and the rotation transformation formula is:

[0048]

[0049] S3. Feature matching: Through a multi-level matching strategy, rough matching is performed first and then precise adjustment is performed to ensure the matching accuracy of multiple perspectives. The specific operations are as follows:

[0050] A). Locality Sensitive Hashing (LSH) coarse matching: Use LSH to map high-dimensional binary descriptors to low-dimensional space. The hash function is defined as:

[0051]

[0052] Where:

[0053] ω i is a random vector, b i is a random bias. K hash tables (typical value K = 10) are constructed to improve the recall rate.

[0054] B). RANSAC geometry verification: Use PROSAC variant for geometry verification and define the error function as follows:

[0055] E(H)=∑i ρ(d(Hx i , x' i ))

[0056] Where: H is the identity matrix, d(Hx i , x' i ) is the geometric distance, ρ is the Huber loss function

[0057] C). Accurate positioning of optical flow

[0058] The initial match is accurately positioned using the pyramid Lucas-Kanade optical flow algorithm. The solution equation is as follows:

[0059]

[0060] in:

[0061] Ix represents the horizontal gradient of the pixel in the image;

[0062] Iy represents the vertical gradient of the pixel in the image;

[0063] It represents the temporal gradient of the pixel in the image;

[0064] Δx and Δy are the displacements in the horizontal and vertical directions, that is, the optical flow values ​​to be solved.

[0065] D). Global consistency optimization

[0066] Construct a matching graph G = (V, E) and define the global consistency energy function E(G):

[0067]

[0068] Where:

[0069] E data Indicates the local matching quality;

[0070] E smooth Indicates the consistency of adjacent matches;

[0071] λ is the balance factor (typical value 0.1);

[0072] The optimal matching is solved by the graph cut algorithm.

[0073] S4. Augmented reality guidance: Based on visual-inertial tracking and adaptive AR content rendering, augmented reality overlay information is generated in real time to guide assembly.

[0074] Furthermore, the visual-inertial fusion posture tracking adopts a tightly coupled visual-inertial fusion method combined with an extended Kalman filter (EKF) to achieve stable posture tracking. The state vector X is defined as follows:

[0075] X=[p,v,q,b a ,b g ] T

[0076] Among them, p is the position, v is the velocity, q is the rotation represented by the quaternion, and b a and b g are the biases of the speedometer and gyroscope, respectively.

[0077] Furthermore, the prediction step of X is:

[0078] X k|k-1 =f(X k-1|k-1 ,u k )

[0079]

[0080] in:

[0081] X k|k-1 represents the state vector predicted based on the k-1 state at time k;

[0082] f function is the nonlinear state transfer function, P k|k-1 is the predicted state covariance matrix, P k-1|k-1 According to the state X at the last moment k|k-1 and the current input u k To predict the current state, Q is the state covariance matrix of the previous moment. k is the process noise covariance matrix, which represents the uncertainty of the system during the state transition process.

[0083] Furthermore, the update steps of X are:

[0084]

[0085] Among them, K k is the Kalman gain, P k|k-1 is the predicted state covariance matrix; H k is the Jacobian matrix, which describes the derivative of the observation function with respect to the state vector, R k is the observation noise covariance matrix, which represents the uncertainty in the observation process.

[0086] X k|k =X k|k-1 +K k (Z k -h(X k|k-1 ))

[0087] P k|k =(IK k Hk ) k|k-1

[0088] Among them, X k|k is the updated state vector, X k|k-1 is the predicted state vector, Z k is the observation value at the current moment, h function is a nonlinear observation function, which maps the state vector to the observation space, P k|k is the updated state covariance matrix, and I is the identity matrix.

[0089] Furthermore, the illumination estimation is calculated as follows:

[0090] Use spherical harmonics to estimate ambient lighting L(ω):

[0091]

[0092] in, is the spherical harmonic basis function, are the spherical harmonic coefficients (usually N=2, a total of 9 coefficients).

[0093] Furthermore, the adaptive AR content rendering includes:

[0094] A). Lighting estimation

[0095] Use spherical harmonics to estimate ambient lighting L(ω):

[0096]

[0097] in, is the spherical harmonic basis function, are the spherical harmonic coefficients (usually N=2, a total of 9 coefficients).

[0098] B). Occlusion processing

[0099] Combining depth map and edge detection for occlusion reasoning, occlusion probability P occl (x) is defined as:

[0100] P occl (x) = σ(α(D rl (x)-D v (x)+βE(x)))

[0101] Among them, D rl (x), D v (x) represents the depth of the real object and the virtual object respectively, E(x) is the edge strength, α and β are weight coefficients, and σ is the Sigmoid activation function.

[0102] C). Adaptive transparency

[0103] The transparency of the AR content is dynamically adjusted by the angle between the line of sight and the normal line of the object surface. The transparency formula is as follows:

[0104] α(x)=α base (1-γcosθ)(1-δC(x))

[0105] Among them, θ is the angle between the line of sight and the surface normal, C(x) is the local complexity measure, γ and δ are adjustment factors, and α base Base transparency (usually 0.8).

[0106] S5. Knowledge graph and intelligent decision-making: The system supports intelligent fault diagnosis and assembly decision-making through knowledge graph.

[0107] Furthermore, the construction process of the knowledge graph is as follows:

[0108] A). Ontology modeling: Use Web Ontology Language (OWL) to define the ontology of the assembly domain. The core categories include: parts, (Tool), operations, faults, and assembly steps.

[0109] B). Knowledge extraction: Combine rules and statistical methods to extract entities and relationships from text, images, and sensor data. The extraction confidence is calculated as follows:

[0110] conf(e,r,e')=freq(e,r,e') / freaq(e)·freq(e')

[0111] Among them, e and e' are entities, r is a relationship, and freq is a frequency function.

[0112] C) Knowledge reasoning: The reasoning is performed by combining description logic reasoning and Markov logic network (MLN). The probability of query q is defined as:

[0113]

[0114] Where:

[0115] Z is a normalization factor used to ensure that the probability value is within a reasonable range. Its function is to make the sum of the probabilities of all possible situations equal to 1;

[0116] ω is a possible world state, that is, a combination of various possible situations;

[0117] q is the query, i.e. the specific event or situation whose probability we want to determine;

[0118] E stands for evidence, i.e. known information or conditions;

[0119] A is {ω∈Ω|ω satisfies query q and satisfies evidence E};

[0120] ω i is the rule weight, which indicates the importance of different rules in reasoning;

[0121] n i (ω) represents the number of times the rule is satisfied in the world state, that is, it counts the number of times a specific rule is satisfied in a specific world state.

[0122] D). Visual query: Based on the query interface provided by SPARQL, it supports complex multi-hop relationship queries and path analysis.

[0123] S6. Predictive maintenance: Based on real-time data, equipment health assessment and maintenance prediction are performed. Further, the health assessment and maintenance prediction include the following steps:

[0124] A). Feature Engineering:

[0125] Combining the time domain and frequency domain features, the equipment health index HI is constructed. The formula is as follows:

[0126]

[0127] Among them, RMS is the root mean square value of the equipment vibration signal in the time domain; Kt is the kurtosis, which indicates the peak degree of the vibration signal amplitude distribution; PSD is the power spectral density, which refers to the energy distribution of the signal in the frequency domain, calculated as the normalized value of the square of the Fourier transform of the signal; Ent is the information entropy, which indicates the uncertainty of the signal; is the weight coefficient, which is determined by cross-validation and satisfies the sum of 1.

[0128] B). Anomaly Detection:

[0129] The isolation forest algorithm is used to detect abnormal data points, and the anomaly score s is defined as:

[0130] s(x,n)=2 -E(h(x)) / c(n)

[0131] in:

[0132] x is the data point to be evaluated, represented as a feature vector containing the RMS, Kt, PSD, and Ent calculated above;

[0133] h(x) is the average path length of data point x, that is, the average path length from the tree root x to the included leaf nodes;

[0134] E(h(x)) is the expected value of h(x) on all trees in the forest;

[0135] n is the number of samples used to train the isolation forest;

[0136] c(n)=2H(n-1)-(2(n-1) / n), where H(i) is the i-th harmonic number, defined as H(i)=ln(i)+Euler's constant;

[0137] The anomaly score s ranges from (0,1], and the closer it is to 1, the more likely it is an anomaly point.

[0138] C). Remaining useful life RUL prediction:

[0139] Long short-term memory network (LSTM) is used for time series prediction, and the loss function is defined as:

[0140] L = MSE (RUL pred ,RUL true )+λ·KL(q(z|X)|p(z))

[0141] in:

[0142] RUL pred The remaining useful life predicted by the model;

[0143] RUL true is the actual remaining useful life;

[0144] MSE is mean square error;

[0145] KL(q(z|X)|p(z)) is the KL divergence, which is used to regularize the difference between the posterior distribution q(z|X) and the prior distribution p(z) of the latent variable z;

[0146] X is the input time series data, including the historical health index HI;

[0147] λ is a balancing factor used to adjust the relative importance of MSE loss and KL divergence.

[0148] D). Decision support:

[0149] Based on the cost-sensitive decision model, taking into account the maintenance cost and downtime loss, the decision formula is:

[0150] Action = argmin a [C m (a)+P f (a) C f ]

[0151] in:

[0152] argmin a Represents the value of the independent variable that makes the function value reach the minimum value;

[0153] a indicates possible maintenance actions, such as "immediate maintenance", "planned maintenance", "continue operation", etc.;

[0154] C m (a) represents the cost of maintenance action a, including labor cost, spare parts cost, etc.;

[0155] P f (a) represents the failure probability after taking action a, estimated based on the RUL prediction results and historical data;

[0156] C f Represents the failure cost, including downtime loss, emergency repair cost, possible safety accident cost, etc.

[0157] The decision model selects the action that minimizes the expected total cost as the optimal decision.

[0158] S7. Data integration and management: Connect with enterprise information systems through standardized interfaces to achieve data synchronization and security management. The specific methods are:

[0159] A microservice architecture based on Docker and Kubernetes is used to ensure the scalability and maintainability of the system. The core services include: image processing service, AR rendering service, knowledge graph service, predictive maintenance service, and data synchronization service. Redis is used to implement distributed caching to optimize data access performance. The cache strategy adopts the LRU (Least Recently Used) algorithm, and the cache expiration time is set to 15 minutes. OAuth 2.0 is used for identity authentication and authorization, and JSON Web Token (JWT) is used to manage user sessions. All API communications are encrypted through HTTPS, and the system implements a fine-grained access control strategy to ensure data security.

[0160] The beneficial effects of the present invention include:

[0161] 1. Efficient image feature detection and matching: By optimizing the adaptive FAST corner detection and rBRIEF descriptor calculation method in the ORB algorithm, the problem of high computational complexity of the traditional ORB algorithm and difficulty in real-time operation on mobile devices is solved. The present invention realizes real-time and efficient image feature detection and matching on mobile devices, significantly improving the performance of the system during the assembly process.

[0162] 2. Enhanced adaptability to complex environments: The introduction of adaptive exposure control and high dynamic range (HDR) imaging technology enables the system to cope with changes in different lighting conditions. At the same time, the use of occlusion reasoning technology based on depth maps enables the system to maintain high recognition accuracy when facing component occlusion. This innovation improves the system's adaptability in complex industrial environments.

[0163] 3. Intuitive augmented reality (AR) guidance interface: Through visual-inertial fusion posture tracking technology and adaptive transparency adjustment technology, the system can provide assemblers with a stable and intuitive AR content overlay experience. Guidance information based on real-time data and equipment status can effectively improve the accuracy of the assembly process and user experience.

[0164] 4. Intelligent knowledge graph support: The system innovatively constructs a knowledge graph in the field of assembly, and combines descriptive logic reasoning and Markov logic networks to support intelligent decision-making and automatic fault diagnosis in the assembly process. This technological breakthrough reduces manual intervention in the assembly process and achieves a higher level of intelligence.

[0165] 5. Predictive maintenance function: By integrating a predictive maintenance module based on machine learning, the system can analyze the health status of the equipment and predict the remaining useful life (RUL), thereby achieving early warning of potential failures and maintenance optimization. This function significantly reduces the unplanned downtime of the equipment and improves the overall efficiency of the production line.

[0166] 6. Modularity and scalability: The system adopts modular design and supports flexible expansion. Through the standardized RESTful API interface, the system can seamlessly integrate the company's existing MES, ERP and other information systems, and support the rapid integration of new algorithms and functional modules, enhancing the scalability and maintainability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0167] Figure 1 A schematic diagram of the architecture of the system of the present invention is shown;

[0168] Figure 2 A schematic diagram of the working process of the system of the present invention is shown. DETAILED DESCRIPTION

[0169] Combination Figure 1 , the present invention proposes a mobile assembly guidance and detection system based on the optimized ORB algorithm, comprising the following modules:

[0170] Image acquisition and preprocessing module: This module uses adaptive exposure control and high dynamic range (HDR) imaging technology to achieve high-quality image acquisition in complex lighting environments; convolutional neural network (DnCNN) is used for denoising to ensure the accuracy of subsequent image processing and improve the system's adaptability in complex environments.

[0171] Optimized ORB feature detection module: Adaptive FAST corner detection and scale space construction are used to improve the efficiency and accuracy of feature point detection; the improved rotation BRIEF descriptor algorithm enhances the robustness of matching, enabling the module to run efficiently on mobile devices with limited resources.

[0172] Multi-level feature matching module: This module achieves efficient coarse matching through locality sensitive hashing (LSH), and uses PROSAC geometric verification and Lucas-Kanade optical flow algorithm for precise adjustment; combined with graph cut optimization technology, it ensures global consistency and adapts to the precise matching requirements under multi-view and complex scenes.

[0173] Augmented reality (AR) guidance module: It uses visual-inertial fusion technology for posture tracking, and uses spherical harmonic functions to perform lighting estimation and depth map occlusion processing to achieve virtual and real AR content superposition; the adaptive transparency adjustment function enhances the user's visual experience and provides more intuitive real-time assembly guidance.

[0174] Knowledge graph construction and application module: Through OWL ontology modeling and Markov logic network reasoning, the system can automatically identify decision points and potential faults in the assembly process; this module supports the automated management and optimization of intelligent assembly processes, improving assembly efficiency and accuracy.

[0175] Predictive maintenance module: This module uses the long short-term memory network (LSTM) to monitor equipment health and predict remaining service life, and combines the isolation forest algorithm to accurately detect equipment anomalies; through the fusion analysis of historical data and real-time data, early warning of equipment maintenance is achieved to reduce unplanned downtime.

[0176] System Integration and Data Management Module: Data is exchanged with the enterprise's MES and ERP systems through standardized RESTful API interfaces; real-time data processing and synchronization are achieved through the Apache Spark-based data analysis pipeline; Redis distributed cache is used to optimize data access performance and ensure system response efficiency under large-scale data; at the same time, data security is ensured through OAuth 2.0 authentication and HTTPS encrypted communication.

[0177] Combination Figure 2 , the system workflow is as follows:

[0178] S1. Image acquisition and preprocessing: The system uses adaptive exposure technology to dynamically adjust exposure parameters according to scene brightness to ensure that the image is neither overexposed nor underexposed. In scenes with complex lighting conditions, HDR images are synthesized through multiple exposures to increase the dynamic range of the image. Image denoising uses a denoising algorithm based on a convolutional neural network (DnCNN) to remove noise from the image, and then uses an adaptive histogram equalization (CLAHE) algorithm to improve the image contrast to enhance the image.

[0179] Furthermore, the denoising function is:

[0180]

[0181] Where y is the input noise image, is the denoised image, F(y,Θ) represents the residual learning function, Θ is the network parameter;

[0182] Further, the image enhancement method is:

[0183]

[0184] Among them, h i is the i-th interval value of the histogram, N is the total number of intervals, and α is the adjustment factor, which is usually between 3 and 4.

[0185] S2. Feature detection: The optimized ORB algorithm detects feature points in the image and calculates the rotated BRIEF descriptor.

[0186] Further, the calculation process of the ORB algorithm is:

[0187] A). Adaptive FAST corner detection

[0188] a). Calculate the local contrast C(p):

[0189] C(p)=(max(I N (p))-min(I N (p))) / (max(I N (p)))+min(I N (p)))+ε)

[0190] Where:

[0191] I N (p) represents the N×N region of pixel p (typical value N=9), and ε is a small positive number (typical value is 10 -6 ), used to avoid the denominator being zero;

[0192] b). Calculate the adaptive threshold T(p)

[0193] T(p)=T base ·(1+α·(C(p)-μ c )) / σ c

[0194] Where:

[0195] T base is the basic threshold (default value is 16), α is the adjustment factor, μ c , σ c are the mean and standard deviation of the contrast of the entire image, respectively.

[0196] B) Scale space construction: Gaussian pyramid is used to construct the scale space of the image, with a total of L layers (typical value L = 8). The scale σ of the mth layer m Defined as:

[0197] l / S

[0198] σ l =σ0·2

[0199] Among them, σ0 is the basic scale (taken as 1.0), and S is the number of layers in each group (usually 3).

[0200] C). Feature point direction allocation:

[0201] The orientation of the feature point is assigned by calculating the main orientation θ of the feature point:

[0202]

[0203] Where:

[0204] mx is the x-direction gradient of the pixels in the feature point area;

[0205] my is the y-direction gradient of the pixels in the feature point area;

[0206] D). rBRIEF descriptor calculation: Based on the main direction θ rotation sampling pattern, calculate the binary descriptor. The i-th bit of the descriptor D is defined as:

[0207]

[0208] Among them, (p', q') is the sampling point pair after rotation, and the rotation transformation formula is:

[0209]

[0210] S3. Feature matching: Through a multi-level matching strategy, rough matching is performed first and then precise adjustment is performed to ensure the matching accuracy of multiple perspectives. The specific operations are as follows:

[0211] A. Locality Sensitive Hashing (LSH) coarse matching: Use LSH to map high-dimensional binary descriptors to low-dimensional space. The hash function is defined as:

[0212]

[0213] Where:

[0214] ω i is a random vector, b i is a random bias. K hash tables (typical value K = 10) are constructed to improve the recall rate.

[0215] B.RANSAC geometry verification: Use PROSAC variant for geometry verification, and define the error function as follows:

[0216] E(H)=∑ i ρ(d(Hx i , x' i ))

[0217] Where: H is the identity matrix, d(Hx i , x' i ) is the geometric distance, ρ is the Huber loss function

[0218] C). Accurate positioning of optical flow

[0219] The initial match is accurately positioned using the pyramid Lucas-Kanade optical flow algorithm. The solution equation is as follows:

[0220]

[0221] in:

[0222] Ix represents the horizontal gradient of the pixel in the image;

[0223] Iy represents the vertical gradient of the pixel in the image;

[0224] It represents the temporal gradient of the pixel in the image;

[0225] Δx and Δy are the displacements in the horizontal and vertical directions, that is, the optical flow values ​​to be solved.

[0226] D). Global consistency optimization

[0227] Construct a matching graph G = (V, E) and define the global consistency energy function E(G):

[0228]

[0229] Where:

[0230] E data Indicates the local matching quality;

[0231] E smooth Indicates the consistency of adjacent matches;

[0232] λ is the balance factor (typical value 0.1);

[0233] The optimal matching is solved by the graph cut algorithm.

[0234] S4. Augmented reality guidance: Based on visual-inertial tracking and adaptive AR content rendering, augmented reality overlay information is generated in real time to guide assembly.

[0235] Furthermore, the visual-inertial fusion posture tracking adopts a tightly coupled visual-inertial fusion method combined with an extended Kalman filter (EKF) to achieve stable posture tracking. The state vector X is defined as follows:

[0236] X=[p,v,q,b a ,b g ] T

[0237] Among them, p is the position, v is the velocity, q is the rotation represented by the quaternion, and b a and b g are the biases of the speedometer and gyroscope, respectively.

[0238] Furthermore, the prediction step of X is:

[0239] X k|k-1 =f(X k-1|k-1 ,u k )

[0240]

[0241] in:

[0242] X k|k-1 represents the state vector predicted based on the k-1 state at time k;

[0243] f function is the nonlinear state transfer function, P k|k-1 is the predicted state covariance matrix, P k-1|k-1 According to the state X at the last moment k|k-1 and the current input u k To predict the current state, Q is the state covariance matrix of the previous moment. k is the process noise covariance matrix, which represents the uncertainty of the system during the state transition process.

[0244] Furthermore, the update steps of X are:

[0245]

[0246] Among them, K k is the Kalman gain, P k|k-1 is the predicted state covariance matrix; H k is the Jacobian matrix, which describes the derivative of the observation function with respect to the state vector, R k is the observation noise covariance matrix, which represents the uncertainty in the observation process.

[0247] X k|k =X k|k-1 +K k (Z k -h(Xk|k-1 ))

[0248] P k|k =(IK k H k ) k|k-1

[0249] Among them, X k|k is the updated state vector, X k|k-1 is the predicted state vector, Z k is the observation value at the current moment, h function is a nonlinear observation function, which maps the state vector to the observation space, P k|k is the updated state covariance matrix, and I is the identity matrix.

[0250] Furthermore, the illumination estimation is calculated as follows:

[0251] Use spherical harmonics to estimate ambient lighting L(ω):

[0252]

[0253] in, is the spherical harmonic basis function, are the spherical harmonic coefficients (usually N=2, a total of 9 coefficients).

[0254] Furthermore, the adaptive AR content rendering includes:

[0255] A). Lighting estimation

[0256] Use spherical harmonics to estimate ambient lighting L(ω):

[0257]

[0258] in, is the spherical harmonic basis function, are the spherical harmonic coefficients (usually N=2, a total of 9 coefficients).

[0259] B). Occlusion processing

[0260] Combining depth map and edge detection for occlusion reasoning, occlusion probability P occl (x) is defined as:

[0261] P occl (x) = σ(α(D rl (x)-D v (x)+βE(x)))

[0262] Among them, D rl (x), D v(x) represents the depth of the real object and the virtual object respectively, E(x) is the edge strength, α and β are weight coefficients, and σ is the Sigmoid activation function.

[0263] C). Adaptive transparency

[0264] The transparency of the AR content is dynamically adjusted by the angle between the line of sight and the normal line of the object surface. The transparency formula is as follows:

[0265] α(x)=α base (1-γcosθ)(1-δC(x))

[0266] Among them, θ is the angle between the line of sight and the surface normal, C(x) is the local complexity measure, γ and δ are adjustment factors, and α base Base transparency (usually 0.8).

[0267] S5. Knowledge graph and intelligent decision-making: The system supports intelligent fault diagnosis and assembly decision-making through knowledge graph.

[0268] Furthermore, the construction process of the knowledge graph is as follows:

[0269] A). Ontology modeling: Use Web Ontology Language (OWL) to define the ontology of the assembly domain. The core categories include: parts, (Tool), operations, faults, and assembly steps.

[0270] B). Knowledge extraction: Combine rules and statistical methods to extract entities and relationships from text, images, and sensor data. The extraction confidence is calculated as follows:

[0271] conf(e,r,e')=freq(e,r,e') / freaq(e)·freq(e')

[0272] Among them, e and e' are entities, r is a relationship, and freq is a frequency function.

[0273] D) Knowledge reasoning: The reasoning is performed by combining description logic reasoning and Markov logic network (MLN). The probability of query q is defined as:

[0274]

[0275] Where:

[0276] Z is a normalization factor used to ensure that the probability value is within a reasonable range. Its function is to make the sum of the probabilities of all possible situations equal to 1;

[0277] ω is a possible world state, that is, a combination of various possible situations;

[0278] q is the query, i.e. the specific event or situation whose probability we want to determine;

[0279] E stands for evidence, i.e. known information or conditions;

[0280] A is {ω∈Ω|ω satisfies query q and satisfies evidence E};

[0281] ω i is the rule weight, which indicates the importance of different rules in reasoning;

[0282] n i (ω) represents the number of times the rule is satisfied in the world state, that is, it counts the number of times a specific rule is satisfied in a specific world state.

[0283] D). Visual query: Based on the query interface provided by SPARQL, it supports complex multi-hop relationship queries and path analysis.

[0284] S6. Predictive maintenance: Based on real-time data, equipment health assessment and maintenance prediction are performed. Further, the health assessment and maintenance prediction include the following steps:

[0285] A). Feature Engineering:

[0286] Combining the time domain and frequency domain features, the equipment health index HI is constructed. The formula is as follows:

[0287]

[0288] Among them, RMS is the root mean square value of the equipment vibration signal in the time domain; Kt is the kurtosis, which indicates the peak degree of the vibration signal amplitude distribution; PSD is the power spectral density, which refers to the energy distribution of the signal in the frequency domain, calculated as the normalized value of the square of the Fourier transform of the signal; Ent is the information entropy, which indicates the uncertainty of the signal; is the weight coefficient, which is determined by cross-validation and satisfies the sum of 1.

[0289] B). Anomaly Detection:

[0290] The isolation forest algorithm is used to detect abnormal data points, and the anomaly score s is defined as:

[0291] s(x,n)=2 -E(h(x)) / c(n)

[0292] in:

[0293] x is the data point to be evaluated, represented as a feature vector containing the RMS, Kt, PSD, and Ent calculated above;

[0294] h(x) is the average path length of data point x, that is, the average path length from the tree root x to the included leaf nodes;

[0295] E(h(x)) is the expected value of h(x) on all trees in the forest;

[0296] n is the number of samples used to train the isolation forest;

[0297] c(n)=2H(n-1)-(2(n-1) / n), where H(i) is the i-th harmonic number, defined as H(i)=ln(i)+Euler's constant;

[0298] The anomaly score s ranges from (0,1], and the closer it is to 1, the more likely it is an anomaly point.

[0299] C). Remaining useful life RUL prediction:

[0300] Long short-term memory network (LSTM) is used for time series prediction, and the loss function is defined as:

[0301] L = MSE (RUL pred ,RUL true )+λ·KL(q(z|X)|p(z))

[0302] in:

[0303] RUL pred The remaining useful life predicted by the model;

[0304] RUL true is the actual remaining useful life;

[0305] MSE is mean square error;

[0306] KL(q(z|X)|p(z)) is the KL divergence, which is used to regularize the difference between the posterior distribution q(z|X) and the prior distribution p(z) of the latent variable z;

[0307] X is the input time series data, including the historical health index HI;

[0308] λ is a balancing factor used to adjust the relative importance of MSE loss and KL divergence.

[0309] D). Decision support:

[0310] Based on the cost-sensitive decision model, taking into account the maintenance cost and downtime loss, the decision formula is:

[0311] Action = argmin a [C m (a)+P f (a) C f ]

[0312] in:

[0313] argmin a Represents the value of the independent variable that makes the function value reach the minimum value;

[0314] a indicates possible maintenance actions, such as "immediate maintenance", "planned maintenance", "continue operation", etc.;

[0315] C m (a) represents the cost of maintenance action a, including labor cost, spare parts cost, etc.;

[0316] P f (a) represents the failure probability after taking action a, estimated based on the RUL prediction results and historical data;

[0317] C f Represents the failure cost, including downtime loss, emergency repair cost, possible safety accident cost, etc.

[0318] The decision model selects the action that minimizes the expected total cost as the optimal decision.

[0319] S7. Data integration and management: Connect with enterprise information systems through standardized interfaces to achieve data synchronization and security management. The specific methods are:

[0320] A microservice architecture based on Docker and Kubernetes is used to ensure the scalability and maintainability of the system. The core services include: image processing service, AR rendering service, knowledge graph service, predictive maintenance service, and data synchronization service. Redis is used to implement distributed caching to optimize data access performance. The cache strategy adopts the LRU (Least Recently Used) algorithm, and the cache expiration time is set to 15 minutes. OAuth 2.0 is used for identity authentication and authorization, and JSON Web Token (JWT) is used to manage user sessions. All API communications are encrypted through HTTPS, and the system implements a fine-grained access control strategy to ensure data security.

[0321] This invention solves the challenge of real-time feature detection and matching on mobile devices by optimizing the ORB algorithm, multi-level feature matching and visual-inertial fusion technology. At the same time, it combines knowledge graph and machine learning technology to achieve intelligent assembly guidance and predictive maintenance. The system adopts a modular design with good scalability and integration capabilities with enterprise information systems. These technological innovations provide new solutions for precision assembly and equipment management in complex industrial environments.

[0322] In terms of the scope of protection of rights, the present invention is not limited to the specific embodiments described herein. It should be understood that the scope of protection of the present invention covers all reasonable changes, modifications and equivalent implementations according to the content of this document. The scope of rights of this application is intended to cover every novel point and creative step of the present invention and its potential application methods to ensure that the originality, practicality and technical foresight of the present invention are fully protected by law. We aim to protect our innovative achievements from unauthorized use and ensure the integrity and respect of intellectual property rights through these rights applications.

Claims

1. A mobile assembly guidance and detection system based on an optimized ORB algorithm, characterized in that: include: Image acquisition and preprocessing module, used to acquire high-quality images and perform preprocessing in complex lighting environments; Optimized ORB feature detection module for feature detection using adaptive FAST corner detection and main direction based rotation BRIEF descriptor algorithm; A multi-level feature matching module for accurate matching through locality-sensitive hashing, PROSAC geometric verification, and Lucas-Kanade optical flow algorithm; Augmented reality (AR) guidance module, used to provide assembly guidance based on visual-inertial fusion pose tracking and adaptive AR content rendering; Knowledge graph construction and application module to support intelligent assembly process management and optimization; Predictive maintenance module for equipment health monitoring and remaining service life prediction; System integration and data management module, used to achieve data interaction and security management with enterprise information systems.

2. The system according to claim 1, characterized in that The image acquisition and preprocessing module adopts adaptive exposure control and high dynamic range (HDR) imaging technology to acquire images, and uses convolutional neural network (DnCNN) to perform image denoising.

3. The system according to claim 1, characterized in that The adaptive FAST corner detection algorithm in the optimized ORB feature detection module has a feature point detection threshold T(p) calculated as follows: T(p)=T base ·(1+α·(C(p)-μ c )) / s c Where: T base is the basic threshold (default value is 16), α is the adjustment factor, μ c , σ c are the mean and standard deviation of the contrast of the entire image, respectively.

4. The system according to claim 1, characterized in that The multi-level feature matching module implements feature matching through the following steps: a) Use Locality Sensitive Hashing (LSH) for coarse matching; b) Geometric validation using PROSAC variants; c) Use the pyramid Lucas-Kanade optical flow algorithm for precise positioning; d) Global consistency optimization through graph cut algorithm.

5. The system according to claim 1, characterized in that The augmented reality (AR) guidance module adopts a tightly coupled visual-inertial fusion method combined with an extended Kalman filter (EKF) to achieve stable posture tracking, and performs illumination estimation and depth map occlusion processing through spherical harmonic functions.

6. The system according to claim 1, characterized in that The knowledge graph construction and application module uses the Web Ontology Language (OWL) to define the ontology of the assembly field, and combines description logic reasoning and Markov logic network (MLN) for knowledge reasoning.

7. The system according to claim 1, characterized in that The predictive maintenance module uses a long short-term memory network (LSTM) to predict the remaining useful life (RUL) of equipment and uses an isolation forest algorithm for anomaly detection.

8. The system according to claim 7, characterized in that The predictive maintenance module performs health assessment and maintenance prediction through the following steps: a) Construct equipment health indicators by combining time domain and frequency domain features; b) Detect abnormal data points using the isolation forest algorithm; c) Use long short-term memory network (LSTM) for remaining useful life prediction; d) Make maintenance decisions based on cost-sensitive decision models.

Images