Intelligent assembly guiding and real-time quality inspection method and system for narrow space

Abstract

The invention relates to the technical field of crossing of augmented reality and intelligent manufacturing, and provides an intelligent assembly guiding and real-time quality inspection method and system for a narrow space. According to the method, multiple sensors are arranged in AR glasses to collect narrow space environment data in real time, high-precision matching of a virtual model and a real environment is realized based on a point cloud registration technology, and a dynamic assembly guide animation is generated; after the process is completed, triggering a quality inspection instruction through an AR interface gesture, and automatically shooting an assembly result by a depth camera carried by a mechanical arm; performing real-time feature comparison by adopting a deep learning algorithm to generate a visual detection report; and when the assembly deviation is detected, the system immediately provides error correction guidance on an AR interface and automatically restarts current process guidance to form an'operation-quality inspection-error correction 'real-time closed loop. The method effectively solves the technical problems that in narrow space assembly, blind area operation difficulty is large, quality verification lags behind, and the rework rate is high.

Description

Technical Field

[0001] This invention relates to the field of augmented reality and intelligent manufacturing, and in particular to an intelligent assembly guidance and real-time quality inspection method and system for confined spaces. Background Technology

[0002] In assembly scenarios within confined spaces such as aircraft engine nacelles, traditional methods face the following technical challenges: 1) Extreme spatial conditions with a depth-to-diameter ratio ≥ 5:1 severely limit the operator's field of vision, resulting in numerous blind spots; 2) The assembly process relies entirely on the operator's experience and feel, making it difficult to guarantee consistent quality; 3) Quality inspection is typically conducted offline after assembly, requiring extensive rework upon discovering problems; 4) Traditional inspection methods are inefficient and lack real-time feedback mechanisms. Existing AR-assisted assembly technologies are mostly designed for open spaces, and suffer from low registration accuracy and poor tracking stability in confined spaces, failing to meet the requirements of high-precision assembly. Summary of the Invention

[0003] 1. The technical problem to be solved:

[0004] How to perform high-precision assembly in a confined space.

[0005] 2. Technical Solution:

[0006] To address the above problems, this invention provides an intelligent assembly guidance and real-time quality inspection method for confined spaces, applicable to enclosed confined spaces with a depth-to-diameter ratio greater than or equal to 5:1, comprising the following steps:

[0007] Step S10: Real-time acquisition of environmental point cloud and RGB images of the confined space environment using the depth sensor and high-resolution camera built into the AR glasses;

[0008] Step S20: Identify the parts to be assembled based on the three-dimensional point cloud data and RGB image, and automatically match the current assembly process;

[0009] Step S30: Generate a dynamic assembly guidance animation based on the current process using point cloud registration technology, and project a step-by-step guide including component movement paths, assembly angles, and fastening prompts to the operator in real time through AR glasses;

[0010] Step S40: After the operator completes the assembly according to the AR real-time guidance, the operator triggers the quality inspection command through gestures on the AR interface;

[0011] Step S50: The binocular depth camera mounted on the robotic arm automatically locates and captures the assembly results according to the instructions, and uses deep learning algorithms to perform real-time feature comparison to generate a visual inspection report;

[0012] Step S60: If the inspection report shows an assembly error, the text correction guide will be displayed in real time on the AR interface and the current process guidance animation will be automatically restarted.

[0013] This invention also provides an intelligent assembly system for real-time AR guidance and quality inspection closed-loop in confined spaces, used to implement the aforementioned intelligent assembly guidance and real-time quality inspection method for confined spaces, including...

[0014] Image acquisition module: Real-time acquisition of 3D point cloud data and RGB images of confined spaces using the depth sensor and high-resolution camera built into the AR glasses;

[0015] Process matching module: Identifies the parts to be assembled based on image data and automatically matches them with the current assembly process;

[0016] AR Real-Time Guidance Module: Generates dynamic assembly guidance animation based on the current process using point cloud registration technology, and projects it to the operator's field of vision in real time;

[0017] Quality inspection trigger module: Receives quality inspection instructions sent by the operator via gestures on the AR interface;

[0018] Real-time quality inspection module: controls the binocular depth camera mounted on the robotic arm to automatically take pictures and uses deep learning algorithms to perform real-time feature comparison and generate a visual inspection report;

[0019] Closed-loop feedback module: When an assembly error is detected, it provides real-time AR error correction guidance and restarts the current process guidance.

[0020] 3. Beneficial effects:

[0021] This invention, through its innovative design of AR real-time guidance and quality inspection closed loop, enables "operation-while-verification" in the assembly process within confined spaces, significantly reducing rework rates. It also solves the technical challenge of low virtual model registration accuracy in extremely limited spaces with a depth-to-diameter ratio ≥ 5:1. Furthermore, it automates and intelligentizes quality inspection through robotic arm-based automatic positioning and imaging, and real-time feature comparison using deep learning. This forms a complete real-time closed loop of "operation-quality inspection-error correction," significantly improving assembly quality and efficiency while reducing reliance on operator experience. Attached Figure Description

[0022] Figure 1 This is a flowchart of the intelligent assembly method of the present invention.

[0023] Figure 2 This is a schematic diagram of the overall architecture of the intelligent assembly system with real-time guidance and quality inspection closed loop of the present invention. Detailed Implementation

[0024] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] like Figure 1 As shown, an intelligent assembly guidance and real-time quality inspection method for confined spaces, applicable to enclosed confined spaces with a depth-to-diameter ratio greater than or equal to 5:1, includes the following steps:

[0026] Step S10: Real-time acquisition of environmental point cloud and RGB images of the confined space environment using the depth sensor and high-resolution camera built into the AR glasses;

[0027] Step S20: Identify the parts to be assembled based on the environmental point cloud and RGB image, and automatically match the current assembly process;

[0028] Step S30: Generate a dynamic assembly guidance animation based on the current process using point cloud registration technology, and project a step-by-step guide including component movement paths, assembly angles, and fastening prompts to the operator in real time through AR glasses;

[0029] Step S40: After the operator completes the assembly according to the AR real-time guidance, the operator triggers the quality inspection command through gestures on the AR interface;

[0030] Step S50: The binocular depth camera mounted on the robotic arm automatically locates and captures the assembly results according to the instructions, and uses deep learning algorithms to perform real-time feature comparison to generate a visual inspection report;

[0031] Step S60: If the inspection report shows an assembly error, the text correction guide will be displayed in real time on the AR interface and the current process guidance animation will be automatically restarted.

[0032] In one embodiment, in step S30, the goal of the point cloud registration technique is to find an optimal rigid body transformation matrix T such that the standard three-dimensional model point cloud of the component to be assembled is such that... Environmental point clouds collected in real time through AR glasses The optimal rigid body transformation matrix T is obtained by optimizing the initially aligned model using an iterative nearest-point algorithm to minimize the error between the two points. The first point cloud representing the standard 3D model of the component to be assembled One point, This represents the first point in the environmental point cloud collected by AR glasses. One point, It is the total number of points in a standard 3D model point cloud. It represents the total number of points in the environmental point cloud.

[0033] The specific method includes the following steps:

[0034] Step S31: Feature extraction: Extract fast point feature histogram features from the real-time acquired environmental point cloud and the standard 3D model point cloud of the component to be assembled.

[0035] For each point P in the point cloud, a simplified feature histogram (SPFH) is calculated, and then combined with the SPFH values ​​of its k nearest neighbors to form the FPFH descriptor for that point. This is a 33-dimensional vector that is robust to point cloud density and noise.

[0036] The specific method for feature extraction is as follows: Define point P and one of its neighboring points. The relative characteristics between them, let and Points P and P are respectively. Using the normal vector, construct a local coordinate system: , , ,

[0037] Calculate the eigenvalues ​​of the three angles:

[0038] ;

[0039] ;

[0040] ;

[0041] in , , These are the three basis vectors of the local coordinate system established by point P; Description in by and The relative relationship between the normal vector of a neighboring point and the direction of the line connecting the two points in Zhang Cheng's plane; To measure the similarity of the orientation of surfaces at two points; The normal vector describing the neighborhood points is given by and Zhang Cheng's inclination angle in the plane.

[0042] These three angle values ​​are quantized into histograms and the neighborhood information is merged to obtain fast point feature histogram features. The neighborhood information refers to the simplified feature histogram features of the k nearest neighbors of the current point p.

[0043] Step S32: Coarse registration: Matching is performed based on the extracted fast point feature histogram features. The initial transformation matrix is ​​estimated through the sample consistency initial registration algorithm, and the standard 3D model point cloud is initially aligned to the real-time environment point cloud.

[0044] The purpose of coarse registration is to provide a good initial value when the positions of two point clouds differ significantly, thus preventing fine registration from getting trapped in local optima. This invention employs feature matching based on Fast Point Feature Histogram (FPFH) combined with the Sample Consistency Initial Registration (SAC-IA) algorithm. The specific method is as follows:

[0045] Step S321: Let from Random selection Feature matching is performed on n sampling points: for each sampling point In environmental point clouds Find the points with the most similar features in the fast point feature histogram. : ;

[0046] Step S322: Transformation estimation: based on matching point pairs The initial transformation is solved using the unit quaternion method. Specifically, solving for the rotation matrix Translation vector Make:

[0047] ;

[0048] Step S323: Interior point statistics: Calculate the proportion of interior points in the current transformation, where an interior point refers to a pair of points that can be correctly matched under the current transformation matrix.

[0049] ,

[0050] in It is an initial transformation matrix to be evaluated during each random sampling and testing process. It is an exponential function. For nearest neighbor search function, The threshold value is the interior point threshold.

[0051] Step S324: Iterative optimization: Repeat the above process multiple times, and select the transformation with the highest proportion of interior points as the coarse registration result.

[0052] Step S33: Fine Registration: The iterative nearest point algorithm is used to optimize the initially aligned model, continuously calculating the optimal rigid body transformation to ultimately achieve the overlay of the standard 3D model point cloud with the real assembly environment. The iterative nearest point algorithm solves for high-precision transformation through iterative optimization, specifically as follows:

[0053] Step S331: Initialization: Iterate until convergence.

[0054] This refers to the initial transformation matrix in the fine registration stage;

[0055] Step S332: Data Association: For each point in the standard 3D model Find the nearest neighbor in the environmental point cloud as follows:

[0056] ;

[0057] Step S333: Transformation Solution: From the currently estimated standard 3D model point cloud positions Transform to corresponding points in the environmental point cloud Incremental transformation, calculate the optimal transformation Make:

[0058] ,

[0059] This refers to the transformation matrix estimated in the k-th iteration. An incremental correction.

[0060] Step S334: Update Transformation: ;

[0061] Step S335: Convergence judgment: If If the iteration fails, stop; otherwise, set k = k + 1 and return to step S331.

[0062] Finally, the optimal transformation matrix is ​​obtained after iterative convergence. .

[0063] In one embodiment, the specific method of step S50 is as follows:

[0064] Step S51: Model preprocessing: Scale the assembly result image captured by the binocular depth camera to a fixed size and normalize the pixel values, and use it as input to the YOLO detection network;

[0065] Step S52: Mesh generation and bounding box prediction: The YOLO model uniformly divides the input image into... Each grid cell is responsible for prediction. Each bounding box and the confidence level of the assembly component to be detected contained in these bounding boxes;

[0066] In one embodiment, each bounding box is defined by the following geometric properties: center coordinates Relative to the top-left corner coordinates of its grid cell ;width and height Normalized relative to the width and height of the entire image.

[0067] The parameters directly predicted by the network are The actual coordinates of the bounding box are calculated using the following formula:

[0068] ;

[0069] ;

[0070] ;

[0071] ;

[0072] in, As an activation function, ensure that the center coordinate offset is constrained between 0 and 1, so that the center point falls within the current grid. The width and height dimensions are the prior bounding boxes obtained through cluster analysis.

[0073] Step S53: Confidence Score and Class Probability Calculation: The confidence score of each bounding box represents the accuracy of the prediction box location and the presence of a target object within the box. The calculation formula is as follows:

[0074] Confidence level = ;

[0075] in, This is the network prediction value. It is the probability that an object exists within the bounding box. It is the intersection-union ratio (IUU) between the predicted bounding box and the ground truth bounding box.

[0076] At the same time, each grid cell predicts a set of conditional class probabilities. The comma indicates that when an object exists within the mesh, that object belongs to that category. The probability, ultimately for each bounding box, for each category The overall confidence score is:

[0077] ;

[0078] Step S54: Generate a visual inspection report: The system will overlay the comparison results, including identified parts and missing assembly anomalies, onto the original image in an augmented reality annotation manner, and transmit the image with the inspection results to the AR glasses through the database to form a visual inspection report. The augmented reality annotation is to add a bright frame and text prompts.

[0079] An intelligent assembly system for real-time AR guidance and quality inspection closed-loop in confined spaces is provided to implement the aforementioned intelligent assembly guidance and real-time quality inspection method for confined spaces. Figure 2 As shown, including

[0080] Image acquisition module: Real-time acquisition of 3D point cloud data and RGB images of confined spaces using the depth sensor and high-resolution camera built into the AR glasses;

[0081] Process matching module: Identifies the parts to be assembled based on image data and automatically matches them with the current assembly process;

[0082] AR Real-Time Guidance Module: Generates dynamic assembly guidance animation based on the current process using point cloud registration technology, and projects it to the operator's field of vision in real time;

[0083] Quality inspection trigger module: Receives quality inspection instructions sent by the operator via gestures on the AR interface;

[0084] Real-time quality inspection module: controls the binocular depth camera mounted on the robotic arm to automatically take pictures and uses deep learning algorithms to perform real-time feature comparison and generate a visual inspection report;

[0085] Closed-loop feedback module: When an assembly error is detected, it provides real-time AR error correction guidance and restarts the current process guidance, forming a complete real-time closed loop of "operation-quality inspection-error correction".

[0086] In one embodiment, the AR real-time guidance module includes a 3D model library that stores standard 3D models of standard parts to be assembled and assembly process parameters, a point cloud registration unit that achieves high-precision registration of the 3D models with the real-time environment through feature matching and iterative nearest point algorithms, and an animation generation unit that generates guidance animations in real time, including part movement paths, assembly angles, and fastening operation prompts.

[0087] In one embodiment, the real-time quality inspection module includes a deep learning detection unit that uses a convolutional neural network to perform real-time feature extraction and comparison of captured images, and a visualization report unit that generates a visualization inspection report containing augmented reality annotations.

[0088] In one embodiment, the closed-loop feedback module includes a real-time error correction unit that immediately generates AR error correction guidance when an assembly error is detected, and a guidance animation that automatically restarts the current process, thereby achieving a complete closed loop of the process.

[0089] The quality inspection response time of this invention is less than 0.5 seconds, which is a significant improvement compared to traditional manual inspection, and the overall assembly qualification rate is greater than 98%.

Claims

1. A method for intelligent assembly guidance and real-time quality inspection in confined spaces, applicable to enclosed confined spaces with a depth-to-diameter ratio greater than or equal to 5:1, characterized in that: Includes the following steps: Step S10: Real-time acquisition of environmental point cloud and RGB images of the confined space environment using the depth sensor and high-resolution camera built into the AR glasses; Step S20: Identify the parts to be assembled based on the environmental point cloud and RGB image, and automatically match the current assembly process; Step S30: Generate a dynamic assembly guidance animation based on the current process using point cloud registration technology, and project a step-by-step guide including component movement paths, assembly angles, and fastening prompts to the operator in real time through AR glasses; Step S40: After the operator completes the assembly according to the AR real-time guidance, the operator triggers the quality inspection command through gestures on the AR interface; Step S50: The binocular depth camera mounted on the robotic arm automatically locates and captures the assembly results according to the instructions, and uses deep learning algorithms to perform real-time feature comparison to generate a visual inspection report; Step S60: If the inspection report shows an assembly error, the text correction guide will be displayed in real time on the AR interface and the current process guidance animation will be automatically restarted.

2. The intelligent assembly guidance and real-time quality inspection method for confined spaces as described in claim 1, characterized in that: In step S30, the goal of the point cloud registration technique is to find an optimal rigid body transformation matrix T such that the point cloud of the standard three-dimensional model of the component to be assembled is such that... Environmental point clouds collected in real time through AR glasses The error between them is the smallest. The first point cloud representing the standard 3D model of the component to be assembled One point, This represents the first point in the environmental point cloud collected by AR glasses. One point, It is the total number of points in a standard 3D model point cloud. It is the total number of points in the environmental point cloud. Specific methods include: Step S31: Feature extraction: Extract fast point feature histogram features from the real-time acquired environmental point cloud and the standard 3D model point cloud of the part to be assembled, respectively; Step S32: Coarse registration: Matching is performed based on the extracted fast point feature histogram features. The initial transformation matrix is ​​estimated through the sampling consistency initial registration algorithm, and the standard 3D model point cloud is initially aligned to the environment point cloud. Step S33: Fine registration: The iterative nearest point algorithm is used to optimize the point cloud of the initially aligned standard 3D model, and the optimal rigid body transformation is calculated iteratively to finally achieve the superposition of the standard model and the real assembly environment.

3. The intelligent assembly guidance and real-time quality inspection method for confined spaces as described in claim 2, characterized in that: The optimal rigid body transformation matrix T is obtained by optimizing the initially aligned model using an iterative nearest-point algorithm.

4. The intelligent assembly guidance and real-time quality inspection method for confined spaces as described in claim 2 or 3, characterized in that: In step S31, the specific method for feature extraction is as follows: define point P and one of its neighboring points. The relative characteristics between them, let and Points P and P are respectively. Using the normal vector, construct a local coordinate system: ， ， ， Calculate the eigenvalues ​​of the three angles: ， ， ， in , , These are the three basis vectors of the local coordinate system established by point P; Description in by and The relative relationship between the normal vector of a neighboring point and the direction of the line connecting the two points in Zhang Cheng's plane; To measure the similarity of the orientation of surfaces at two points; The normal vector describing the neighborhood points is given by and Zhang Cheng's inclination angle within the plane; These three angle values ​​are quantized into histograms and then merged with neighborhood information to obtain fast point feature histogram features.

5. The intelligent assembly guidance and real-time quality inspection method for confined spaces as described in claim 4, characterized in that: The specific method for step S32 is as follows: Step S321: Let from Random selection Feature matching is performed on n sampling points: for each sampling point In environmental point clouds Find the points with the most similar features in the fast point feature histogram. : ; Step S322: Transformation estimation: based on matching point pairs The initial transformation is solved using the unit quaternion method. Solve for the rotation matrix Translation vector Make: ； Step S323: Interior Point Statistics: Calculate the proportion of interior points in the current transformation. ， in It is an initial transformation matrix to be evaluated during each random sampling and testing process. It is an exponential function. For nearest neighbor search function, The threshold value is the interior point threshold. Step S324: Iterative optimization: Repeat the above process multiple times, and select the transformation with the highest proportion of interior points as the coarse registration result.

6. The intelligent assembly guidance and real-time quality inspection method for confined spaces as described in claim 5, characterized in that: In step S33: The iterative nearest point algorithm solves for the high-precision transformation through iterative optimization, specifically as follows: Step S331: Initialization: Iterate until convergence. This refers to the initial transformation matrix in the fine registration stage; Step S332: Data Association: For each point in the standard 3D model Find the nearest neighbor in the environmental point cloud as follows: ； Step S333: Transformation Solution: From the currently estimated standard 3D model point cloud positions Transform to corresponding points in the environmental point cloud Incremental transformation, calculate the optimal transformation Make: ， This refers to the transformation matrix estimated in the k-th iteration. An incremental correction; Step S334: Update Transformation: ; Step S335: Convergence judgment: If If the iteration fails, stop; otherwise, set k = k + 1 and return to step S331. Finally, the optimal transformation matrix is ​​obtained after iterative convergence. .

7. The intelligent assembly guidance and real-time quality inspection method for confined spaces as described in claim 1, characterized in that: The specific method for step S50 is as follows: Step S51: Model preprocessing: Scale the assembly result image captured by the binocular depth camera to a fixed size and normalize the pixel values, and use it as input to the YOLO detection network; Step S52: Mesh generation and bounding box prediction: The YOLO model uniformly divides the input image into... Each grid cell is responsible for prediction. Each bounding box and the confidence level of the assembly component to be detected contained in these bounding boxes; Step S53: Confidence Score and Class Probability Calculation: The confidence score of each bounding box represents the accuracy of the prediction box location and the presence of a target object within the box. The calculation formula is as follows: Confidence level = , in, This is the network prediction value. It is the probability that an object exists within the bounding box. It is the intersection-union ratio of the predicted bounding box and the ground truth bounding box; At the same time, each grid cell predicts a set of conditional class probabilities. The comma indicates that when an object exists within the mesh, that object belongs to that category. The probability, ultimately for each bounding box, for each category The overall confidence score is: ； Step S54: Generate a visual inspection report: The system will overlay the comparison results, including identified parts and missing assembly anomalies, onto the original image in an augmented reality annotation manner, and transmit the image with the inspection results to the AR glasses through the database to form a visual inspection report. The augmented reality annotation is to add a bright frame and text prompts.

8. The intelligent assembly guidance and real-time quality inspection method for confined spaces as described in claim 7, characterized in that: Each bounding box is defined by the following geometric properties: center coordinates Relative to the top-left corner coordinates of its grid cell ; width and height Normalized relative to the width and height of the entire image. The parameters directly predicted by the network are The actual coordinates of the bounding box are calculated using the following formula: ， ， ， ， in, As an activation function, ensure that the center coordinate offset is constrained between 0 and 1, so that the center point falls within the current grid. The width and height dimensions are the prior bounding boxes obtained through cluster analysis.

9. An intelligent assembly system for real-time AR guidance and quality inspection closed-loop in confined spaces, used to implement the intelligent assembly guidance and real-time quality inspection method for confined spaces as described in any one of claims 1-8, characterized in that: include Image acquisition module: Real-time acquisition of 3D point cloud data and RGB images of confined spaces using the depth sensor and high-resolution camera built into the AR glasses; Process matching module: Identifies parts to be assembled based on image data and automatically matches them with the current assembly process; AR Real-Time Guidance Module: Generates dynamic assembly guidance animation based on the current process using point cloud registration technology, and projects it to the operator's field of vision in real time; Quality inspection trigger module: Receives quality inspection instructions sent by the operator via gestures on the AR interface; Real-time quality inspection module: controls the binocular depth camera mounted on the robotic arm to automatically take pictures and uses deep learning algorithms to perform real-time feature comparison and generate a visual inspection report; Closed-loop feedback module: When an assembly error is detected, it provides real-time AR error correction guidance and restarts the current process guidance.

10. The intelligent assembly system for AR real-time guidance and quality inspection closed loop in confined spaces as described in claim 9, characterized in that: The AR real-time guidance module includes: 3D Model Library: Stores standard 3D models of standard components to be assembled, along with assembly process parameters; Point cloud registration unit: Achieves high-precision registration between 3D models and the real-time environment through feature matching and iterative nearest-point algorithm; Animation generation unit: generates guided animations in real time, including component movement paths, assembly angles, and fastening operation prompts; The real-time quality inspection module includes: Deep learning detection unit: Employs a convolutional neural network to perform real-time feature extraction and comparison of captured images. Visualization Report Unit: Generates a visual inspection report containing augmented reality annotations; The closed-loop feedback module includes: Real-time error correction unit: generates AR error correction guidance immediately upon detecting an assembly error; Process restart unit: Automatically restarts the guidance animation of the current process to achieve a complete process loop.

Images