Automobile part quality detection method and system based on artificial intelligence vision detection

Abstract

This application discloses a method and system for quality inspection of automotive parts based on artificial intelligence visual inspection, belonging to the field of artificial intelligence machine vision inspection. First, the acquired RGB and depth images are registered, and the part regions are extracted using a saliency detection network. Two-dimensional key points are extracted from the RGB region images and matched with key points of the 3D model. The PnP algorithm is used to obtain the initial 3D pose. Then, the point cloud generated from the depth image is iteratively registered with the 3D model to obtain a refined 3D pose. Based on this pose, the geometric transformation matrix from the part to the standard frontal pose is calculated, and pose correction is performed on the RGB and depth region images. Subsequently, a feature detection and matching network is used to match the corrected image with a standard template image to correct the detection window position. Finally, within the corrected detection window, the 3D dimensions of the part are calculated based on the depth value, and tolerance is determined. This significantly improves the accuracy, robustness, and automation level of online inspection of automotive parts.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence machine vision inspection technology, and in particular to a method and system for quality inspection of automotive parts based on artificial intelligence vision inspection. Background Technology

[0002] As automobile manufacturing moves towards intelligence and high precision, the demand for online inspection of parts is increasing. Traditional visual inspection methods typically rely on single RGB or depth images, which can easily lead to inaccurate part positioning, deviations in geometric feature extraction, and unstable 3D dimension measurements when faced with complex production line environments (such as changes in lighting, surface reflection, pose differences, and background interference).

[0003] Furthermore, most existing inspection methods only target the appearance or contour features in two-dimensional images, failing to utilize the true three-dimensional shape information of parts for accurate measurement, thus making it difficult to meet the high-precision tolerance control requirements of automotive parts. While some point cloud-based inspection methods can provide three-dimensional information, point cloud noise is significant, pose estimation robustness is insufficient, and there is a lack of effective pose correction mechanisms, making the position of subsequent inspection windows easily affected, leading to cumulative measurement deviations. On the other hand, long-term operation of production lines can also cause camera calibration parameter drift, but traditional systems mostly rely on manual recalibration and cannot achieve online self-calibration. Summary of the Invention

[0004] This invention addresses the technical problems existing in the background art by proposing a method and system for quality inspection of automotive parts based on artificial intelligence visual inspection.

[0005] To solve the technical problem, the technical solution of the present invention is as follows:

[0006] A method for quality inspection of automotive parts based on artificial intelligence visual inspection, the method comprising:

[0007] S1: Acquire RGB and depth images of the target automotive parts and register them to obtain pixel-aligned RGB images and corresponding depth images. Use a saliency detection network to extract the part regions from the pixel-aligned RGB images to obtain RGB region images and depth region images.

[0008] It is understandable that by simultaneously acquiring and registering RGB and depth images, rich texture and color information is preserved, and accurate three-dimensional geometric information is obtained, providing a reliable data foundation for subsequent detection and analysis combining two-dimensional and three-dimensional elements, and significantly improving the information completeness and environmental adaptability of the detection system.

[0009] S2: Based on the RGB region image, extract two-dimensional key points and match them with the key points of the three-dimensional model, and use the PnP algorithm to solve the initial three-dimensional pose; at the same time, convert the depth region image into a point cloud, and use the initial three-dimensional pose as the starting point to perform iterative point cloud registration with the three-dimensional model to obtain a refined three-dimensional pose.

[0010] It is understandable that the saliency detection network is used to automatically extract the component region and effectively eliminate background interference. By combining two-dimensional key point matching and three-dimensional point cloud iterative registration, high-precision attitude estimation from coarse to fine is achieved, ensuring that even when the pose of the component changes in a complex production line environment, its three-dimensional spatial attitude can still be obtained stably and accurately.

[0011] S3: Calculate the geometric transformation matrix from the current pose to the standard frontal pose based on the refined three-dimensional pose, and perform synchronous spatial transformation on the RGB region image and the depth region image based on the geometric transformation matrix to obtain the corrected RGB image and the corrected depth image;

[0012] It is understandable that by uniformly transforming the component images to a standard frontal view based on the refined 3D pose, the detection error caused by the difference in viewpoint and pose is eliminated, so that subsequent feature matching and size calculation are performed in a consistent spatial reference system, which greatly improves the reliability and repeatability of the detection results.

[0013] S4: Perform feature matching between the corrected RGB image and the standard template image, and correct the position of the detection window according to the matching result to obtain the final detection window after position correction;

[0014] Understandably, by matching the corrected RGB image with the standard template, the detection window position is automatically corrected, effectively overcoming the impact of slight positional offsets of parts on the production line, ensuring that geometric feature extraction and dimensional measurement are both located in the correct target area, and reducing false detections and missed detections.

[0015] S5: Extract the geometric features of the component within the corrected final detection window, calculate the three-dimensional dimensions of the component based on the depth value in the corrected depth image, and determine the acceptance based on the design tolerance.

[0016] It is understandable that within a precisely positioned detection window, by combining the geometric features of the calibrated RGB image with the true depth value of the calibrated depth image, high-precision calculation of the three-dimensional dimensions of the parts can be achieved, and the pass / fail determination can be automatically completed based on the preset design tolerances, so that the detection results are objective and accurate and meet the strict quality inspection standards for automotive parts.

[0017] S6: When multiple components are continuously detected to have systematic deviations in their three-dimensional dimensions that exceed a preset threshold, the calibration process is automatically triggered. The camera parameters are re-estimated and updated by re-acquiring images and depth data of the calibration board or standard components.

[0018] Understandably, when a systematic deviation is detected, the system can automatically trigger the calibration process to re-estimate and update the camera parameters, effectively addressing long-term changes such as equipment wear and temperature drift, ensuring that the detection system maintains high accuracy and stability throughout its entire life cycle, and reducing maintenance costs and the need for manual intervention.

[0019] Furthermore, step S1 includes:

[0020] Simultaneously acquire RGB and depth images of target automotive parts in production line transmission state, perform camera calibration and spatial registration on the RGB and depth images to obtain pixel-aligned RGB images and corresponding depth images;

[0021] The pixel-aligned RGB image is input into a pre-trained saliency detection network to generate a salient region map of the component. The salient region map is then binarized to obtain a binary mask. Based on the binary mask, the RGB region image and depth region image corresponding to the salient region of the component are extracted from the pixel-aligned RGB image and the corresponding depth image, respectively.

[0022] Furthermore, step S2 includes:

[0023] Based on the RGB region image, key points of the component in two-dimensional image are extracted by a key point detection network; the key points of the two-dimensional image are mapped to the coordinates of the three-dimensional key points pre-stored in the three-dimensional model of the component, and the PnP algorithm is used to calculate the initial three-dimensional pose of the component relative to the camera; at the same time, the depth region image is converted into three-dimensional point cloud data.

[0024] Using the initial three-dimensional pose as the initial value, the three-dimensional point cloud data and the three-dimensional model of the component are used for iterative registration and optimization to obtain a refined three-dimensional pose.

[0025] Furthermore, step S3 includes:

[0026] Based on the refined three-dimensional pose, calculate the geometric transformation matrix of the component from the current pose to the standard frontal pose; using the geometric transformation matrix, perform a synchronous rotation and translation composite spatial transformation on the RGB region image and the depth region image to generate the pose-normalized rectified RGB image and rectified depth image.

[0027] Furthermore, step S4 includes:

[0028] The calibrated RGB image and the pre-stored standard component template image are input into a pre-trained feature detection and matching network to obtain matching feature point pairs between the two images. Based on the pixel coordinate offset of the matching feature point pairs, the residual position offset of the component in the calibrated RGB image is calculated. Based on the residual position offset, the detection area window preset on the calibrated RGB image is translated and corrected to obtain the final detection window after position correction.

[0029] Furthermore, the processing procedure of the feature detection and matching network includes:

[0030] First, local features are extracted from the corrected RGB image and the pre-stored standard component template image using a learned keypoint detection and descriptor extraction network.

[0031] Then, the extracted local features are matched using a graph neural network matcher to obtain the matching feature point pairs. The matching quality is evaluated based on the confidence level and inlier ratio of the matching feature point pairs. When the evaluated matching quality is lower than a preset threshold, the system switches to an alternate path. The alternate path estimates the residual position offset using a template matching algorithm or a registration method based on the corrected depth image.

[0032] Furthermore, step S5 includes:

[0033] Within the image area defined by the final detection window, the outline and key internal geometric features of the component are extracted; based on the pixel coordinates of the geometric features in the calibrated RGB image, the depth value at the corresponding position in the calibrated depth image is queried, and the actual size parameters of the component are calculated; each of the actual size parameters is compared with a preset design tolerance range. If any of the actual size parameters exceeds the corresponding design tolerance range, the component is determined to be a defective product; otherwise, it is determined to be a qualified product.

[0034] Furthermore, the dimensional parameters include: the length, width, thickness, hole spacing, and hole diameter of the component; the tolerance range is pre-configured according to the design standards of automotive components.

[0035] An automotive parts quality inspection system based on artificial intelligence visual inspection, the system being used to perform any of the methods described above, the system comprising:

[0036] The data acquisition and segmentation module is used to acquire and register the RGB and depth images of the target parts to obtain pixel-aligned RGB images and corresponding depth images. The module also uses a saliency detection network to segment the parts region from the RGB images to obtain RGB region images and depth region images.

[0037] The three-dimensional pose estimation module is used to extract two-dimensional key points based on the RGB region image and match them with the key points of the three-dimensional model, and use the PnP algorithm to solve the initial three-dimensional pose; at the same time, the depth region image is converted into a point cloud, and starting from the initial three-dimensional pose, iterative point cloud registration is performed with the three-dimensional model to obtain a fine three-dimensional pose.

[0038] The attitude correction module is used to calculate the geometric transformation matrix from the current attitude to the standard frontal attitude based on the fine three-dimensional attitude, and to perform synchronous spatial transformation on the RGB region image and the depth region image based on this matrix to obtain the corrected RGB image and the corrected depth image.

[0039] The detection window refinement module is used to perform feature matching between the corrected RGB image and the standard template image, and to correct the position of the preset detection window based on the matching result to obtain the final detection window;

[0040] The three-dimensional measurement and judgment module is used to extract the geometric features of the parts within the final inspection window, calculate the three-dimensional dimensional parameters of the parts based on the corresponding depth values ​​in the corrected depth image, and make a pass / fail judgment based on the design tolerances.

[0041] The system self-calibration module is used to automatically trigger the calibration process when a systematic deviation in the attitude or size measurement results of multiple components is detected and exceeds a preset threshold. The camera parameters are re-estimated and updated by re-acquiring the image and depth data of the calibration board or standard components.

[0042] This application has the following advantages:

[0043] This invention proposes a method and system for quality inspection of automotive parts based on artificial intelligence visual inspection, achieving accurate segmentation, robust pose estimation, and 3D dimension measurement of parts. First, a saliency detection network is used to extract part regions from complex backgrounds, significantly improving foreground removal capabilities. Then, PnP pose estimation is achieved by combining 2D keypoints with a 3D model, and further iterative registration of point clouds and the 3D model is employed to achieve high-precision 3D pose estimation, eliminating the influence of pose deviations at their source. Subsequent spatial geometric transformations standardize parts to a unified viewpoint, enabling precise positioning of the inspection window and improving measurement repeatability. Combining a feature matching network and a backup path mechanism ensures stable positioning of the inspection window under various abnormal conditions. Furthermore, this invention utilizes the true depth values ​​from depth images to calculate the 3D dimensions of parts, effectively avoiding errors caused by relying solely on 2D features. The system also possesses automatic online self-calibration capabilities, automatically triggering calibration when parameters drift, significantly improving the long-term stability and reliability of the inspection system. The overall method is applicable to online high-precision inspection of various types of automotive parts, offering advantages such as high robustness, high precision, and high automation. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A schematic diagram of the overall process of the quality inspection method for automotive parts based on artificial intelligence visual inspection provided in the embodiments;

[0046] Figure 2 A schematic diagram of the process for acquiring RGB and depth images and extracting salient regions provided in the embodiment;

[0047] Figure 3 A schematic diagram of the process for obtaining fine pose through two-dimensional key point extraction and point cloud-three-dimensional model iterative registration provided in the embodiment;

[0048] Figure 4 A schematic diagram of the process for synchronous pose correction of RGB / depth images based on geometric transformation matrix provided in the embodiment;

[0049] Figure 5 A schematic diagram of the process for feature matching between the corrected image and the template image and correction of the detection window position provided in the embodiment;

[0050] Figure 6 A schematic diagram of the three-dimensional dimension measurement and tolerance determination process within the final inspection window provided in the embodiment;

[0051] Figure 7 The following is a flowchart illustrating the automatic calibration process triggered by systematic deviations, as provided in the embodiments. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] Example 1:

[0054] like Figure 1 As shown in the figure, this embodiment proposes a specific implementation process for an automotive parts quality inspection method based on artificial intelligence visual inspection, which includes the following steps:

[0055] S1: Acquire RGB and depth images of the target automotive parts and register them to obtain pixel-aligned RGB images and corresponding depth images. Use a saliency detection network to extract the part regions from the pixel-aligned RGB images, resulting in RGB region images and depth region images, such as... Figure 2 As shown.

[0056] Image acquisition system hardware configuration:

[0057] RGB camera: Industrial camera with a resolution of [resolution value missing]. Pixels:

[0058] Depth camera: Structured light depth camera, depth map resolution is Pixels, depth data is in meters;

[0059] Installation method: rigid installation, fixed relative position.

[0060] Sampling frequency: It is synchronized with the production line rhythm.

[0061] Camera calibration and image registration:

[0062] Camera model: pinhole model;

[0063] Calibration method: Zhang Zhengyou calibration method.

[0064] Output parameters:

[0065] RGB camera intrinsic parameter matrix: ;

[0066] Depth camera intrinsic parameter matrix: ;

[0067] The extrinsic transformation matrix from depth camera to RGB camera: .

[0068] Registration operation: using Map the depth image onto the RGB camera pixel plane.

[0069] Output data: Pixel-aligned RGB image and corresponding depth image .

[0070] Data cleaning: masking invalid values ​​in the depth image.

[0071] Extraction of salient regions of components:

[0072] Detection network: U-Net variant network, input size is Pixels, outputting a saliency heatmap of the same size; U-Net variant network, short for U-Net Architecture, is a model based on convolutional neural networks (CNN).

[0073] Binarization threshold: Used to convert heatmaps into binary masks ;

[0074] Region selection: for mask Perform connected component analysis and select the connected component with the largest area as the target component region;

[0075] Image cropping: Based on the minimum bounding rectangle of the above region, from... and Cut out from the middle:

[0076] RGB region image ;

[0077] Deep region image ;

[0078] Record information: Synchronously record the pixel coordinate transformation matrix of the cropped area relative to the original image for use in subsequent steps.

[0079] In this embodiment, the saliency detection network is used to predict the salient regions of components from the input RGB image, and its input is the size-normalized RGB image. The output is a significance probability plot of the same size. ;in, Represents an image; H Indicates the height of the image. W Indicates the width of the image. This indicates the number of channels in the image.

[0080] The saliency detection network can be implemented using a convolutional neural network with an encoder-decoder structure. The encoder is used to extract multi-scale features, and the decoder is used to upsample step by step and restore spatial resolution.

[0081] Significance prediction and binarization:

[0082] Significance probability graph of network output The following binarization rule is used to generate a saliency mask. :

[0083]

[0084] in: Represents pixel coordinates; The significance threshold is defined as follows, and its value range is: By using a mask Perform connected component analysis and select the connected region with the largest area as the salient region of the target component.

[0085] The network training method involves training the saliency detection network offline using manually labeled salient region data of components, and employing a binary cross-entropy loss function during training. :

[0086]

[0087] in, This represents the actual labeled value for the corresponding pixel.

[0088] S2: Based on the RGB region image, extract two-dimensional key points and match them with the key points of the three-dimensional model. Calculate the initial three-dimensional pose using the PnP (Perspective-n-Point) algorithm. Simultaneously, convert the depth region image into a point cloud. Starting from the initial three-dimensional pose, iteratively register the point cloud with the three-dimensional model to obtain a refined three-dimensional pose, such as... Figure 3 As shown.

[0089] For example, will Input a pre-trained keypoint detection network to extract N sets of two-dimensional keypoints. Output the confidence score for each point. Discard with a confidence level lower than 0. The key points. Among them, and Represents the x and y coordinates in a two-dimensional coordinate system. Indicates the first The coordinates of the points; where the key point detection network can be a key point detector based on a high-resolution network.

[0090] The system pre-stores the 3D model of the component and its M 3D key points. Based on the design, a one-to-one correspondence is established between 2D key points and 3D key points; where each point The coordinates are from The components represent the point in the three-dimensional coordinate system. , and coordinate.

[0091] Using the PnP algorithm supported by RANSAC, input The selected 2D-3D correspondence yields the initial rotation vector. With translation vector Synthesize the initial homogeneous transformation matrix The RANSAC in-point threshold is set to 8 pixels.

[0092] Will Effective depth values ​​by pixel coordinates With depth Using depth camera intrinsics Back projection into point cloud (Unit: meters). The point cloud was downsampled using a 0.5mm voxel grid and denoised using statistical filtering with parameters set to a 50-point neighborhood and a threshold of 1.0 standard deviation; where, This refers to a segmented or extracted portion of the depth image. The specific extraction strategy typically utilizes a binary mask generated by a saliency detection network. This mask identifies the region of interest in the image, allowing the algorithm to further analyze the depth information of that portion. Represents a set The three-dimensional coordinates of each point in the array are as follows: : Represents the X coordinate (horizontal position) of the point in the world coordinate system; : Represents the Y coordinate (vertical position) of the point in the world coordinate system; : Represents the Z coordinate (depth, i.e., the distance of the point from the observer) of the point in the world coordinate system.

[0093] by As the initial transformation, the point cloud First, transform to the model coordinate system and perform ICP iterative registration; Example algorithms: point-to-plane ICP, nearest point search using KD-tree, k-dimensional tree, k-dimensional tree:

[0094] Maximum number of iterations ;

[0095] Convergence condition: Change in root mean square error between two consecutive registrations m or RMSE < 0.001m;

[0096] Outliers are removed by using a weighted truncation strategy, which means that points whose residuals are greater than 3 times the median residual are identified as outliers and removed.

[0097] Obtain fine three-dimensional pose ;

[0098] This is a homogeneous transformation matrix representing the fine three-dimensional pose of an object.

[0099] This is a 3x3 rotation matrix representing the rotation of an object relative to the world coordinate system. This matrix can transform points in the object's coordinate system to the world coordinate system through rotation.

[0100] This is a separator used to distinguish between rotation matrices and translation vectors.

[0101] This is a 3x1 translation vector, representing the position of the object in three-dimensional space.

[0102] In this embodiment, the key point detection network is used to predict the positions of two-dimensional key points of components, and its input is an RGB region image. The output is Heatmap of key points Each heatmap Indicates the first The probability distribution of the presence of key points in the image.

[0103] Keypoint coordinates and confidence level calculation; for the first A key point, its two-dimensional pixel coordinates Obtained from the location of the maximum response on the heatmap:

[0104]

[0105] Corresponding key point confidence Defined as:

[0106]

[0107] when At that time, the key point was deemed unreliable and removed; among them, The confidence threshold for key points.

[0108] Network training loss; the keypoint detection network uses the mean squared error loss function during the training phase. :

[0109]

[0110] in, For the first The true Gaussian heat map of each key point; : Indicates the image position First The probability of the existence of each key point, with a numerical range of [value range missing]. between.

[0111] S3: Calculate the geometric transformation matrix from the current pose to the standard frontal pose based on the refined 3D pose. Perform a synchronous spatial transformation on the RGB region image and the depth region image based on the geometric transformation matrix to obtain a corrected RGB image and a corrected depth image, such as... Figure 4 As shown.

[0112] For example, a standard frontal view is preset in the 3D model coordinate system. The translation is defined as: aligning the model's principal axis with the camera's optical axis; at the same time, the translation amount should ensure that the model's center is located at the center of the image.

[0113] The geometric transformation matrix from the fine pose to the standard pose is:

[0114]

[0115] in ; It is a homogeneous transformation matrix. This represents a 3x3 rotation matrix, indicating a rotation from the current coordinate system to the target or standard coordinate system. The translation vector is 3x1, representing the position of the object in three-dimensional space.

[0116] This is a homogeneous transformation matrix, representing the transformation of the standard frontal pose. :This is The inverse transformation matrix is ​​used to calculate the transformation from the current attitude to the standard attitude.

[0117] Synchronous spatial transformation: Performing composite rotation and translation transformations within the image domain on the cropped RGB region image and the corresponding depth region image:

[0118] Each pixel in the RGB region image Back projection into three-dimensional points ;right application get Then project it onto the pixel plane to obtain the pixel. Corrected RGB images are generated using bilinear interpolation. The corresponding depth is re-interpolated to generate a corrected depth image. . Representing a three-dimensional point P After geometric transformation Then, the coordinates in the new coordinate system or in the new spatial location.

[0119] To avoid holes, an interpolation and depth filling strategy based on inverse mapping is adopted. Holes are filled with the median depth of the neighborhood, and the maximum allowable filling distance does not exceed 2mm.

[0120] The result is the pose-standardized result. and .

[0121] S4: Perform feature matching between the corrected RGB image and the standard template image. Based on the matching results, adjust the position of the detection window to obtain the final detection window after position correction. Figure 5As shown.

[0122] For example, the system pre-stores standard component template images. Shot in a standard head-on orientation, resolution and They are the same, and both have their expected detection window. .

[0123] right and Two sets of local features are obtained by using a learned keypoint detection network and a descriptor network, respectively. Then, a graph neural network matcher is used for context-aware matching, outputting a set of matching pairs. and its confidence level, This represents the set of keypoint correspondences from the corrected image to the template image.

[0124] Calculate the inlier ratio and average confidence level; set the matching quality threshold: inlier ratio And average confidence level RANSAC stands for Random Sample Consensus; specifically, the ratio of interior points is calculated using RANSAC to determine the number of interior points / total number of matches during homography or affine transformation.

[0125] Pixel-level offset calculated based on interior point pairing The median of the paired offsets can be taken, or the translation components can be obtained by least-squares fitting of the affine transformation.

[0126] Preset detection window exist Chinese Press Translation yields the final detection window after position correction. .

[0127] If the matching quality is below the threshold, an alternative path is used: first in Use template-based correlation matching, i.e., normalized cross-correlation NCC, or based on... Registration involves using local point clouds and template point clouds to perform ICP estimation of residual migrations. Here, NCC stands for Normalized Cross-Correlation.

[0128] In this embodiment, the feature detection and matching network includes a learned feature extraction module and a graph neural network matching module. The feature extraction module extracts features from the corrected RGB image and the standard template image respectively, obtaining two sets of local features:

[0129] Each local feature includes: two-dimensional pixel coordinates Feature descriptor vector and key point confidence values The feature extraction module can be implemented using a learning-based keypoint detection and descriptor joint network known in the field, such as a feature extraction structure based on a convolutional neural network.

[0130] The graph neural network matcher is constructed as follows: local features from the corrected RGB image and the template image are respectively constructed into two graph structures; each node in the graph corresponds to a local feature point; the node feature consists of the descriptor vector of the feature point and its pixel position information; the edges between nodes are established based on the spatial adjacency relationship or feature similarity of the feature points in the image. These two graphs are input into the graph neural network matcher, and the node features are updated through a multi-layer message passing mechanism to enhance the contextual consistency of the features.

[0131] The graph neural network matcher outputs the matching confidence value between each pair of candidate feature points, thus forming a set of matching feature point pairs.

[0132] Perform matching quality assessment and offset estimation; based on the matched feature point pairs, use the RANSAC algorithm to estimate the geometric transformation relationship between images, and calculate: the inlier ratio and the average confidence of the matched point pairs; when the inlier ratio is higher than the first threshold and the average confidence is higher than the second threshold, the matching result is determined to be valid; calculate the residual position offset based on the pixel coordinate difference of the inlier matching point pairs.

[0133] The backup path is triggered and implemented as follows: When the matching quality assessment result is lower than a preset threshold, the system automatically switches to the backup path. The backup path includes the following methods: template-based image matching: In the calibrated RGB image, normalized cross-correlation matching is performed on the region near the preset detection window, and the positional offset corresponding to the maximum correlation coefficient is calculated; or depth image-based registration: Local point clouds within the detection window are extracted in the calibrated depth image, and iterative nearest point (ICP) registration is performed with the point clouds corresponding to the standard template. The residual translational offset is estimated based on the registration result. When the backup path successfully estimates the residual positional offset, this offset is used to correct the position of the detection window.

[0134] S5: Extract the geometric features of the component within the corrected final inspection window, calculate the three-dimensional dimensions of the component based on the depth values ​​in the corrected depth image, and determine acceptance based on design tolerances. Figure 6 As shown.

[0135] For example, in the final detection window Internally, Canny edge detection, connected component analysis, and polygon fitting are first used to correct RGB images. Extract the component contours; simultaneously, identify key internal geometric features, including hole centers, groove edges, and reference surfaces. Then, for each feature pixel... In the corresponding corrected depth image Query its depth value And ultimately utilize the RGB camera intrinsics Backproject it into three-dimensional space points .

[0136] Example of geometric quantity calculation:

[0137] Length / Width: For key points at both ends in 3D space Calculate Euclidean distance .

[0138] Thickness: Calculated by taking the average z-value difference between the two planes and projecting it into the vertical direction.

[0139] Aperture / Aperture Spacing: Aperture measurement involves identifying the edge profile of the hole and using a least-squares method based on three-dimensional points to fit a circle to obtain the radius. The hole spacing is calculated directly by measuring the Euclidean distance between the two hole centers in three-dimensional space.

[0140] Gaussian filtering was applied to the measurement points to reduce the impact of depth noise on single-point measurements; the measurements were repeated three times and the average value was taken as the final measurement value.

[0141] Each measurement parameter is compared with the preset design tolerance range; if any parameter exceeds the tolerance, it is deemed unqualified; otherwise, it is deemed qualified. The results are recorded in the production database, including the measurement value, threshold, posture information, and original image path.

[0142] S6: When multiple component attitude or dimensional measurement results show systematic deviations exceeding preset thresholds, the calibration process is automatically triggered. This involves re-acquiring images and depth data from the calibration board or standard components to re-estimate and update the camera parameters. Figure 7 As shown.

[0143] For example, the system will calculate a statistical deviation index for the most recent M detection results: calculate a measurement deviation sequence for each critical dimension. Calculate the moving average With sliding standard deviation If K consecutive parts Exceeding the set threshold ,and If this occurs, a systematic deviation is determined to exist, and an automatic calibration process is triggered; among which, It is the first The actual measured value of an object or component; It is a standard value or tolerance specification; This indicates a preset threshold or standard value.

[0144] The system prompts the production line robot or a person to place the calibration board or standard parts to the acquisition position, and automatically collects N sets of RGB+ depth data, covering different views and positions.

[0145] Re-estimation using Zhang's method on color images Distortion parameters and camera relative pose; scale and depth deviation correction using depth maps and known calibration board 3D points, and estimation of depth scale factor. With offset , making If RGB-D pairs are used simultaneously, the intrinsic parameters are jointly optimized for depth calibration, minimizing the weighted sum of pixel reprojection error and depth error. Among these, It is a standardized or corrected depth value. It is the uncorrected, raw depth value.

[0146] Write the new calibration parameters to the temporary configuration and quickly verify whether the measurement deviation returns to the normal range on subsequent T-parts with a low confidence threshold; if the verification passes, persist and update the parameters; if the verification fails, roll back and alert for manual intervention.

[0147] The calibration process saves all raw data, calibration results, and verification curves; the system can be set to automatically roll back to prevent performance degradation during new calibrations.

[0148] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0149] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for quality inspection of automotive parts based on artificial intelligence visual inspection, characterized in that, The method includes: S1: Acquire RGB and depth images of the target automotive parts and register them to obtain pixel-aligned RGB images and corresponding depth images. Use a saliency detection network to extract the part regions from the pixel-aligned RGB images to obtain RGB region images and depth region images. S2: Based on the RGB region image, extract two-dimensional key points and match them with the key points of the three-dimensional model, and use the PnP algorithm to solve the initial three-dimensional pose; at the same time, convert the depth region image into a point cloud, and use the initial three-dimensional pose as the starting point to perform iterative point cloud registration with the three-dimensional model to obtain a refined three-dimensional pose. S3: Calculate the geometric transformation matrix from the current pose to the standard frontal pose based on the refined three-dimensional pose, and perform synchronous spatial transformation on the RGB region image and the depth region image based on the geometric transformation matrix to obtain the corrected RGB image and the corrected depth image; S4: Perform feature matching between the corrected RGB image and the standard template image, and correct the position of the detection window according to the matching result to obtain the final detection window after position correction; S5: Extract the geometric features of the component within the corrected final detection window, calculate the three-dimensional dimensions of the component based on the depth value in the corrected depth image, and determine the acceptance based on the design tolerance. S6: When multiple components are continuously detected to have systematic deviations in their three-dimensional dimensions that exceed a preset threshold, the calibration process is automatically triggered. The camera parameters are re-estimated and updated by re-acquiring the image and depth data of the calibration board or standard components. Step S4 includes: The calibrated RGB image and the pre-stored standard component template image are input into a pre-trained feature detection and matching network to obtain matching feature point pairs between the two images; the residual position offset of the component in the calibrated RGB image is calculated based on the pixel coordinate offset of the matching feature point pairs; the detection area window preset on the calibrated RGB image is translated and corrected based on the residual position offset to obtain the final detection window after position correction. The processing steps of the feature detection and matching network include: First, local features are extracted from the corrected RGB image and the pre-stored standard component template image using a learned keypoint detection and descriptor extraction network. Then, the extracted local features are matched using a graph neural network matcher to obtain the matching feature point pairs; the matching quality is evaluated based on the confidence level and inlier ratio of the matching feature point pairs; when the evaluated matching quality is lower than a preset threshold, the system switches to an alternate path, which estimates the residual position offset using a template matching algorithm or a registration method based on the corrected depth image. Step S5 includes: Within the image area defined by the final detection window, the outline and key internal geometric features of the component are extracted; based on the pixel coordinates of the geometric features in the calibrated RGB image, the depth value at the corresponding position in the calibrated depth image is queried, and the actual size parameters of the component are calculated; each of the actual size parameters is compared with a preset design tolerance range. If any of the actual size parameters exceeds the corresponding design tolerance range, the component is determined to be a defective product; otherwise, it is determined to be a qualified product. The dimensional parameters include: the length, width, thickness, hole spacing, and hole diameter of the component; the tolerance range is pre-configured according to the design standards of automotive components.

2. The method for quality inspection of automotive parts based on artificial intelligence visual inspection according to claim 1, characterized in that, Step S1 includes: Simultaneously acquire RGB and depth images of target automotive parts in production line transmission state, perform camera calibration and spatial registration on the RGB and depth images to obtain pixel-aligned RGB images and corresponding depth images; The pixel-aligned RGB image is input into a pre-trained saliency detection network to generate a salient region map of the component. The salient region map is then binarized to obtain a binary mask. Based on the binary mask, the RGB region image and depth region image corresponding to the salient region of the component are extracted from the pixel-aligned RGB image and the corresponding depth image, respectively.

3. The method for quality inspection of automotive parts based on artificial intelligence visual inspection according to claim 1, characterized in that, Step S2 includes: Based on the RGB region image, key points of the component in two-dimensional image are extracted by a key point detection network; the key points of the two-dimensional image are mapped to the coordinates of the three-dimensional key points pre-stored in the three-dimensional model of the component, and the PnP algorithm is used to calculate the initial three-dimensional pose of the component relative to the camera; at the same time, the depth region image is converted into three-dimensional point cloud data. Using the initial three-dimensional pose as the initial value, the three-dimensional point cloud data and the three-dimensional model of the component are used for iterative registration and optimization to obtain a refined three-dimensional pose.

4. The method for quality inspection of automotive parts based on artificial intelligence visual inspection according to claim 1, characterized in that, Step S3 includes: Based on the refined three-dimensional pose, calculate the geometric transformation matrix of the component from the current pose to the standard frontal pose; using the geometric transformation matrix, perform a synchronous rotation and translation composite spatial transformation on the RGB region image and the depth region image to generate the pose-normalized rectified RGB image and rectified depth image.

5. An automotive parts quality inspection system based on artificial intelligence visual inspection, characterized in that, The system is used to perform the steps of any one of claims 1-4, the system comprising: The data acquisition and segmentation module is used to acquire and register the RGB and depth images of the target parts to obtain pixel-aligned RGB images and corresponding depth images. The module also uses a saliency detection network to segment the parts region from the RGB images to obtain RGB region images and depth region images. The three-dimensional pose estimation module is used to extract two-dimensional key points based on the RGB region image and match them with the key points of the three-dimensional model, and use the PnP algorithm to solve the initial three-dimensional pose; at the same time, the depth region image is converted into a point cloud, and starting from the initial three-dimensional pose, iterative point cloud registration is performed with the three-dimensional model to obtain a fine three-dimensional pose. The attitude correction module is used to calculate the geometric transformation matrix from the current attitude to the standard frontal attitude based on the fine three-dimensional attitude, and to perform synchronous spatial transformation on the RGB region image and the depth region image based on this matrix to obtain the corrected RGB image and the corrected depth image. The detection window refinement module is used to perform feature matching between the corrected RGB image and the standard template image, and to correct the position of the preset detection window based on the matching result to obtain the final detection window; The three-dimensional measurement and judgment module is used to extract the geometric features of the parts within the final inspection window, calculate the three-dimensional dimensional parameters of the parts based on the corresponding depth values ​​in the corrected depth image, and make a pass / fail judgment based on the design tolerances. The system self-calibration module is used to automatically trigger the calibration process when a systematic deviation in the attitude or size measurement results of multiple components is detected and exceeds a preset threshold. The camera parameters are re-estimated and updated by re-acquiring the image and depth data of the calibration board or standard components.

Images