Door guard visual automatic detection system

Abstract

The present application relates to the technical field of data processing, a door guard plate visual automatic detection system, comprising an image acquisition module for synchronously acquiring multi-view original images; a curved surface modeling module for reconstructing a three-dimensional curved surface grid model based on stereo vision; a projection correction module for re-projecting image pixels to a virtual orthogonal plane to generate a standard correction image sequence; a double-path registration module for performing registration based on SIFT features and deep learning regression in parallel, and outputting a registration result with confidence; a registration optimization module for fusing objective similarity indexes and confidence for weighted scoring, and selecting an optimal registration image; a defect detection module for locating a defect area through differential operation and morphological processing; and a result classification module for realizing defect classification in combination with shape and texture features. The present application effectively solves registration errors caused by curved surface distortion, and improves the accuracy and consistency of defect detection.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a visual automatic inspection system for door panels. Background Technology

[0002] Existing door panel visual inspection technologies suffer from the following technical challenges: Door panels have complex three-dimensional curved surface structures. In automated production line online visual inspection, when multiple industrial cameras acquire images from fixed positions, perspective distortion and feature point position shifts occur due to changes in the surface geometry. This makes it difficult for standard image registration algorithms, such as feature point matching methods, to accurately align multiple images or template images, as the correspondence between feature points is distorted by the changing viewing angle. For example, when detecting scratches on the door panel surface, the curvature of the surface causes the scratches to appear as curved or broken shapes in the image, resulting in a mismatch between the registered defect location and the standard template. This leads to misclassification of defect size and type by the inspection system, thereby reducing defect detection consistency and quality classification accuracy. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a door panel visual automatic inspection system, which solves the technical problem that the difference in image acquisition perspective caused by the three-dimensional curved surface structure of the door panel results in low image registration accuracy of the vision system, thus affecting the consistency of defect detection.

[0004] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:

[0005] The automatic visual inspection system for door panels provided by this invention includes:

[0006] The image acquisition module is used to simultaneously acquire raw images of the door panel surface from different perspectives;

[0007] The surface modeling module is used to receive the original image and calculate and generate a three-dimensional curved surface mesh model of the door panel surface;

[0008] The projection correction module is used to receive the original image and the three-dimensional curved surface mesh model, map the pixels of the original image onto the surface of the three-dimensional curved surface mesh model, and then reproject it onto a virtual orthogonal projection plane to generate a standard corrected image sequence.

[0009] The dual-path registration module is used to receive the standard corrected image sequence, perform registration based on SIFT features and registration based on deep learning network regression in parallel, and output the registration results of the two registration paths and the corresponding registration confidence scores.

[0010] The registration optimization module is used to receive the registration results and registration confidence of the two registration paths, calculate the objective similarity index between each registration result and the pre-stored standard template image, integrate the registration confidence and the objective similarity index to perform a comprehensive score, and select the registration result with the highest score as the optimal registration image for output.

[0011] The defect detection module is used to receive the optimal registration image, calculate the difference map between the optimal registration image and the standard template image, segment and locate potential defect regions, and output the contour information of the defect regions.

[0012] The result classification module is used to receive the contour information of the defect area and the optimal registration image, extract the features of the defect area and classify them, and output a detection report containing the defect category and location.

[0013] Furthermore, in the door panel visual automatic inspection system of the present invention, the curved surface modeling module is used for:

[0014] Corner detection is performed on the received original image, feature points are extracted, and feature descriptors are generated;

[0015] Between the original images from different viewpoints, the feature points are matched based on descriptor similarity to establish pairs of feature points with the same name;

[0016] Based on the camera calibration parameters, the three-dimensional spatial coordinates of the corresponding feature point pairs are calculated using the binocular parallax principle to generate an initial sparse point cloud.

[0017] The initial sparse point cloud is densified by interpolation using a global optimization algorithm to generate a dense three-dimensional point cloud.

[0018] The dense 3D point cloud is subjected to Delaunay triangulation to construct the 3D surface mesh model with topological connectivity.

[0019] Furthermore, in the door panel visual automatic detection system of the present invention, the dual-path registration module performs SIFT feature-based registration for:

[0020] On the standard calibrated image sequence and the pre-stored standard template image, scale-invariant feature transform (SIFT) detection is performed to obtain key points and corresponding SIFT descriptors;

[0021] Using the K-nearest neighbor algorithm, preliminary matching is performed between key points of the standard corrected image sequence and the standard template image based on the Euclidean distance of the SIFT descriptor;

[0022] The Random Sample Consensus (RANSAC) algorithm is applied to remove mismatched point pairs from the initial matching results and to estimate the homography transformation matrix that describes the spatial transformation relationship between the two images.

[0023] The homography transformation matrix is ​​used to perform geometric transformation and resampling on the standard corrected image sequence to generate a first registration result image, and the number of correctly matched point pairs retained after filtering by the RANSAC algorithm is used as the registration confidence of the first path.

[0024] Furthermore, in the door panel visual automatic detection system of the present invention, the dual-path registration module performs registration based on deep learning network regression for:

[0025] The standard corrected image sequence and the pre-stored standard template image are concatenated in the channel dimension to construct a dual-channel input tensor;

[0026] The dual-channel input tensor is input into a pre-trained convolutional neural network, and the fully connected layer at the end of the convolutional neural network outputs a six-dimensional vector, which represents the six parameters of the affine transformation.

[0027] The six-dimensional vector is decoded into a specific affine transformation matrix;

[0028] The standard corrected image sequence is spatially transformed and interpolated using the affine transformation matrix obtained by decoding to generate a second registration result image, and the activation value of the six-dimensional vector corresponding to the output layer of the convolutional neural network is used as the registration confidence of the second path.

[0029] Furthermore, in the door panel visual automatic detection system of the present invention, the registration and selection module is used for:

[0030] The normalized mutual information value and structural similarity index (SSIM) between the first registration result image, the second registration result image and the standard template image are calculated respectively, and used as the objective similarity index.

[0031] The registration confidence scores of the first path and the second path are converted to the same numerical range using the max-min normalization method.

[0032] For each registration path, the normalized registration confidence, the normalized mutual information value, and the structural similarity index (SSIM) are weighted and summed according to a preset weighting factor to obtain a comprehensive score for each path.

[0033] The comprehensive scores corresponding to the first registration result image and the second registration result image are compared, and the registration result image with the higher comprehensive score and the corresponding transformation matrix are determined as the optimal registration image and transformation parameters.

[0034] Furthermore, in the door panel visual automatic inspection system of the present invention, the defect detection module is used for:

[0035] Perform pixel-by-pixel grayscale difference operation on the optimal registration image and the standard template image, and obtain the initial difference map by taking the absolute value;

[0036] The Otsu method is applied to the initial difference map to adaptively calculate the segmentation threshold, and the initial difference map is binarized using the segmentation threshold to obtain a preliminary defect region mask.

[0037] Morphological closing and opening operations are sequentially performed on the preliminary defect area mask to fill the small holes inside the defect area and remove edge burrs and isolated noise points;

[0038] Connected component labeling and analysis are performed on the morphologically processed binary image. The area and minimum bounding rectangle coordinates of each connected component are calculated. Connected components with an area less than a preset threshold are filtered out. The contour pixel coordinates of the remaining connected components are output as the contour information of the defect region.

[0039] Furthermore, in the door panel visual automatic inspection system of the present invention, the result classification module extracts defect area features for:

[0040] Based on the defect region contour information, the corresponding image block is extracted from the optimal registration image;

[0041] For each defect region, calculate the geometric moments of the defect region contour, and derive the rectangularity and aspect ratio as shape features based on the geometric moments;

[0042] For each image patch extracted from the optimally registered image, the gray-level co-occurrence matrix of the image patch is calculated, and energy and contrast are extracted from the gray-level co-occurrence matrix as texture features;

[0043] The shape features and texture features are combined to construct a multidimensional feature vector representing the defect region.

[0044] Furthermore, in the door panel visual automatic detection system of the present invention, the gray-level co-occurrence matrix of the calculated image block is used for:

[0045] Based on the gray level of the image block and the preset pixel displacement, the frequency of pixel pairs that meet the preset distance is counted in four directions: 0 degrees, 45 degrees, 90 degrees and 135 degrees, and primary co-occurrence matrices are generated in the four directions respectively.

[0046] The average of the matrix elements of the primary co-occurrence matrices in the four directions is calculated to generate a comprehensive gray-level co-occurrence matrix;

[0047] The energy and contrast features are extracted from the comprehensive gray-level co-occurrence matrix.

[0048] Furthermore, in the door panel visual automatic detection system of the present invention, the support vector machine model used by the result classification module for classification is trained for:

[0049] Collect sample images of door panel defects containing known categories;

[0050] The multidimensional feature vector is extracted from each defect sample image and labeled with the corresponding defect category label to form a training dataset;

[0051] The support vector machine model was trained using the training dataset, and the kernel function and penalty parameters of the model were optimized using grid search and cross-validation.

[0052] Furthermore, in the door panel visual automatic detection system of the present invention, the image acquisition module is used for:

[0053] Receives arrival trigger signals from the conveyor belt of the automated production line;

[0054] In response to the arrival trigger signal, a synchronous acquisition command is sent to all industrial cameras;

[0055] Control each industrial camera to synchronously capture a single frame of color image according to pre-calibrated internal and exposure parameters;

[0056] The captured color image is subjected to Gaussian filtering for noise reduction, and the denoised image is converted from the RGB color space to the grayscale color space.

[0057] The beneficial effects of this invention are:

[0058] This invention effectively eliminates perspective distortion caused by the complex curved surface structure of door panels by constructing a three-dimensional curved mesh model. It utilizes a virtual orthogonal projection plane to generate a standardized corrected image sequence with unified viewpoints, fundamentally solving the problem of inconsistent registration benchmarks for multi-view images. The dual-path registration module combines the advantages of traditional feature matching and deep learning regression, significantly improving the robustness and accuracy of image registration through parallel computing and confidence evaluation mechanisms. The registration optimization module employs a weighted scoring strategy based on multi-index fusion to adaptively select the optimal registration result, ensuring precise correspondence between defect detection locations and standard templates. Finally, through multi-feature fusion classification technology, it comprehensively uses shape and texture features for defect discrimination, greatly improving the consistency and accuracy of identification and classification of typical defects such as scratches and dents, providing reliable quality inspection assurance for door panel production lines. Attached Figure Description

[0059] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on the drawings without creative effort.

[0060] Figure 1 This is a system architecture diagram of the door panel visual automatic inspection system of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The technical solutions provided by various embodiments of this invention will be described in detail below with reference to the accompanying drawings.

[0062] To better understand the purpose of this invention, the invention will now be described in further detail.

[0063] Please see Figure 1 The automatic visual inspection system for door panels provided by the present invention includes:

[0064] The image acquisition module is used to simultaneously acquire raw images of the door panel surface from different perspectives;

[0065] The surface modeling module is used to receive the original image and calculate and generate a three-dimensional curved surface mesh model of the door panel surface;

[0066] The projection correction module is used to receive the original image and the three-dimensional curved surface mesh model, map the pixels of the original image onto the surface of the three-dimensional curved surface mesh model, and then reproject it onto a virtual orthogonal projection plane to generate a standard corrected image sequence.

[0067] The dual-path registration module is used to receive the standard corrected image sequence, perform registration based on SIFT features and registration based on deep learning network regression in parallel, and output the registration results of the two registration paths and the corresponding registration confidence scores.

[0068] The registration optimization module is used to receive the registration results and registration confidence of the two registration paths, calculate the objective similarity index between each registration result and the pre-stored standard template image, integrate the registration confidence and the objective similarity index to perform a comprehensive score, and select the registration result with the highest score as the optimal registration image for output.

[0069] The defect detection module is used to receive the optimal registration image, calculate the difference map between the optimal registration image and the standard template image, segment and locate potential defect regions, and output the contour information of the defect regions.

[0070] The result classification module is used to receive the contour information of the defect area and the optimal registration image, extract the features of the defect area and classify them, and output a detection report containing the defect category and location.

[0071] The door panel vision automatic inspection system achieves high-precision defect detection through the collaborative work of multiple modules. After receiving the arrival trigger signal from the production line conveyor belt, the image acquisition module sends a synchronous acquisition command to all industrial cameras; each industrial camera synchronously captures a single frame of color image according to the pre-calibrated internal parameters and exposure parameters; then, the acquired color image is subjected to Gaussian filtering for noise reduction, and the denoised image is converted from the RGB color space to the grayscale color space to generate a raw image data stream suitable for subsequent processing.

[0072] The surface modeling module receives the raw image data stream output by the image acquisition module. First, it performs corner detection on the raw image to extract feature points and generate feature descriptors. Then, it performs feature point matching based on descriptor similarity between raw images from different viewpoints to establish pairs of feature points with the same name. According to the camera calibration parameters, it uses the principle of binocular parallax to calculate the three-dimensional spatial coordinates of the pairs of feature points with the same name to generate an initial sparse point cloud. It then uses a global optimization algorithm to perform densification interpolation on the initial sparse point cloud to generate a dense three-dimensional point cloud. Finally, it performs Delaunay triangulation on the dense three-dimensional point cloud to construct a three-dimensional surface mesh model with topological connectivity.

[0073] The projection correction module simultaneously receives the original image data stream and the 3D curved surface mesh model. Based on the camera extrinsic matrix, it back-projects the coordinates of each pixel in the original image onto the surface of the 3D mesh model to obtain 3D coordinates. It assigns the pixel grayscale values ​​of the original image to the corresponding 3D coordinate points to form texture associations. It defines a virtual orthogonal camera viewpoint perpendicular to the main plane of the door panel. It renders the 3D mesh model and its associated textures onto the 2D imaging plane under the virtual viewpoint to generate a standard corrected image sequence without perspective distortion.

[0074] The dual-path registration module receives the standard corrected image sequence and initiates parallel computation. The SIFT-based registration path performs scale-invariant feature transformation detection on the standard corrected image sequence and the pre-stored standard template image to obtain key points and SIFT descriptors; it uses the K-nearest neighbor algorithm to perform preliminary matching based on the Euclidean distance of the descriptors; it applies the random sampling consensus algorithm to eliminate mismatched point pairs and estimate the homography transformation matrix; it uses the homography matrix to perform geometric transformation and resampling on the standard corrected image sequence to generate the first registration result image, using the number of correctly matched point pairs as the registration confidence. The deep learning network regression-based registration path concatenates the standard corrected image sequence and the standard template image into a dual-channel input tensor; it regresses the affine transformation parameters through a pre-trained convolutional neural network and decodes them into a transformation matrix; it applies the transformation matrix to perform spatial transformation on the image to generate the second registration result image, using the network output activation value as the registration confidence.

[0075] The registration optimization module receives dual-path registration result data packets and calculates the normalized mutual information value and structural similarity index between the first and second registration result images and the standard template image, respectively. It transforms the registration confidence of the two paths to the same dimension using the max-min normalization method. It performs a weighted summation of the normalized confidence, mutual information value, and structural similarity index of each path according to preset weights to obtain a comprehensive score. It compares the comprehensive scores of the two paths and selects the registration result image and transformation matrix with the higher score as the optimal output.

[0076] After receiving the optimal registration image, the defect detection module performs pixel-by-pixel grayscale difference operation between the optimal registration image and the standard template image, and obtains the initial difference map by taking the absolute value. The Otsu method is applied to adaptively calculate the segmentation threshold and binarize the initial difference map to generate a preliminary defect region mask. Morphological closing and opening operations are sequentially performed on the binary image to fill holes and remove noise. Connected component labeling and analysis are performed on the processed binary image, the area of ​​each connected component and the coordinates of the bounding rectangle are calculated, and the defect region contour information is output after filtering out regions with too small an area.

[0077] The result classification module extracts corresponding image patches from the optimally registered image based on the defect region contour information; calculates the geometric moments of the defect region contour and derives the rectangularity and aspect ratio as shape features; calculates the gray-level co-occurrence matrix of the image patch and extracts energy and contrast as texture features; combines shape features and texture features to construct a multi-dimensional feature vector; inputs the feature vector into a pre-trained support vector machine model for classification decision; and finally generates a structured detection report containing defect location, category, and size, completing the entire detection process from image acquisition to defect recognition.

[0078] Upon receiving the raw image data, the surface modeling module initiates the 3D reconstruction process. First, the module performs corner detection on each raw image, using the Harris corner detection algorithm to locate pixels with significant gradient changes as feature points. For each detected feature point, it calculates the gradient direction histogram of the surrounding pixel region, generating a rotation-invariant SIFT feature descriptor. After feature extraction for a single image, the module performs feature matching between raw images from different viewpoints. It finds the best matching point pair by comparing the Euclidean distance between feature descriptors; when the distance between the descriptors of two feature points is less than a preset threshold, they are considered a pair of feature points with the same name.

[0079] After establishing pairs of feature points with the same name, the module calculates the 3D spatial coordinates of the feature points using epipolar geometric constraints in stereo vision, based on pre-calibrated camera intrinsic and extrinsic parameter matrices. The matched 2D image points are then back-projected into 3D space using triangulation to generate an initial sparse point cloud composed of discrete 3D points. Due to the insufficient density of the initial point cloud, the module employs a graph-cut-based global optimization algorithm to densify the point cloud. While maintaining geometric continuity, new 3D points are interpolated into the gaps between the points, ultimately forming a dense 3D point cloud that completely covers the surface of the door panel.

[0080] After obtaining a dense 3D point cloud, the module executes the Delaunay triangulation algorithm to transform the discrete point cloud into a continuous surface model. The triangulation process ensures that all point clouds are covered by non-overlapping triangular facets, and that the circumcircle of each triangle does not contain vertices of other point clouds, thus guaranteeing the smoothness and geometric accuracy of the mesh model. The final generated 3D surface mesh model contains vertex coordinates, normal vector information, and the topological connectivity of the triangular facets, providing a geometric basis for subsequent projection correction.

[0081] In the SIFT-based registration path, the module simultaneously extracts scale-invariant features from both the standard corrected image sequence and the pre-stored standard template image. The SIFT detector locates scale-stable keypoints by constructing a Gaussian difference pyramid and assigns a 128-dimensional feature vector to each keypoint. The feature matching stage employs a KD-tree-accelerated nearest neighbor search algorithm to quickly find the most similar matching pair between two sets of feature points.

[0082] To improve matching accuracy, the module introduces a Random Sampling Consensus (RANSAC) algorithm to filter the initial matching results. The RANSAC algorithm randomly selects the minimum sample set to estimate the transform model, then counts the number of inliers that match the model, and iteratively optimizes to find the transform matrix with the most inliers. Finally, the optimal transform matrix is ​​used to resample the standard corrected image sequence, generating the first registration result image spatially aligned with the template image.

[0083] In the deep learning-based registration path, the module employs a specialized convolutional neural network structure to handle image registration tasks. The network input is a two-channel tensor composed of a standard corrected image sequence and a standard template image. High-level features are extracted through layers of convolution and pooling operations. The fully connected layers at the network's end regress six affine transformation parameters, which are then decoded to form a complete transformation matrix. During the training phase, the network is trained end-to-end using a large number of labeled image pairs to learn the nonlinear mapping relationship from image appearance to spatial transformation.

[0084] The registration optimization module employs a multi-index fusion decision-making mechanism to evaluate the output quality of the two registration paths. The module calculates the normalized mutual information value between the first and second registered images and the standard template image, reflecting the similarity between the two images at the information entropy level. Simultaneously, it calculates the structural similarity index to comprehensively evaluate image similarity from three dimensions: brightness, contrast, and structure.

[0085] For the registration confidence score inherent in each path, the module uses the max-min normalization method to map confidence scores of different dimensions to the 0-1 interval, eliminating the influence of numerical range differences on the score. The normalized registration confidence score and the objective similarity index are linearly weighted according to preset weights. The weight coefficients are obtained through extensive experimental data calibration to ensure that the comprehensive score can accurately reflect the true quality of the registration results.

[0086] The defect detection module locates potential defect regions through differential processing. The module first compares the pixel-by-pixel grayscale values ​​of the optimal registration image and the standard template image, obtaining an initial difference map by taking the absolute value. Due to the influence of illumination variations and image noise, direct thresholding segmentation will generate a large number of false defect regions. Therefore, the module uses Otsu's method to automatically calculate the optimal segmentation threshold. This algorithm determines the optimal threshold for dividing the image into foreground and background by maximizing the inter-class variance.

[0087] After obtaining the binarized defect mask, the module sequentially performs morphological closing and opening operations to optimize the shape of the defect region. The closing operation uses a circular structuring element to perform a dilation-erosion operation on the mask, filling in small holes and fracture areas inside the defect. The opening operation performs an erosion-dilation operation to eliminate edge burrs and isolated noise points. The morphologically processed binary image then uses a connected component analysis algorithm to label all continuous regions and calculates the geometric properties of each region, such as area, perimeter, and minimum bounding rectangle.

[0088] The classification module employs a multi-feature fusion strategy to accurately classify defects. For each detected defect region, the module extracts shape features such as rectangularity and aspect ratio from the defect contour. Rectangularity reflects the similarity between the defect region and its minimum bounding rectangle, while aspect ratio describes the overall morphological proportions of the defect. Simultaneously, the module calculates the gray-level co-occurrence matrix from the image patches of the defect region, extracting texture features such as energy and contrast. Energy values ​​reflect the uniformity of the image texture, while contrast characterizes the clarity of the texture.

[0089] Features are combined into multidimensional feature vectors and then input into a support vector machine (SVM) classifier. During the training phase, the SVM uses a kernel function to map the feature vectors to a high-dimensional space to find the optimal classification hyperplane. The model employs grid search and cross-validation methods to optimize the penalty and kernel function parameters, ensuring that the classifier maintains high generalization ability even on unseen samples. The final classification results are integrated with defect location information to generate a structured inspection report, providing data support for quality control and process improvement.

[0090] The image acquisition module is tightly integrated with the automated production line to achieve efficient collaborative operation. The module receives the arrival trigger signal from the conveyor belt encoder via an industrial Ethernet interface, precisely controlling the timing of image acquisition. Synchronous acquisition commands are simultaneously sent to all industrial cameras via hardware trigger signal lines, ensuring that multi-view images are captured at the same time, eliminating image registration errors caused by object movement.

[0091] The camera parameter pre-calibration process consists of two parts: intrinsic parameter calibration and extrinsic parameter calibration. Intrinsic parameter calibration calculates the camera's focal length, principal point, and distortion coefficients by photographing a checkerboard calibration board. Extrinsic parameter calibration determines the relative positions and poses between cameras. Calibration parameters are stored in a configuration file for use during image acquisition. Acquired color images are Gaussian filtered to remove noise and then converted to grayscale to reduce the computational complexity of subsequent processing while retaining sufficient texture information for feature extraction and matching.

[0092] The specific implementation of the door panel visual automatic inspection system involves the coordinated operation of multiple modules to solve the problems of low image registration accuracy and inconsistent defect detection caused by the three-dimensional curved surface structure of door panels. The system operates in an automated production line environment and achieves high-precision inspection through the following detailed technical solutions.

[0093] The image acquisition module is integrated with the production line conveyor belt, receiving position trigger signals from the conveyor belt encoder via an industrial Ethernet interface. The module sends synchronous acquisition commands to all industrial cameras, each capturing a single-frame color image according to pre-calibrated internal and exposure parameters. These internal parameters, including focal length, principal point, and distortion coefficients, are pre-calculated by photographing a checkerboard calibration board. The captured color images undergo Gaussian filtering to remove noise, and are then converted from the RGB color space to the grayscale color space, generating a uniformly formatted raw image data stream. This synchronous acquisition mechanism eliminates image registration errors caused by object movement, providing high-quality input for subsequent processing.

[0094] After receiving the raw image data stream, the surface modeling module initiates the 3D reconstruction process. The module uses the Harris corner detection algorithm to locate feature points in each image and calculates the gradient direction histogram of the region surrounding each feature point to generate SIFT feature descriptors. Between raw images from different viewpoints, matching is performed by comparing the Euclidean distance of the feature descriptors; when the distance is less than a preset threshold, pairs of corresponding feature points are established. Using the intrinsic and extrinsic parameter matrices calibrated by the camera, triangulation is performed on the pairs of corresponding feature points based on the principle of binocular parallax, and the 3D spatial coordinates are calculated to generate an initial sparse point cloud. A graph-cut-based global optimization algorithm is used to densify the sparse point cloud, increasing its density to cover the door panel surface. Finally, Delaunay triangulation is performed, transforming the point cloud into a 3D surface mesh model with topological connectivity. This model contains vertex coordinates and triangle facet information, providing a geometric basis for projection correction.

[0095] The projection correction module simultaneously receives the original image and the 3D curved surface mesh model. Based on the camera extrinsic matrix, the module reverse-projects the coordinates of each pixel in the original image onto the surface of the 3D mesh model, obtaining 3D coordinate points and assigning the grayscale values ​​of the original image to the corresponding points to form texture associations. A virtual orthogonal camera viewpoint perpendicular to the main plane of the door panel is defined, and the surface texture of the 3D model is reprojected onto a 2D imaging plane under this viewpoint. This process generates a standard corrected image sequence without perspective distortion, eliminating the viewpoint differences caused by the curved surface structure, allowing subsequent registration operations to be performed under a unified viewpoint.

[0096] The dual-path registration module receives a sequence of standard-corrected images and executes two registration schemes in parallel. The SIFT-based registration path extracts scale-invariant features from the standard-corrected and standard template images, locates key points by constructing a Gaussian difference pyramid, and generates a 128-dimensional descriptor. It uses the K-nearest neighbor algorithm for feature matching, employs a random sampling consensus algorithm to remove mismatched point pairs, estimates the homography transformation matrix, and performs geometric transformations on the images to generate the first registration result image. The deep learning-based registration path concatenates image pairs into a dual-channel input tensor, regresses affine transformation parameters through a pre-trained convolutional neural network, decodes the transformation matrix, and generates the second registration result image. Both paths output registration results and confidence scores, providing data support for subsequent optimization.

[0097] After receiving the dual-path registration results, the registration optimization module initiates the evaluation process. The module calculates the normalized mutual information value and structural similarity index between the first and second registered images and the standard template image, serving as objective similarity indicators. The registration confidence scores of the two paths are subjected to max-min normalization to eliminate dimensional differences. The normalized confidence scores and objective indicators are then weighted and summed according to preset weights to obtain a comprehensive score for each path. By comparing the scores, the optimal registered image and transformation matrix are selected to ensure the accuracy and reliability of the registration results.

[0098] The defect detection module performs pixel-by-pixel grayscale difference between the optimal registration image and the standard template image, and generates an initial difference map by taking the absolute value. Otsu's method is applied to adaptively calculate the segmentation threshold, and the difference map is binarized to obtain a preliminary defect region mask. Morphological closing operations are used to fill the internal holes of the defect, opening operations are used to remove edge noise, and finally, connected component analysis is performed to filter out regions with excessively small areas, outputting accurate defect region contour information.

[0099] The classification module extracts image patches from the optimally registered image based on the defect region contour. It calculates the geometric moments of the defect contour to derive rectangularity and aspect ratio as shape features, and simultaneously calculates the gray-level co-occurrence matrix of the image patch to extract energy and contrast as texture features. These combined features are then input into a support vector machine classifier, which uses grid search and cross-validation methods to optimize the kernel function parameters, achieving accurate classification of defect types. Finally, it generates an inspection report containing defect location, category, and size, completing the automated inspection process.

[0100] The system's modules are connected via standardized data interfaces, forming a complete processing chain. The construction of a 3D curved surface mesh model solves the perspective distortion problem, dual-path registration and optimization mechanisms improve image alignment accuracy, and multi-feature fusion classification ensures consistent defect identification. When applied in a door panel production line, this invention effectively distinguishes common defect types such as scratches and dents, providing reliable technical support for quality control.

Claims

1. A door panel visual automatic inspection system, characterized in that, include: The image acquisition module is used to simultaneously acquire raw images of the door panel surface from different perspectives; The surface modeling module is used to receive the original image and calculate and generate a three-dimensional curved surface mesh model of the door panel surface; The projection correction module is used to receive the original image and the three-dimensional curved surface mesh model, map the pixels of the original image onto the surface of the three-dimensional curved surface mesh model, and then reproject it onto a virtual orthogonal projection plane to generate a standard corrected image sequence. The dual-path registration module is used to receive the standard corrected image sequence, perform registration based on SIFT features and registration based on deep learning network regression in parallel, and output the registration results of the two registration paths and the corresponding registration confidence scores. The registration optimization module is used to receive the registration results and registration confidence of the two registration paths, calculate the objective similarity index between each registration result and the pre-stored standard template image, integrate the registration confidence and the objective similarity index to perform a comprehensive score, and select the registration result with the highest score as the optimal registration image for output. The registration selection module is also used for: The normalized mutual information value and structural similarity index (SSIM) between the first registration result image, the second registration result image and the standard template image are calculated respectively, and used as objective similarity indicators. The registration confidence scores of the first path and the second path are transformed to the same numerical range using the max-min normalization method. For each registration path, the normalized registration confidence, normalized mutual information value, and structural similarity index (SSIM) are weighted and summed according to preset weighting factors to obtain a comprehensive score for each path. Compare the comprehensive scores corresponding to the first registration result image and the second registration result image, and determine the registration result image with the higher comprehensive score and the corresponding transformation matrix as the optimal registration image and transformation parameters; The defect detection module is used to receive the optimal registration image, calculate the difference map between the optimal registration image and the standard template image, segment and locate potential defect regions, and output the contour information of the defect regions. The result classification module is used to receive the contour information of the defect area and the optimal registration image, extract the features of the defect area and classify them, and output a detection report containing the defect category and location.

2. The automatic visual inspection system for door panels according to claim 1, characterized in that, The surface modeling module is used for: Corner detection is performed on the received original image, feature points are extracted, and feature descriptors are generated; Between the original images from different viewpoints, the feature points are matched based on descriptor similarity to establish pairs of feature points with the same name; Based on the camera calibration parameters, the three-dimensional spatial coordinates of the corresponding feature point pairs are calculated using the binocular parallax principle to generate an initial sparse point cloud. The initial sparse point cloud is densified by interpolation using a global optimization algorithm to generate a dense three-dimensional point cloud. The dense 3D point cloud is subjected to Delaunay triangulation to construct the 3D surface mesh model with topological connectivity.

3. The automatic visual inspection system for door panels according to claim 1, characterized in that, The dual-path registration module performs SIFT-based registration for: On the standard calibrated image sequence and the pre-stored standard template image, scale-invariant feature transform (SIFT) detection is performed to obtain key points and corresponding SIFT descriptors; Using the K-nearest neighbor algorithm, preliminary matching is performed between key points of the standard corrected image sequence and the standard template image based on the Euclidean distance of the SIFT descriptor; The Random Sample Consensus (RANSAC) algorithm is applied to remove mismatched point pairs from the initial matching results and to estimate the homography transformation matrix that describes the spatial transformation relationship between the two images. The homography transformation matrix is ​​used to perform geometric transformation and resampling on the standard corrected image sequence to generate a first registration result image, and the number of correctly matched point pairs retained after filtering by the RANSAC algorithm is used as the registration confidence of the first path.

4. The automatic visual inspection system for door panels according to claim 3, characterized in that, The dual-path registration module performs registration based on deep learning network regression for: The standard corrected image sequence and the pre-stored standard template image are concatenated in the channel dimension to construct a dual-channel input tensor; The dual-channel input tensor is input into a pre-trained convolutional neural network, and the fully connected layer at the end of the convolutional neural network outputs a six-dimensional vector, which represents the six parameters of the affine transformation. The six-dimensional vector is decoded into a specific affine transformation matrix; The standard corrected image sequence is spatially transformed and interpolated using the affine transformation matrix obtained by decoding to generate a second registration result image, and the activation value of the six-dimensional vector corresponding to the output layer of the convolutional neural network is used as the registration confidence of the second path.

5. The automatic visual inspection system for door panels according to claim 1, characterized in that, The defect detection module is used for: Perform pixel-by-pixel grayscale difference operation on the optimal registration image and the standard template image, and obtain the initial difference map by taking the absolute value; The Otsu method is applied to the initial difference map to adaptively calculate the segmentation threshold, and the initial difference map is binarized using the segmentation threshold to obtain a preliminary defect region mask. Morphological closing and opening operations are sequentially performed on the preliminary defect area mask to fill the small holes inside the defect area and remove edge burrs and isolated noise points; Connected component labeling and analysis are performed on the morphologically processed binary image. The area and minimum bounding rectangle coordinates of each connected component are calculated. Connected components with an area less than a preset threshold are filtered out. The contour pixel coordinates of the remaining connected components are output as the contour information of the defect region.

6. The automatic visual inspection system for door panels according to claim 1, characterized in that, The result classification module extracts defect region features for: Based on the defect region contour information, the corresponding image block is extracted from the optimal registration image; For each defect region, calculate the geometric moments of the defect region contour, and derive the rectangularity and aspect ratio as shape features based on the geometric moments; For each image patch extracted from the optimally registered image, the gray-level co-occurrence matrix of the image patch is calculated, and energy and contrast are extracted from the gray-level co-occurrence matrix as texture features; The shape features and texture features are combined to construct a multidimensional feature vector representing the defect region.

7. The automatic visual inspection system for door panels according to claim 6, characterized in that, The gray-level co-occurrence matrix of the calculated image patch is used for: Based on the gray level of the image block and the preset pixel displacement, the frequency of pixel pairs that meet the preset distance is counted in four directions: 0 degrees, 45 degrees, 90 degrees and 135 degrees, and primary co-occurrence matrices are generated in the four directions respectively. The average of the matrix elements of the primary co-occurrence matrices in the four directions is calculated to generate a comprehensive gray-level co-occurrence matrix; The energy and contrast features are extracted from the comprehensive gray-level co-occurrence matrix.

8. The automatic visual inspection system for door panels according to claim 7, characterized in that, The support vector machine model used by the result classification module for classification is trained for: Collect sample images of door panel defects containing known categories; The multidimensional feature vector is extracted from each defect sample image and labeled with the corresponding defect category label to form a training dataset; The support vector machine model was trained using the training dataset, and the kernel function and penalty parameters of the model were optimized using grid search and cross-validation.

9. The automatic visual inspection system for door panels according to claim 1, characterized in that, The image acquisition module is used for: Receives arrival trigger signals from the conveyor belt of the automated production line; In response to the arrival trigger signal, a synchronous acquisition command is sent to all industrial cameras; Control each industrial camera to synchronously capture a single frame of color image according to pre-calibrated internal and exposure parameters; The captured color image is subjected to Gaussian filtering for noise reduction, and the denoised image is converted from the RGB color space to the grayscale color space.

Images