Automobile interior and exterior decoration detection method and system based on visual fusion

Abstract

The invention discloses an automobile interior and exterior decoration detection method and system based on visual fusion, and relates to the field of automobile door installation and detection. The method comprises the steps of obtaining and preprocessing two-dimensional image data and three-dimensional point cloud data, obtaining a spatial transformation matrix through data registration and generating a three-dimensional edge guide line, clustering and segmenting a vehicle door point cloud set and a vehicle body point cloud set based on the three-dimensional edge guide line, calculating surface difference data and gap data, and obtaining a vehicle door point cloud set and a vehicle body point cloud set; and comparing the surface difference data with the gap data and the tolerance, and generating a detection report. According to the method, the advantages of multi-modal data are fused, the problems of inaccurate edge identification, low measurement precision and the like caused by surface reflection and low texture interference of a traditional single detection method are effectively solved, and high-precision and automatic detection of the automobile door gap surface difference is realized.

Description

Technical Field

[0001] This invention relates to the field of vehicle inspection, and more specifically, to a method and system for inspecting the interior and exterior of automobiles based on vision fusion. Background Technology

[0002] With the automotive manufacturing industry's ever-increasing demands for assembly quality, the surface difference between the gaps between car doors and the body has become a key indicator for measuring the overall vehicle manufacturing quality. In recent years, two-dimensional vision inspection methods and three-dimensional scanning technology have been gradually introduced into this field. Two-dimensional vision inspection methods analyze gap widths by extracting image edges, while three-dimensional point cloud inspection methods calculate surface difference heights through surface fitting. However, in real-world industrial scenarios, due to complex factors such as reflective paint, low-texture components, and multi-angle installation deviations, single-modal inspection methods have significant limitations, making it difficult to achieve high-precision automated full inspection.

[0003] On the one hand, detection methods based on two-dimensional vision are susceptible to interference from ambient lighting and surface reflections, leading to inaccurate edge feature extraction. For example, when detecting vehicles with dark bodies or high-gloss paint, strong reflections can create large areas of overexposure or light spots in the image, covering or interfering with the edge information of gaps. On the other hand, low-texture areas such as door trim lack sufficient image gradient changes, making it difficult for traditional edge detection algorithms to reliably extract effective features, ultimately resulting in deviations in the measured gap values.

[0004] On the other hand, detection methods based on 3D point clouds suffer from data gaps and insufficient accuracy when dealing with highly reflective surfaces and edge localization. For example, when scanning chrome-plated door handles or high-gloss black-painted car doors, the excessively high laser reflectivity can lead to a large number of noise points or areas with missing data in the point cloud. In addition, the density of the point cloud decreases at physical edges and there is a problem of mixed pixels, resulting in low edge localization accuracy when relying solely on point cloud fitting.

[0005] In summary, existing inspection technologies cannot operate stably under the complex and variable lighting and surface conditions of automotive manufacturing environments. There is a lack of a high-precision inspection solution that can integrate the advantages of multimodal data and overcome problems such as surface reflection interference, low texture, and edge blurring. Therefore, the industry urgently needs an automated, high-precision, and highly reliable method for detecting surface differences in car door gaps that can coordinate two-dimensional images and three-dimensional point cloud data to meet the stringent quality control requirements of the automotive manufacturing industry. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a method and system for detecting the interior and exterior trim of automobiles based on visual fusion.

[0007] The first aspect of this invention provides a method for detecting the interior and exterior trim of automobiles based on visual fusion, comprising:

[0008] Acquire 2D image data and 3D point cloud data and perform preprocessing operations;

[0009] The transformation matrix of two-dimensional space is obtained based on two-dimensional image data and three-dimensional point cloud data, and the three-dimensional edge guide line is obtained based on the transformation strategy.

[0010] Clustering is performed based on 3D edge guide lines to obtain door point clouds and body point clouds. Surface difference data and gap data are then obtained from the door point clouds and body point clouds to generate an inspection report.

[0011] According to a preferred embodiment, acquiring two-dimensional image data and three-dimensional point cloud data and performing preprocessing operations includes:

[0012] The edge contrast in the two-dimensional image data is enhanced based on the adaptive histogram equalization algorithm, and the reflection interference in the two-dimensional image data is suppressed based on the homomorphic filtering algorithm. The reflection interference is represented as high-frequency interference caused by reflection from the car door surface.

[0013] Outlier noise points in the 3D point cloud data are filtered out based on statistical filtering, and points in the 3D point cloud data with curvature changes below a preset threshold are filtered out through feature-aware filtering.

[0014] The two-dimensional image data includes two-dimensional image data of the vehicle body and two-dimensional image data of the vehicle door, and the three-dimensional point cloud data includes three-dimensional point cloud data of the vehicle body and three-dimensional point cloud data of the vehicle door.

[0015] According to a preferred embodiment, a two-dimensional spatial transformation matrix is ​​obtained based on two-dimensional image data and three-dimensional point cloud data, and a three-dimensional edge guide line is obtained based on a transformation strategy, including:

[0016] ORB feature extraction and feature matching are performed on 2D image data and 3D point cloud data to obtain optimized correspondence. Based on the optimized correspondence, the 2D and 3D spatial transformation matrix is ​​obtained.

[0017] The Canny operator is used to perform edge detection on the two-dimensional image data to obtain two-dimensional edge data. The Zernike moment algorithm is used to locate the two-dimensional edge data to obtain the sub-pixel two-dimensional edge contour.

[0018] Based on the 2D and 3D transformation matrix, the subpixel 2D edge contour is back-projected into the 3D point cloud data to obtain the 3D edge contour point set. The 3D edge contour point set is then fitted into a continuous 3D space curve to generate a 3D edge guide line.

[0019] According to a preferred embodiment, ORB feature extraction and feature matching are performed on two-dimensional image data and three-dimensional point cloud data to obtain an optimized correspondence. Based on the optimized correspondence, a two-dimensional / three-dimensional spatial transformation matrix is ​​obtained, including:

[0020] The 3D point cloud data is projected to generate 3D depth map data. ORB features are extracted from the 3D depth map data and the 2D image data to obtain 2D ORB feature data and 3D ORB feature data respectively.

[0021] The two-dimensional ORB feature data includes two-dimensional feature point data and two-dimensional descriptors, and the three-dimensional ORB feature data includes three-dimensional feature point data and three-dimensional descriptors.

[0022] Feature matching is performed on two-dimensional and three-dimensional descriptors to establish a preliminary correspondence between two-dimensional feature point data and three-dimensional feature point data. The preliminary correspondence includes multiple sets of matching point pairs.

[0023] The preliminary correspondence is iteratively filtered using a random sampling consensus algorithm to remove incorrect matching point pairs from the preliminary correspondence and obtain an optimized correspondence, which includes multiple sets of correct matching point pairs.

[0024] The two-dimensional spatial transformation matrix between two-dimensional image data and three-dimensional point cloud data is obtained based on the optimized correspondence.

[0025] According to a preferred embodiment, edge detection is performed on two-dimensional image data using the Canny operator to obtain two-dimensional edge data, and the two-dimensional edge data is located using the Zernike moment algorithm to obtain sub-pixel two-dimensional edge contours, including:

[0026] Gaussian filtering is applied to the two-dimensional image data to calculate the gradient magnitude and direction of each pixel in the two-dimensional image data. The gradient magnitude is then locally compared along the gradient direction using a non-maximum suppression algorithm. Local maxima points along the gradient direction are retained, and non-maximum points are filtered out to obtain the single-pixel width edge.

[0027] A dual-threshold hysteresis processing method is used to process single-pixel width edges. Pixels with gradient magnitudes higher than the high threshold are identified as strong edge points, and pixels with gradient magnitudes between the high and low thresholds are identified as weak edge points. After filtering out weak edge points that are not connected to strong edge points, the strong edge points and weak edge points are combined to form two-dimensional edge data.

[0028] Based on the two-dimensional edge data, the Zernike moment algorithm is used to perform convolution calculation on the edge points within a preset neighborhood window to obtain the sub-pixel level offset of the edge points.

[0029] The two-dimensional edge data is corrected based on sub-pixel level offset to obtain the sub-pixel two-dimensional edge contour.

[0030] According to a preferred embodiment, clustering is performed based on three-dimensional edge guide lines to obtain door point clouds and body point clouds. Surface difference data and gap data are then obtained based on the door point clouds and body point clouds to generate an inspection report, including:

[0031] The neighboring point cloud is obtained based on the 3D edge guide line, and the neighboring point cloud is clustered to obtain the door cluster point set and the body cluster point set.

[0032] Obtain the point cloud normal data and curvature data of the door cluster point set and the body cluster point set, and use the door cluster point set and the body cluster point set as seed point sets;

[0033] Region growing is performed on the seed point set based on point cloud normal data and curvature data to obtain the door point cloud and the body point cloud;

[0034] A fitting operation is performed based on the door point cloud and the body point cloud, and the preset measurement section data is obtained at the same time. The surface difference data and gap data are obtained based on the fitting operation and the preset measurement section data.

[0035] An inspection report is generated based on surface difference data, gap data, and preset measurement section data.

[0036] According to a preferred embodiment, a fitting operation is performed based on the door point set and the body point set, and preset measurement section data is obtained simultaneously. Surface difference data and gap data are obtained based on the fitting operation and the preset measurement section data, including:

[0037] Based on the principal component analysis algorithm, plane fitting is performed on the door point cloud and the body point cloud respectively to obtain the first fitting plane and the second fitting plane.

[0038] Wherein, the first fitting plane represents the door surface, and the second fitting plane represents the vehicle body surface;

[0039] A first intersection line is obtained based on the first fitting plane and the preset measurement section, and a second intersection line is obtained based on the second fitting plane and the preset measurement section;

[0040] Surface difference data is obtained based on the first and second fitted planes;

[0041] Gaps data are obtained based on the first and second intersection lines.

[0042] According to a preferred embodiment, preset measurement cross-sectional data includes:

[0043] Several evenly distributed sections are marked on the body of a standard vehicle as measurement sections;

[0044] Obtain a standard vehicle dataset, which includes the surface difference and gap values ​​of a standard vehicle at each measurement section;

[0045] The standard vehicle dataset is processed based on the three sigma criterion to obtain the tolerance range at each measurement section;

[0046] Combine the tolerance range with the measurement section to form preset measurement section data.

[0047] According to a preferred embodiment, an inspection report is generated based on surface difference data, gap data, and preset measurement section data, including:

[0048] Compare the surface difference data and gap data with the tolerance range in the preset measurement section data;

[0049] Obtain surface difference data and gap data within the tolerance range, and add a pass mark to the measurement section in the corresponding preset measurement section data;

[0050] Acquire surface difference data and gap data that are outside the tolerance range, calculate their out-of-tolerance values, and add a failing mark to the measurement section in the corresponding preset measurement section data;

[0051] The preset measurement section data, pass mark, fail mark and corresponding out-of-tolerance value are combined into a visual inspection report;

[0052] A second aspect of the present invention also provides a vision fusion-based automotive interior and exterior trim inspection system, comprising:

[0053] The data preprocessing module is used to acquire two-dimensional image data and three-dimensional point cloud data and perform preprocessing operations.

[0054] A cross-latitude registration module is used to obtain a two-dimensional or three-dimensional spatial transformation matrix based on two-dimensional image data and three-dimensional point cloud data.

[0055] The fusion segmentation module is used to obtain three-dimensional edge guide lines based on a transformation strategy, and perform clustering based on the three-dimensional edge guide lines to obtain door point clouds and body point clouds.

[0056] The result output module is used to obtain surface difference data and gap data based on the door point set and the body point set, and generate an inspection report.

[0057] Based on the above, this application embodiment implements the following: based on the two-dimensional edge data, the Zernike moment algorithm is used to perform convolution calculation on edge points within a preset neighborhood window to obtain the sub-pixel offset of the edge points. Specifically, for each edge point in the two-dimensional edge data, a neighborhood window is defined. Within the neighborhood window, the gray values ​​of the pixels surrounding the edge point are convolved with a Zernike polynomial. The actual position of the edge is fitted by the polynomial, and the sub-pixel offset of the edge point relative to the current pixel center is calculated based on the fitting result. Compared to the problems of inaccurate edge recognition and low edge positioning accuracy in traditional two-dimensional edge data extraction, this step utilizes the orthogonality and rotation invariance of the Zernike polynomial on the unit circle to refine the pixel-level edges extracted by traditional operators such as Canny. It analyzes the gray-level distribution within the neighborhood window of the edge point to fit a more accurate, sub-pixel-level continuous edge contour.

[0058] Based on the 2D / 3D transformation matrix, the subpixel 2D edge contour is back-projected onto the 3D point cloud data to obtain the 3D edge contour point set. The 3D edge contour point set is fitted into a continuous 3D spatial curve to generate a 3D edge guide line. The 2D / 3D spatial transformation matrix is ​​called to back-project each subpixel level coordinate in the subpixel 2D edge contour onto the 3D point cloud data to obtain the 3D spatial coordinates corresponding to each edge point and combine them into a 3D edge contour point set. Based on the B-spline curve, the 3D edge contour point set is curve-fitted. The parameters of the B-spline curve are adjusted to closely match the 3D edge contour point set. Based on the adjusted B-spline curve, a continuous and smooth 3D spatial curve is generated, which is the 3D edge guide line. Compared with the traditional single-modal detection method between the door and the body, which is limited by complex and variable lighting and surface conditions, this step constructs a 3D edge guide line in 3D space through the back projection from 2D space to 3D space, which accurately reflects the edge of the gap between the door and the body.

[0059] This invention integrates the advantages of multimodal data, effectively overcoming the problems of inaccurate edge recognition and low measurement accuracy caused by surface reflection and low texture interference in traditional single detection methods, and realizing high-precision and automated detection of surface differences in car door gaps. Attached Figure Description

[0060] Figure 1 The execution flowchart of the present invention, which is a method for detecting the interior and exterior trim of automobiles based on visual fusion, is presented.

[0061] Figure 2 A schematic diagram of a vision fusion-based automotive interior and exterior trim inspection system of the present invention is shown. Detailed Implementation

[0062] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0063] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0064] like Figure 1 , Figure 2 As shown:

[0065] The first aspect of this invention provides a method for detecting the interior and exterior trim of automobiles based on visual fusion, comprising:

[0066] Step S100: Acquire two-dimensional image data and three-dimensional point cloud data and perform preprocessing operations.

[0067] In this embodiment, step S100 includes:

[0068] Step S110: Enhance the edge contrast in the two-dimensional image data based on the adaptive histogram equalization algorithm, and suppress the reflection interference in the two-dimensional image data based on the homomorphic filtering algorithm. The reflection interference is represented as high-frequency interference caused by reflection from the car door surface.

[0069] Among them, the adaptive histogram equalization algorithm divides the two-dimensional image data into multiple local blocks, calculates the histogram of each local block and adjusts the gray-level distribution. The homomorphic filtering algorithm is based on the image gray level = illumination component + reflection component. It separates the low-frequency illumination component and the high-frequency reflection component, suppresses the high-frequency reflection component, and at the same time retains effective high-frequency information such as door edges and gaps.

[0070] In some possible embodiments, suppose the rear door of a car is being inspected. The rear door is made of ultra-deep-drawn hot-dip galvanized steel sheet with severe surface reflection. Adaptive histogram equalization processing improves the clarity of the boundary contour between the door and the car body, and homomorphic filtering suppresses a large amount of reflective noise.

[0071] Step S120: Outlier noise points in the three-dimensional point cloud data are filtered out based on statistical filtering, and points in the three-dimensional point cloud data with curvature changes below a preset threshold are filtered out through feature-aware filtering. The two-dimensional image data includes vehicle body two-dimensional image data and vehicle door two-dimensional image data, and the three-dimensional point cloud data includes vehicle body three-dimensional point cloud data and vehicle door three-dimensional point cloud data.

[0072] Among them, statistical filtering obtains the deviation from the mean by the distance distribution between each point and its neighboring points, and identifies points whose deviation from the mean exceeds a preset threshold as outliers and filters them out. Feature perception filtering analyzes local point cloud data, identifies points whose curvature changes are below a threshold as smooth region point clouds, and filters them out.

[0073] In some possible embodiments, assuming that the rear door of a car is being inspected, the point cloud data of the door obtained by 3D laser scanning contains a large number of outliers and redundant data. Statistical filtering removes outliers and feature-aware filtering removes redundant points while retaining key features such as the door edge.

[0074] Step S200: Obtain the two-dimensional and three-dimensional spatial transformation matrix based on the two-dimensional image data and the three-dimensional point cloud data, and obtain the three-dimensional edge guide line based on the transformation strategy.

[0075] In this embodiment, step S200 includes:

[0076] Step S210: Perform ORB feature extraction and feature matching on the two-dimensional image data and three-dimensional point cloud data to obtain the optimized correspondence, and obtain the two-dimensional and three-dimensional spatial transformation matrix based on the optimized correspondence.

[0077] In this embodiment, step S210 includes:

[0078] Step S211: Project the 3D point cloud data to generate 3D depth map data. Perform ORB feature extraction on the 3D depth map data and 2D image data to obtain 2D ORB feature data and 3D ORB feature data respectively. The 2D ORB feature data includes 2D feature point data and 2D descriptors, and the 3D ORB feature data includes 3D feature point data and 3D descriptors.

[0079] The process involves projecting 3D point cloud data to generate a depth map, and then using the ORB algorithm to extract features: the FAST corner detection algorithm is used to identify focal points in the image as feature points, and the BRIEF binary descriptor of the feature points is calculated.

[0080] In some possible embodiments, suppose that the rear door of a car is being detected, and ORB features are extracted from the two-dimensional image and the three-dimensional point cloud depth map of the door, respectively. 200-300 feature points are extracted from each image, among which the feature points such as the door edge have the highest density.

[0081] Step S212: Perform feature matching on the two-dimensional descriptor and the three-dimensional descriptor to establish a preliminary correspondence between the two-dimensional feature point data and the three-dimensional feature point data. The preliminary correspondence includes multiple sets of matching point pairs.

[0082] The descriptor is a 32-byte binary vector, and the Hamming distance is the number of corresponding bits that are different between two vectors. Based on two-dimensional feature points, the two three-dimensional feature points with the smallest Hamming distance are found. If the ratio of the smallest distance to the second smallest distance is less than a preset threshold, it is determined to be an initial matching point pair, and feature points with a gray-level variance of less than 10 are filtered out.

[0083] Understandably, a smaller Hamming distance indicates a higher grayscale distribution and local texture similarity of feature points. Feature points with a grayscale variance of less than 10 may be low-texture areas caused by reflection.

[0084] Step S213: The preliminary correspondence is iteratively filtered using a random sampling consensus algorithm to remove incorrect matching point pairs in the preliminary correspondence and obtain an optimized correspondence, which includes multiple sets of correct matching point pairs.

[0085] The process involves obtaining three initial matching point pairs and solving the initial transformation matrix using these three pairs. All initial matching point pairs are then substituted into this matrix to calculate the projection error. If the error is less than a preset threshold, the point is determined to be an interior point. This process of selecting point pairs, solving the initial transformation matrix, and calculating interior points is repeated. The interior point set corresponding to the matrix with the most interior points is selected as the optimized correspondence.

[0086] Understandably, after substituting the two-dimensional feature points into the initial transformation matrix, we obtain the two-dimensional to three-dimensional coordinates. The Euclidean distance between the two-dimensional to three-dimensional coordinates and the coordinates of the corresponding three-dimensional feature point is the projection error.

[0087] Furthermore, the preset threshold for projection error can be set to 0.05mm, depending on the accuracy requirements of the door detection.

[0088] Step S214: Obtain the two-dimensional spatial transformation matrix of the two-dimensional image data and the three-dimensional point cloud data based on the optimized correspondence.

[0089] Among them, the two- or three-dimensional spatial transformation matrix can be obtained by optimizing the correspondence using the least squares method.

[0090] Step S220: The Canny operator is used to perform edge detection on the two-dimensional image data to obtain two-dimensional edge data. The Zernike moment algorithm is used to locate the two-dimensional edge data to obtain the sub-pixel two-dimensional edge contour.

[0091] In this embodiment, step S220 includes:

[0092] Step S221: Perform Gaussian filtering on the two-dimensional image data, calculate the gradient magnitude and direction of each pixel in the two-dimensional image data, and use the non-maximum suppression algorithm to locally compare the gradient magnitude along the gradient direction, retain the local maxima points in the gradient direction, filter out non-maximum points, and obtain the single-pixel width edge.

[0093] Among them, the gradient magnitude is the strength of the gray-scale change around the pixel. The pixel magnitude at the junction of the door and the body is much higher than that of the flat part. The gradient direction is the direction of the most drastic gray-scale change.

[0094] Furthermore, the magnitude of the current pixel is compared with that of its neighboring pixels along the gradient direction, and redundant pixels on both sides of the edge are filtered out, so that the edge retains only a single pixel width.

[0095] Step S222: Use dual threshold hysteresis to process single-pixel width edges. Pixels with gradient magnitudes higher than the high threshold are identified as strong edge points, and pixels with gradient magnitudes between the high and low thresholds are identified as weak edge points. Weak edge points that are not connected to strong edge points are filtered out. Strong edge points and retained weak edge points are combined to form two-dimensional edge data.

[0096] Pixels with gradient magnitudes higher than a threshold are identified as strong edge points, which are definite edge pixels, such as the edge of a car door gap. Pixels with gradient magnitudes between the high and low thresholds are identified as weak edge points, which may be edge extensions or noise. Pixels with gradient magnitudes lower than the low threshold are directly filtered out.

[0097] Furthermore, weak edge points directly connected to strong edge points are considered edge extensions and retained, while weak edge points not directly connected to strong edge points are considered noise points and filtered out. Strong edge points and edge extensions are combined to form complete and continuous two-dimensional edge data.

[0098] Step S223: Based on the two-dimensional edge data, perform convolution calculation on the edge points within a preset neighborhood window using the Zernike moment algorithm to obtain the sub-pixel offset of the edge points.

[0099] Specifically, for each edge point in the two-dimensional edge data, a neighborhood window is defined. Within the neighborhood window, the gray values ​​of the pixels surrounding the edge point are convolved with a Zernike polynomial. The actual position of the edge is fitted by the polynomial, and the sub-pixel offset of the edge point relative to the current pixel center is calculated based on the fitting result.

[0100] Understandably, subpixel-level offset is the slight deviation between the actual position of an edge point and its pixel-level coordinates, with a higher precision than pixel-level offset.

[0101] Furthermore, Zernike moments are rotationally invariant orthogonal moments that can accurately describe the local gray-level distribution of an image through polynomial fitting. They are suitable for locating fine structures such as car door edges, avoiding the influence of rotation angle on edge detection.

[0102] Furthermore, the neighborhood window is a fixed-size pixel area centered on the edge point. The window size needs to cover the gray value change information of the edge point, but it cannot be too large to avoid interference caused by the gray value intrusion of other areas. It can be set to 5*5 to 9*9 based on the edge fineness requirements.

[0103] Step S224: Correct the two-dimensional edge data based on the sub-pixel level offset to obtain the sub-pixel two-dimensional edge contour.

[0104] Understandably, the pixel-level coordinates of each edge point are combined with the corresponding sub-pixel-level offset to obtain the sub-pixel-level coordinates of the edge points. All sub-pixel-level coordinates are sorted according to the gradient direction, redundant edge points are filtered out, and all edge points are fitted into a sub-pixel two-dimensional edge contour based on the sub-pixel-level coordinates.

[0105] Redundant edge points are those with excessively small spacing. Since the small spacing does not improve the accuracy of the edge description, filtering them out can reduce the amount of computation.

[0106] Step S230: Based on the two-dimensional transformation matrix, the sub-pixel two-dimensional edge contour is back-projected into the three-dimensional point cloud data to obtain the three-dimensional edge contour point set. The three-dimensional edge contour point set is then fitted into a continuous three-dimensional space curve to generate a three-dimensional edge guide line.

[0107] Understandably, the two-dimensional spatial transformation matrix obtained in step S214 is called to back-project each sub-pixel level coordinate in the sub-pixel two-dimensional edge contour to the three-dimensional point cloud data, obtain the three-dimensional spatial coordinates corresponding to each edge point and combine them into a three-dimensional edge contour point set, perform curve fitting on the three-dimensional edge contour point set based on the B-spline curve, adjust the parameters of the B-spline curve to closely match the three-dimensional edge contour point set, and generate a continuous and smooth three-dimensional spatial curve based on the adjusted B-spline curve, which is the three-dimensional edge guide line.

[0108] Step S300: Clustering is performed based on the three-dimensional edge guide lines to obtain the door point cloud and the body point cloud, and surface difference data and gap data are obtained based on the door point cloud and the body point cloud to generate an inspection report.

[0109] In this embodiment, step S300 includes:

[0110] Step S310: Obtain the neighboring point cloud based on the three-dimensional edge guide line, perform clustering operation on the neighboring point cloud, and obtain the door cluster point set and the vehicle body cluster point set.

[0111] Understandably, a cylindrical neighborhood is defined centered on the 3D edge guide line, and the point cloud within the cylindrical neighborhood is extracted from the 3D point cloud data, which is the neighboring point cloud.

[0112] The radius of the cylindrical neighborhood is set according to the distance between the door and the body, and it needs to cover the edge points of both types of components.

[0113] Step S320: Obtain the point cloud normal data and curvature data of the door cluster point set and the body cluster point set, and use the door cluster point set and the body cluster point set as seed point sets.

[0114] Understandably, a fixed number of neighborhood points are obtained for each point, and the covariance matrix of the neighborhood points is calculated through principal component analysis. The eigenvector with the smallest eigenvalue is the normal vector of that point. The curvature is calculated based on the eigenvalues ​​of the covariance matrix, and the curvature is the ratio of the smallest eigenvalue to the sum of the eigenvalues.

[0115] The seed point set is the initial point set used for region growth. It has clear component point cloud affiliation and geometric feature information. The component point cloud includes door cluster point set and body cluster point set. The geometric feature information includes point cloud normal data and curvature data. Subsequently, neighboring points are attracted and combined into a complete component point cloud through feature similarity.

[0116] Step S330: Perform region growing on the seed point set based on the point cloud normal data and curvature data to obtain the door point cloud set and the body point cloud set.

[0117] Understandably, the unclassified points in the 3D point cloud data are traversed and the normal angle and curvature difference between them and all points in the seed point set are calculated. If the normal angle between the unclassified point and any seed point is less than a preset threshold and the curvature difference is less than the threshold, then the unclassified point is determined to have the same surface features as the seed point and is classified into the same component point cloud.

[0118] In some possible embodiments, assuming that an unclassified point and a seed point are used for region growing, and the seed point belongs to the door cluster, the angle between their normals is 7°, which is less than the normal angle threshold of 8°, and the curvature difference is 0.004, which is less than the curvature difference threshold of 0.005, then it is determined that the unclassified point belongs to the door cluster.

[0119] Step S340: Perform a fitting operation based on the door point cloud and the body point cloud, and simultaneously acquire preset measurement section data. Obtain surface difference data and gap data based on the fitting operation and the preset measurement section data.

[0120] In this embodiment, step S340 includes:

[0121] Step S341: Based on the principal component analysis algorithm, perform plane fitting on the door point cloud and the body point cloud respectively to obtain the first fitting plane and the second fitting plane, wherein the first fitting plane represents the door surface and the second fitting plane represents the body surface.

[0122] Understandably, the mean coordinates of all points in the door point cloud are calculated, and the mean coordinates of each point are subtracted from the mean coordinates to obtain the centered coordinates. A covariance matrix is ​​constructed based on the centered coordinates, and three eigenvalues ​​are obtained through eigenvalue decomposition. The direction with the smallest eigenvalue is the normal direction of the point cloud surface. Based on the normal direction and the mean coordinates, a first fitting plane representing the door surface is constructed. The same operation is performed on the vehicle body point cloud to construct a second fitting plane representing the vehicle body surface. The three-dimensional coordinate data of the first and second fitting planes on a standard vehicle are then obtained.

[0123] Step S342: Obtain the predicted cross-sectional data.

[0124] In this embodiment, step S342 includes:

[0125] Step S3421: Mark several evenly distributed sections on the body of the standard vehicle as measurement sections.

[0126] Understandably, the measurement section is a rectangle perpendicular to the car door surface. Several measurement sections are evenly selected along the vertical and horizontal directions of the car door to avoid the sections being concentrated in a certain area. The measurement sections need to cover the key areas where the car door and the car body connect, ensuring that each section can reflect the surface difference and gap values ​​of different areas, and obtain the three-dimensional coordinate data of each measurement section on a standard vehicle.

[0127] Key areas where the door connects to the body may include the top edge of the door, the handle, the bottom edge, the hinge, and the latch.

[0128] The three-dimensional coordinate data includes the coordinates of the center point and the normal vector.

[0129] In some possible embodiments, assuming a standard vehicle A, three sections are defined along the height direction of the left front door of the standard vehicle A: the top section, 50mm from the top edge of the door, named measurement section 1; the middle section, passing through the door handle, named measurement section 2; and the bottom section, 50mm from the bottom edge of the door, named measurement section 3. Two sections are defined along the width direction of the door: the left section, passing through the door hinge, named measurement section 4; and the right section, passing through the door latch, named measurement section 5. There are a total of 5 measurement sections, each of which is a rectangle with a width of 50mm. The three-dimensional coordinate data of each section is recorded. For example, the coordinates of the center point of measurement section 2 are (1260.0mm, 890.0mm, 450.0mm), and the normal vector is (0, 0.866, 0.5).

[0130] Step S3422: Obtain a standard vehicle dataset, which includes the surface difference value and gap value of a standard vehicle at each measurement section.

[0131] Understandably, several standard vehicles of the same model and batch are selected, and the surface difference and gap values ​​of each standard vehicle at each measurement section are obtained. This information is then combined into a standard vehicle dataset.

[0132] In some possible embodiments, assume a standard vehicle A has a surface difference of 0.25 mm and a gap of 0.30 mm measured at measurement section 1, and the standard vehicle dataset is [standard vehicle A, measurement section 1, 0.25 mm, 0.30 mm].

[0133] Step S3423: Process the standard vehicle dataset based on the three sigma criterion to obtain the tolerance range at each measurement section.

[0134] The process involves obtaining the surface difference value for each measurement section, calculating the mean and standard deviation of the surface difference values ​​for the same measurement section across all standard vehicles, calculating the upper and lower limits of the surface difference tolerance using the three sigma criterion, obtaining the gap value for each measurement section, calculating the mean and standard deviation of the gap value for the same measurement section across all standard vehicles, calculating the upper and lower limits of the gap value tolerance using the three sigma criterion, and finally merging the upper and lower limits of the surface difference tolerance into a tolerance range.

[0135] Understandably, the three sigma criterion is a statistical method used to exclude extreme outliers. It assumes that the data follows a normal distribution and that most data will fall within the range of "mean ± 3 standard deviations". This range can be used as a reasonable tolerance range. If the calculated lower tolerance limit is negative, then the lower tolerance limit is set to 0.

[0136] In some possible embodiments, assuming the mean surface difference of the left front door of 100 standard vehicles at measurement section 2 is 0.2mm and the standard deviation is 0.05mm, according to the three sigma criterion, the upper limit of the surface difference tolerance is 0.20 + 3 * 0.05 = 0.35mm and the lower limit of the surface difference tolerance is 0.20 - 3 * 0.05 = 0.05mm. Assuming the mean gap of the left front door of 100 standard vehicles at measurement section 2 is 0.3mm and the standard deviation is 0.05mm, according to the three sigma criterion, the upper limit of the gap tolerance is 0.30 + 3 * 0.05 = 0.45mm and the lower limit of the gap tolerance is 0.30 - 3 * 0.05 = 0.15mm. Then the tolerance range is [0.35mm ≥ tolerance data ≥ 0.05mm, 0.45mm ≥ gap data ≥ 0.15mm].

[0137] Step S3424: Combine the tolerance range and the measurement section into preset measurement section data.

[0138] The preset measurement section data includes name, three-dimensional coordinate data, and tolerance range. The data can be stored in structured formats such as tables and JSON for easy retrieval in subsequent steps.

[0139] Step S343: Obtain the first intersection line based on the first fitting plane and the preset measurement section, and obtain the second intersection line based on the second fitting plane and the preset measurement section.

[0140] Specifically, the equations of the first and second fitting planes are calculated based on the three-dimensional coordinate data obtained in step S341, the equation of the measurement section is calculated based on the three-dimensional coordinate data obtained in step S3421, the equation of the first intersection line is obtained by simultaneously solving the equations of the first fitting plane and the measurement section, and the equation of the first intersection line is solved by simultaneously solving the equations of the second fitting plane and the measurement section.

[0141] Furthermore, the direction vector of the intersection line is the cross product of the normal vectors of the two planes. By substituting any common point of the two planes into the equation, the specific location of the intersection line can be obtained.

[0142] In some possible embodiments, assume there exists a measurement section 2 for the left front door, perpendicular to the door surface, with center coordinates (1260.0mm, 890.0mm, 450.0mm) and normal vector (0.99, -0.02, 0.05). Then the equation of the measurement section is 0.99x - 0.02y + 0.05z - 1245.3 = 0. Solving the system of equations for the first fitted plane (0.02x + 0.05y - 0.99z + 430.2 = ...), we get the equation of the first fitted plane. 0) and the equation of the measuring section, the direction vector is (0.99, -0.02, 0.05)*(0.02, 0.05, -0.99)=(0.019, 0.980, 0.050), select the common point (1260.0mm, 890.0mm, 450.0mm), and obtain the straight line equation of the first intersection line x=1260.0+0.019t, y=890.0+0.980t, z=450.0+0.050t.

[0143] Step S344: Obtain surface difference data based on the first fitting plane and the second fitting plane, and obtain gap data based on the first intersection line and the second intersection line.

[0144] Understandably, the normal distance between the first fitting plane and the second fitting plane at the preset measurement cross-section is calculated, and the absolute value of this normal distance is taken as the measurement cross-section.

[0145] surface difference data

[0146] Furthermore, the first intersection line and the second intersection line are projected onto the two-dimensional plane where the preset measurement section is located to obtain two two-dimensional projection lines. The average width between the two two-dimensional projection lines is calculated, and the average width is used as the gap data of the measurement section.

[0147] Step S350: Generate an inspection report based on the surface difference data, gap data, and preset measurement section data.

[0148] In this embodiment, step S350 includes:

[0149] Step S351: Compare the surface difference data and gap data with the tolerance range in the preset measurement section data.

[0150] Understandably, if the surface difference data of the measured section is greater than or equal to the lower limit of the surface difference tolerance and less than or equal to the upper limit of the surface difference tolerance, the surface difference data is considered to be qualified. If the surface difference data is less than the lower limit of the surface difference tolerance or greater than the upper limit of the surface difference tolerance, the surface difference data is considered to be unqualified.

[0151] Furthermore, if both the surface difference data and the gap data of a certain measurement section are qualified, then the measurement section as a whole is qualified; if either data is unqualified, then the section as a whole is unqualified. The section number, surface difference data, and gap data are merged into a measurement section dataset.

[0152] In some possible embodiments, it is assumed that there is a measurement section 2 for the left front door, with a tolerance range of [0.35mm ≥ tolerance data ≥ 0.05mm, 0.45mm ≥ gap data ≥ 0.15mm]. The surface difference data at measurement section 2 is 0.25mm, 0.35mm ≥ 0.25mm ≥ 0.05mm, which is acceptable. The gap data is 0.30mm, 0.45mm ≥ 0.30mm ≥ 0.15mm, which is also acceptable. Then, measurement section 2 is generally acceptable, and the measurement section dataset [measurement section 2, 0.25mm, 0.30mm] is output.

[0153] Step S352: Obtain the surface difference data and gap data within the tolerance range, and add a pass mark to the measurement section in the corresponding preset measurement section data.

[0154] It is understandable that a pass mark and out-of-tolerance data are added to the overall qualified cross-section dataset, and the surface difference out-of-tolerance data and gap out-of-tolerance data are all recorded as 0mm.

[0155] In some possible embodiments, it is assumed that there is a measurement section 2 for the left front door, and the cross-sectional dataset of measurement section 2 is [measurement section 2, 0.25mm, 0.30mm]. After comparison, if the overall result is qualified, a pass mark is added to the cross-sectional dataset, and the cross-sectional dataset is [measurement section 2, qualified, 0.25mm, 0mm, 0.30mm, 0mm].

[0156] Step S353: Obtain surface difference data and gap data outside the tolerance range, calculate their out-of-tolerance values, and add a failing mark and out-of-tolerance value to the measurement section in the corresponding preset measurement section data.

[0157] Specifically, if the surface difference data is greater than the upper limit of the surface difference tolerance, the excess surface difference value = surface difference data - upper limit of surface difference; if the surface difference data is less than the lower limit of the surface difference tolerance, the excess surface difference value = lower limit of tolerance - surface difference data; if the gap data is greater than the upper limit of the gap value tolerance, the excess gap value = gap data - upper limit of gap value; if the gap data is less than the lower limit of the gap value tolerance, the excess gap value = lower limit of tolerance - gap data.

[0158] Understandably, out-of-tolerance values ​​reflect the degree of numerical deviation, and all out-of-tolerance values ​​are positive.

[0159] In some possible embodiments, assume there is a measurement section 3 for the left front door, with a tolerance range of [0.35mm ≥ tolerance data ≥ 0.05mm, 0.45mm ≥ gap data ≥ 0.15mm]. The cross-sectional data set for measurement section 3 is [measurement section 3, 0.4mm, 0.30mm]. If the surface difference data is found to be unqualified, the entire structure is considered unqualified. Since the surface difference data exceeds the upper limit of the surface difference tolerance, the excess surface difference is 0.4mm - 0.35mm = 0.05mm.

[0160] Then add a failing mark to the cross-section dataset, which is [Measurement cross-section 3, Fail, 0.4mm, 0.05mm, 0.30mm, 0mm].

[0161] Step S354: Combine the preset measurement section data, pass mark, fail mark and corresponding out-of-tolerance value into a visual inspection report.

[0162] A second aspect of the present invention provides a mounting system for automotive central control electronic components, comprising:

[0163] The data preprocessing module is used to acquire two-dimensional image data and three-dimensional point cloud data and perform preprocessing operations.

[0164] A cross-latitude registration module is used to obtain a two-dimensional or three-dimensional spatial transformation matrix based on two-dimensional image data and three-dimensional point cloud data.

[0165] The fusion segmentation module is used to obtain three-dimensional edge guide lines based on a transformation strategy, and perform clustering based on the three-dimensional edge guide lines to obtain door point clouds and body point clouds.

[0166] The result output module is used to obtain surface difference data and gap data based on the door point set and the body point set, and generate an inspection report.

[0167] Based on the above, this invention integrates the advantages of multimodal data, effectively overcoming the problems of inaccurate edge recognition and low measurement accuracy caused by surface reflection and low texture interference in traditional single detection methods, and realizing high-precision and automated detection of surface differences in automotive door gaps.

[0168] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method and system for detecting the interior and exterior trim of automobiles based on vision fusion, characterized in that, The method includes: Acquire 2D image data and 3D point cloud data and perform preprocessing operations; The transformation matrix of two-dimensional space is obtained based on two-dimensional image data and three-dimensional point cloud data, and the three-dimensional edge guide line is obtained based on the transformation strategy. Clustering is performed based on 3D edge guide lines to obtain door point clouds and body point clouds. Surface difference data and gap data are then obtained from the door point clouds and body point clouds to generate an inspection report.

2. The method for detecting automotive interior and exterior trim based on visual fusion according to claim 1, characterized in that, Acquire 2D image data and 3D point cloud data and perform preprocessing operations, including: The edge contrast in the two-dimensional image data is enhanced based on the adaptive histogram equalization algorithm, and the reflection interference in the two-dimensional image data is suppressed based on the homomorphic filtering algorithm. The reflection interference is represented as high-frequency interference caused by reflection from the car door surface. Outlier noise points in the 3D point cloud data are filtered out based on statistical filtering, and points in the 3D point cloud data with curvature changes below a preset threshold are filtered out through feature-aware filtering. The two-dimensional image data includes two-dimensional image data of the vehicle body and two-dimensional image data of the vehicle door, and the three-dimensional point cloud data includes three-dimensional point cloud data of the vehicle body and three-dimensional point cloud data of the vehicle door.

3. The method for detecting automotive interior and exterior trim based on visual fusion according to claim 1, characterized in that, Based on 2D image data and 3D point cloud data, a 2D / 3D spatial transformation matrix is ​​obtained, and a 3D edge guide line is obtained based on the transformation strategy, including: ORB feature extraction and feature matching are performed on 2D image data and 3D point cloud data to obtain optimized correspondence. Based on the optimized correspondence, the 2D and 3D spatial transformation matrix is ​​obtained. The Canny operator is used to perform edge detection on the two-dimensional image data to obtain two-dimensional edge data. The Zernike moment algorithm is used to locate the two-dimensional edge data to obtain the sub-pixel two-dimensional edge contour. Based on the 2D and 3D transformation matrix, the subpixel 2D edge contour is back-projected into the 3D point cloud data to obtain the 3D edge contour point set. The 3D edge contour point set is then fitted into a continuous 3D space curve to generate a 3D edge guide line.

4. The method for detecting automotive interior and exterior trim based on visual fusion according to claim 3, characterized in that, ORB feature extraction and feature matching are performed on 2D image data and 3D point cloud data to obtain optimized correspondences. Based on the optimized correspondences, the 2D and 3D spatial transformation matrices are obtained, including: The 3D point cloud data is projected to generate 3D depth map data. ORB features are extracted from the 3D depth map data and the 2D image data to obtain 2D ORB feature data and 3D ORB feature data respectively. The two-dimensional ORB feature data includes two-dimensional feature point data and two-dimensional descriptors, and the three-dimensional ORB feature data includes three-dimensional feature point data and three-dimensional descriptors. Feature matching is performed on two-dimensional and three-dimensional descriptors to establish a preliminary correspondence between two-dimensional feature point data and three-dimensional feature point data. The preliminary correspondence includes multiple sets of matching point pairs. The preliminary correspondence is iteratively filtered using a random sampling consensus algorithm to remove incorrect matching point pairs from the preliminary correspondence and obtain an optimized correspondence, which includes multiple sets of correct matching point pairs. The two-dimensional spatial transformation matrix between two-dimensional image data and three-dimensional point cloud data is obtained based on the optimized correspondence.

5. The method for detecting automotive interior and exterior trim based on visual fusion according to claim 3, characterized in that, The Canny operator is used to perform edge detection on the two-dimensional image data to obtain two-dimensional edge data. The Zernike moment algorithm is then used to locate the two-dimensional edge data to obtain sub-pixel two-dimensional edge contours, including: Gaussian filtering is applied to the two-dimensional image data to calculate the gradient magnitude and direction of each pixel in the two-dimensional image data. The gradient magnitude is then locally compared along the gradient direction using a non-maximum suppression algorithm. Local maxima points along the gradient direction are retained, and non-maximum points are filtered out to obtain the single-pixel width edge. A dual-threshold hysteresis processing method is used to process single-pixel width edges. Pixels with gradient magnitudes higher than the high threshold are identified as strong edge points, and pixels with gradient magnitudes between the high and low thresholds are identified as weak edge points. After filtering out weak edge points that are not connected to strong edge points, the strong edge points and weak edge points are combined to form two-dimensional edge data. Based on the two-dimensional edge data, the Zernike moment algorithm is used to perform convolution calculation on the edge points within a preset neighborhood window to obtain the sub-pixel level offset of the edge points. The two-dimensional edge data is corrected based on sub-pixel level offset to obtain the sub-pixel two-dimensional edge contour.

6. The method for detecting automotive interior and exterior trim based on visual fusion according to claim 1, characterized in that, Clustering is performed based on 3D edge guide lines to obtain door point clouds and body point clouds. Surface difference data and gap data are then obtained from the door point clouds and body point clouds to generate an inspection report, including: The neighboring point cloud is obtained based on the 3D edge guide line, and the neighboring point cloud is clustered to obtain the door cluster point set and the body cluster point set. Obtain the point cloud normal data and curvature data of the door cluster point set and the body cluster point set, and use the door cluster point set and the body cluster point set as seed point sets; Region growing is performed on the seed point set based on point cloud normal data and curvature data to obtain the door point cloud and the body point cloud; A fitting operation is performed based on the door point cloud and the body point cloud, and the preset measurement section data is obtained at the same time. The surface difference data and gap data are obtained based on the fitting operation and the preset measurement section data. An inspection report is generated based on surface difference data, gap data, and preset measurement section data.

7. The method for detecting automotive interior and exterior trim based on visual fusion according to claim 6, characterized in that, A fitting operation is performed based on the door point cloud and the body point cloud, and preset measurement section data is obtained simultaneously. Surface difference data and gap data are obtained based on the fitting operation and the preset measurement section data, including: Based on the principal component analysis algorithm, plane fitting is performed on the door point cloud and the body point cloud respectively to obtain the first fitting plane and the second fitting plane. Wherein, the first fitting plane represents the door surface, and the second fitting plane represents the vehicle body surface; A first intersection line is obtained based on the first fitting plane and the preset measurement section, and a second intersection line is obtained based on the second fitting plane and the preset measurement section; Surface difference data is obtained based on the first and second fitted planes; Gaps data are obtained based on the first and second intersection lines.

8. The method for detecting automotive interior and exterior trim based on visual fusion according to claim 7, characterized in that, Preset measurement section data, including: Several evenly distributed sections are marked on the body of a standard vehicle as measurement sections; Obtain a standard vehicle dataset, which includes the surface difference and gap values ​​of a standard vehicle at each measurement section; The standard vehicle dataset is processed based on the three sigma criterion to obtain the tolerance range at each measurement section; Combine the tolerance range with the measurement section to form preset measurement section data.

9. The method and system for detecting automotive interior and exterior trim based on vision fusion according to claim 6, characterized in that, An inspection report is generated based on surface difference data, gap data, and preset measurement section data, including: Compare the surface difference data and gap data with the tolerance range in the preset measurement section data; Obtain surface difference data and gap data within the tolerance range, and add a pass mark to the measurement section in the corresponding preset measurement section data; Acquire surface difference data and gap data that are outside the tolerance range, calculate their out-of-tolerance values, and add a failing mark to the measurement section in the corresponding preset measurement section data; The preset measurement section data, pass mark, fail mark and corresponding out-of-tolerance value are combined into a visual inspection report.

10. A vision fusion-based automotive interior and exterior trim inspection system, employing the method described in any one of claims 1 to 9, characterized in that, include: The data preprocessing module is used to acquire two-dimensional image data and three-dimensional point cloud data and perform preprocessing operations. A cross-latitude registration module is used to obtain a two-dimensional or three-dimensional spatial transformation matrix based on two-dimensional image data and three-dimensional point cloud data. The fusion segmentation module is used to obtain three-dimensional edge guide lines based on a transformation strategy, and perform clustering based on the three-dimensional edge guide lines to obtain door point clouds and body point clouds. The result output module is used to obtain surface difference data and gap data based on the door point set and the body point set, and generate an inspection report.

Images