A wheel hub defect detection method and system based on visual multi-features

Abstract

This application provides a method and system for detecting wheel hub defects based on multiple visual features, relating to the field of wheel hub visual inspection technology. The method includes: simultaneously acquiring RGB images and 3D point clouds of the wheel hub using a color camera and a 3D profilometer; projecting the 3D point cloud to obtain a multi-view projection; downsampling the acquired RGB images and 3D point cloud; extracting 3D point cloud features, RGB image features, and multi-view projection features; storing the three features contained in each training sample into the corresponding feature library; obtaining the three features of the test sample using the same steps; calculating feature scores based on the feature libraries; and determining defects based on defect thresholds. This application establishes multiple feature libraries, solving the problems of inaccurate detection and localization from single data sources and insufficient abnormal data. It classifies and identifies defects through overall feature differences, without relying on local high-gradient features. Various features are identified independently without the need for feature fusion.

Description

Technical Field

[0001] This application relates to the field of wheel hub visual inspection technology, and more specifically, to a wheel hub defect detection method and system based on multiple visual features. Background Technology

[0002] As a core component of the vehicle's driving system, the quality of the wheel hub directly affects driving safety and comfort. Defects such as cracks, porosity, and wear are major causes of wheel hub breakage at high speeds, and traditional manual visual inspection has a high rate of missed detections. With the transformation of automobile manufacturing towards "intelligent and automated" production lines and increased production speeds, traditional inspection methods cannot keep up with high-speed assembly line operations, resulting in low inspection accuracy and high inspection costs. To improve the efficiency of wheel hub defect detection by identifying defects through machine vision, the following are some typical directions for technological improvement in existing technologies: (1) Single Feature Comparison – Comparison with the Sample Itself: Chinese Invention Patent CN114972325A discloses an image processing-based method for detecting defects in automobile wheel hubs, including acquiring X-ray images of the wheel hub; obtaining difference pixels; acquiring unstructured pixels of the difference pixels; obtaining defect and non-defect periodic regions based on the frequency of unstructured pixels in each periodic region… calculating the defect probability of each pixel in the defect periodic region based on the noise compliance rate of the non-defect periodic region and the noise description weight of the non-defect periodic region; and obtaining the defect region based on the defect probability of each pixel. Chinese Invention Patent CN119359639A also adopts a similar technical solution. However, this solution is highly dependent on the structural regularity and symmetry of the wheel hub, and also on specific defect types that exhibit high gradients in image recognition, and the defects identified by a single feature are limited.

[0003] (2) Single Feature Comparison – Compared with the Original Design: Chinese Invention Patent CN118037661A discloses a method, device, equipment, storage medium, and product for detecting apparent defects in wheel hubs. It aligns the template wheel hub image corresponding to the template wheel hub with the test wheel hub image corresponding to the test wheel hub to obtain the image to be inspected; candidate defect areas are determined in the overall difference map between the image to be inspected and the template wheel hub image. This scheme also suffers from the problem of relying on specific defect types that exhibit high gradients in image recognition, and the limited number of defects identified by a single feature.

[0004] (3) Multi-feature comparison – same-position matching: Chinese invention patent CN114419038A discloses a method and device for identifying wheel hub surface defects, a storage medium, and an electronic device. It collects two-dimensional images of the sample under test to obtain a first candidate defect list, collects three-dimensional information to obtain a second candidate defect list, and identifies the same position based on the first and second candidate defect lists to determine whether the target wheel hub has a defect. Chinese invention patent CN117095002A also adopts a similar scheme. This scheme requires different features to have similar identification capabilities for the same defect, which has a fundamental flaw in the underlying logic.

[0005] (4) Multi-feature fusion: Chinese invention patent with publication number CN118172308A discloses a method, device, electronic device and storage medium for detecting wheel hub surface defects by fusing attention mechanism and deformable convolution, including inputting an enhanced dataset into a pre-trained neural network to obtain the detection result of the wheel hub to be detected; wherein, the pre-trained neural network includes a feature extraction module that fuses multimodal attention mechanism and a multi-channel feature extraction module that fuses deformable convolution.

[0006] Although the actual method employed is to enhance features from the same source, it effectively possesses the ability to jointly identify features from multiple sources. The difficulty of multi-feature fusion identification schemes lies in determining the weights of multiple features during feature concatenation; this scheme does not address this issue but instead directly concatenates the features.

[0007] Given the shortcomings of existing wheel hub defect identification schemes, an effective wheel hub defect detection method is still needed. Summary of the Invention

[0008] To address the aforementioned problems, this application employs a method for detecting wheel hub defects based on visual multi-features, comprising the following steps: Data acquisition: RGB images and 3D point clouds of the wheel hub are acquired simultaneously using a color camera and a 3D profilometer; Data preprocessing: Project the 3D point cloud onto multiple views to obtain multi-view projection, and downsample the acquired RGB image and 3D point cloud; Feature extraction: A deep convolutional neural network is used to extract 3D point cloud features; a residual neural network is used to extract RGB image features; a lightweight convolutional neural network is used to extract multi-view projection features, and then feature aggregation is performed on the multi-view projection features; Feature library construction: Store the 3D point cloud features, RGB image and multi-view projection features contained in each training sample into the corresponding feature library; Defect identification: Following the above steps of data acquisition, data preprocessing, and feature extraction, three features of the test sample are obtained. Feature scores are calculated based on the feature library, and defect judgment is made based on the defect threshold. Test samples identified as defective are segmented and located, and the features of defective test samples are stored in the defect library. The features of test samples identified as normal are stored in the feature library, and the size of the feature library is controlled by the fast Bayesian core set sampling method.

[0009] Optionally, projecting the 3D point cloud onto multiple views to obtain a multi-view projection includes: projecting the 3D point cloud onto the same image physical coordinate system as the 2D RGB image based on the extrinsic and intrinsic parameter matrices of the color camera to obtain a reference projection view; using the plane where the reference view is located as the reference plane, performing a rotation transformation on the reference view using a rotation function to obtain a rotated projection view, generating an even number of rotated projection views of the point cloud on different planes, with the absolute value of the rotation angle not exceeding pi / 12; Data preprocessing also includes: for multi-view projection, normalizing the reference view and the projected view obtained by rotation transformation and enhancing the contrast of CLAHE, and using bilateral filtering to reduce noise in the image while preserving edge information; then the segmentation module divides each view into non-overlapping patches of a fixed size.

[0010] Optionally, the data preprocessing of the RGB image includes using the nearest neighbor interpolation algorithm on the RGB image, downsampling the R, G, and B channels respectively, and then using height threshold filtering, normal vector consistency filtering, and connected component analysis to remove low-profile workbench background, workbench surface background, and isolated noise. Finally, the downsampled image is subjected to a bicubic interpolation algorithm.

[0011] Optionally, the data preprocessing of the 3D point cloud includes downsampling using the volumetric centroid method, followed by height threshold filtering, normal vector consistency filtering, and connected component analysis to remove low-profile workbench background, workbench surface background, and isolated noise points.

[0012] Optionally, using a residual neural network to extract RGB image features includes: The RGB image after data preprocessing is subjected to two convolution processes to increase the number of input channels; Then it enters the first level, which contains two multi-scale branches. After feature extraction from the two branches, it outputs high-resolution features after feature concatenation, channel compression, and residual connection. Then, it enters the second level, uses convolutional layers for downsampling, and then goes through three multi-scale branches to obtain medium-resolution features; Next, we move to the third level, where we use convolutional layers for downsampling, and then go through four multi-scale branches to obtain low-resolution features. The feature maps of the second and third levels are upsampled to the same size as those of the first level and stitched together to form a feature map. The number of channels is then compressed through the fusion module. Finally, after downsampling and reshaping, a patch-level feature map is obtained.

[0013] Optionally, using a deep convolutional neural network to extract 3D point cloud features includes: The downsampled 3D point cloud is divided into... Individual voxel meshes are used to aggregate the features of each voxel mesh. The output after meshing the entire point cloud is as follows: , The number of non-empty voxels; The voxel mesh is converted into a dense tensor, and the overall geometric features are extracted from it using 3D convolution, outputting a geometric feature tensor. by Centered on a non-empty voxel, with a radius of... The neighborhood voxels within the search radius form One patch; The features of each patch are used as input to the spatial attention mechanism network, and spatial attention is applied to each patch. Max pooling aggregation is performed to obtain patch features of the specified dimensions.

[0014] Optionally, using a lightweight convolutional neural network to extract multi-view projection features includes: sequentially passing the segmented patch through a first convolutional layer, a max pooling layer, a second convolutional layer, a max pooling layer, a third convolutional layer, an average pooling layer, and an L2 normalization layer for feature extraction; After feature extraction, the feature map is reconstructed by view alignment. The rotation function is used to align all rotated projected view features to the reference view coordinate system. After alignment, the position code of each position is added to the feature. The weight is calculated based on the feature variance of each view, and then a weighted average is performed to achieve cross-view fusion. The fused feature vector is reconstructed into a feature map, thus completing the feature aggregation of multi-view projection features.

[0015] Optionally, calculating feature scores based on the feature library includes: Let, For point cloud feature library, For RGB image feature library, For multi-view image feature library, For any feature library, This is a type of sample data for testing samples. From The feature set extracted from the dataset is used to calculate the feature scores according to the following formula: ; In the formula, The feature score for a specific patch of the sample. For any feature in a feature library, for In a single patch feature, for For each Patch feature, iterate through the feature library. M , to match the feature library Features with minimal differences Recorded as ,exist Of all the patch features, with The most different features Recorded as .

[0016] Optionally, defect determination based on defect thresholds includes: determining whether a defect exists based on the 3D point cloud features, RGB image features, and multi-view projection features of the test sample according to the defect threshold; if two or more features indicate that the sample has a defect, it is directly determined to have a defect; if only one feature indicates that the sample has a defect, manual intervention is introduced to identify and determine the final result; if it is determined to be a defect, the features of the test sample are stored in the defect database; if it is determined not to be a defect, the features of the test sample are stored in the three feature databases. The defect threshold is the sum of the mean of the feature scores and twice the standard deviation.

[0017] This application also provides a wheel hub defect detection system based on visual multi-features, applicable to implementing any of the aforementioned wheel hub defect detection methods based on visual multi-features, including: Data acquisition module: used to simultaneously acquire RGB images and 3D point clouds of the wheel hub based on a color camera and a 3D profilometer; Data preprocessing module: used to project 3D point clouds onto multiple views to obtain multi-view projection, and to downsample the acquired RGB images and 3D point clouds; Feature extraction module: used to extract 3D point cloud features using deep convolutional neural networks; extract RGB image features using residual neural networks; extract multi-view projection features using lightweight convolutional neural networks, and then perform feature aggregation on the multi-view projection features; Defect identification module: It calls the data acquisition module, data preprocessing module, and feature extraction module to obtain three features of the test sample, calculates the feature score based on the feature library module, and makes a defect judgment based on the defect threshold; it segments and locates the defects of the test samples identified as defective, and stores the features of the defective test samples into the defect library module; it stores the features of the test samples identified as normal into the feature library module. Feature library module: used to store the three features contained in the training samples and the test samples identified as normal, and to control the size of the feature library using the fast Bayesian core set sampling method. Defect Library Module: Used to store the three features contained in test samples identified as defects.

[0018] The beneficial effects of the wheel hub defect detection method and system based on visual multi-feature provided in this application are as follows: This application establishes a feature library based on RGB image features, 3D point cloud features, and multi-view projection features. Feature scores are calculated based on the feature library, and defect judgment is performed based on defect thresholds. This solves the problem of inaccurate detection and localization from a single data source. By using normal samples to construct the feature library, the problem of insufficient abnormal data is solved. Classification and recognition are performed based on overall feature differences, without relying on local high-gradient features. Moreover, various features are recognized independently, eliminating the problem of different features having different recognition capabilities for the same defect. Furthermore, there is no need to fuse different features, and there is no problem of difficulty in determining feature splicing weights. This effectively utilizes various data sources and improves detection accuracy. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0020] Figure 1 This is a flowchart of the wheel hub defect detection method based on visual multi-features provided in the embodiments of this application; Figure 2 This is a diagram of the RGB image feature extraction network structure provided in the embodiments of this application; Figure 3 This is a diagram of the point cloud feature extraction network structure provided in the embodiments of this application; Figure 4 This is a diagram of the multi-view image feature extraction network structure provided in the embodiments of this application. Detailed Implementation

[0021] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0022] like Figure 1 As shown, this application provides a wheel hub defect detection method based on visual multi-features, which performs the following during the training phase: Data acquisition: RGB images and 3D point clouds of the wheel hub are acquired simultaneously using a color camera and a 3D profilometer; Data preprocessing: Project the 3D point cloud onto multiple views to obtain multi-view projection, and downsample the acquired RGB image and 3D point cloud; Feature extraction: A deep convolutional neural network is used to extract 3D point cloud features; a residual neural network is used to extract RGB image features; a lightweight convolutional neural network is used to extract multi-view projection features, and then feature aggregation is performed on the multi-view projection features; To distinguish between neural networks that extract 3D point cloud features and those that extract multi-view projection features, this application refers to a neural network for extracting 3D point cloud features that includes multi-stage feature transformations and has a deep architecture as a "deep" convolutional neural network; and refers to a neural network for extracting multi-view projection features that only contains a 7-layer network architecture as a "lightweight" convolutional neural network.

[0023] Feature library construction: The 3D point cloud features, RGB image, and multi-view projection features contained in each training sample are stored in their respective feature libraries; after feature extraction from the above three types of data, they are stored in three different feature libraries, namely point cloud feature libraries. RGB image feature library Multi-view image feature library In the initial construction of the feature library, all features of the samples are stored in the feature library. As the feature library is periodically replenished, its size will increase. In order to improve the inference efficiency of the feature library, a fast Bayesian core set sampling method is used to maintain the size of the feature library, with each feature library limited to 3,000,000 features.

[0024] During the training phase, multiple sample data points need to be collected, with a minimum of 1000 samples. Data acquisition involves using a color camera to capture RGB images, requiring a camera resolution of at least 4000×3000 pixels and a frame rate of at least 30 frames per second. A high-precision 3D profilometer is used to acquire wheel hub contour data, stored as a point cloud. The profilometer's measurement accuracy must reach ±0.01mm, and the scanning speed must be at least 5000 points per second. A synchronous triggering mechanism is used to achieve RGB image acquisition and 3D point cloud acquisition, ensuring time synchronization. All samples collected during the training phase are normal samples.

[0025] During the testing phase, the following is performed: Defect identification: Following the above steps of data acquisition, data preprocessing, and feature extraction, three features of the test sample are obtained. Feature scores are calculated based on the feature library, and defect judgment is made based on the defect threshold. Test samples identified as defective are segmented and located, and the features of defective test samples are stored in the defect library. The features of test samples identified as normal are stored in the feature library, and the size of the feature library is controlled by the fast Bayesian core set sampling method.

[0026] Because the amount of data on wheel hub defects is relatively small, all training samples are normal samples. When acquiring RGB data from the training samples, the height and shooting angle of the color camera are pre-set. In this embodiment, the camera is placed perpendicular to the wheel hub surface, with the lens facing the wheel hub and a distance of 30cm from the surface. Point cloud data is acquired using a high-precision 3D profilometer and stored in PCD format. The color camera and the high-precision 3D profilometer are synchronized in sampling time using a trigger mechanism. This embodiment uses 1568 samples, all of which are local data from the wheel hub.

[0027] RGB image data preprocessing includes applying nearest-neighbor interpolation to the RGB image, downsampling the R, G, and B channels respectively, and then sequentially applying height threshold filtering, normal vector consistency filtering, and connected component analysis to remove low-profile workbench background, workbench surface background, and isolated noise. The downsampled image is then subjected to bicubic interpolation. In this embodiment, a 4000×3000 resolution RGB image is downsampled using nearest-neighbor interpolation to obtain a 512×512 image, after which the background is removed. Finally, the 512×512 RGB image is subjected to bicubic interpolation to obtain a 224×224 RGB image.

[0028] like Figure 2 As shown, the extraction of RGB image features using a residual neural network includes: first, using convolution to extract and embed patch features from the image; then, using convolution kernels with different dilation rates in a wide residual neural network to extract features at different scales; and finally, performing channel compression on the features at different scales to fuse multi-scale features. Specifically: The RGB image after data preprocessing is subjected to two convolution processes to increase the number of input channels; Then it enters the first level, which contains two multi-scale branches. After feature extraction from the two branches, it outputs high-resolution features after feature concatenation, channel compression, and residual connection. Then, it enters the second level, uses convolutional layers for downsampling, and then goes through three multi-scale branches to obtain medium-resolution features; Next, we move to the third level, where we use convolutional layers for downsampling, and then go through four multi-scale branches to obtain low-resolution features. The feature maps of the second and third levels are upsampled to the same size as those of the first level and stitched together to form a feature map. The number of channels is then compressed through the fusion module. Finally, after downsampling and reshaping, a patch-level feature map is obtained.

[0029] In this embodiment, the acquired 224×224 RGB image undergoes two convolutional processes, increasing the number of input channels from 3 to 64. Then, it enters the first layer, which contains two multi-scale branches with dilation rates of 1 and 2. After feature extraction from these two branches, feature concatenation, channel compression, and residual connection are performed to output high-resolution features with a size of 224×224×64. Next, it enters the second layer, using a convolutional layer with a stride of 2 and a kernel of 3×3 for downsampling, resulting in 128 channels. Then, it passes through three multi-scale branches with dilation rates of 1, 2, and 4 to obtain medium-resolution features. Finally, it enters the third layer, using a convolutional layer with a stride of 2 and a kernel of 3×3 for downsampling, resulting in 56 channels. Then, it passes through four multi-scale branches with dilation rates of 1, 2, 4, and 6 to obtain low-resolution features. The feature maps of the second and third layers are upsampled to the same size as the first layer and stitched together to form a 224×224×192 feature map, which is then compressed to 64 channels by a fusion module. The fused features are then enhanced with details by downsampling to 56×56×64 using a 4×4 convolution kernel, and then reshaped to 3136×64 to obtain a patch-level feature map.

[0030] Data preprocessing and feature extraction were performed on all training samples in batches to obtain the RGB image color features of all samples, which were then stored in the RGB image feature library. .

[0031] For the 3D point cloud, data preprocessing includes downsampling using the voxel-based centroid method, followed by height threshold filtering, normal consistency filtering, and connected component analysis to remove low-profile workbench background, workbench surface background, and isolated noise. In this embodiment, a 2mm voxel downsampling scheme is used, retaining approximately 50,000 point clouds. Then, the normal vector of each point is calculated, and height threshold filtering, normal consistency filtering, and connected component analysis are applied sequentially to remove low-profile workbench background, workbench surface background, and isolated noise.

[0032] like Figure 3 As shown, the extraction of 3D point cloud features using a deep convolutional neural network includes: The downsampled 3D point cloud is divided into... Individual voxel meshes are used to aggregate the features of each voxel mesh. The output after meshing the entire point cloud is as follows: , The number of non-empty voxels; The voxel mesh is converted into a dense tensor, and the overall geometric features are extracted from it using 3D convolution, outputting a geometric feature tensor. by Centered on a non-empty voxel, with a radius of... The neighborhood voxels within the search radius form One patch; The features of each patch are used as input to the spatial attention mechanism network, and spatial attention is applied to each patch. Max pooling aggregation is performed to obtain patch features of the specified dimensions.

[0033] In this embodiment, the downsampled point cloud is divided into Each voxel mesh is used to aggregate the features of each voxel mesh. Each voxel mesh outputs a feature vector with a dimension of 3. The output of the entire point cloud after meshing is as follows: , The number of non-empty voxels. The voxel mesh is converted into a dense tensor, and its overall geometric features are extracted using 3D convolution, outputting a geometric feature tensor. ,in , and These represent the number of voxel grids along the x, y, and z coordinate axes, respectively. Centered on a non-empty voxel, with a radius of... The search radius contains neighborhood voxels, and each patch contains a maximum of 32 voxels, forming... There are patches, and at this point, we know that the feature dimension of each patch is . The features of each patch are used as input to a spatial attention network, which applies spatial attention to each patch, resulting in an output dimension of [dimensionality missing]. Max pooling is performed to compress the features of each patch, obtaining patch features of the specified dimensions. The output is... .

[0034] Data preprocessing and feature extraction are performed on all training samples in batches to obtain the point cloud features of all samples, which are then stored in a point cloud feature library. .

[0035] The process of projecting a 3D point cloud onto multiple views to obtain a multi-view projection includes: projecting the 3D point cloud onto the same image physical coordinate system as the 2D RGB image based on the external and internal parameter matrices of the color camera to obtain a reference projection view; using the plane where the reference view is located as the reference plane, performing a rotation transformation on the reference view using a rotation function to obtain a rotated projection view, generating an even number of rotated projection views of the point cloud on different planes, with the absolute value of the rotation angle not exceeding pi / 12. make For the intrinsic parameter matrix of the color camera, For the external parameter matrix, For the first The pixel plane coordinates of each point The first point cloud after downsampling The spatial coordinates of a point are given by the following formula: ; in This is a normalization function that converts depth values ​​in a 224×224 grid into grayscale values. The view obtained using the above formula is the reference view. The coordinate system used is a right-handed coordinate system. These are Euler angles for rotation about the x, y, and z axes, respectively. When all are 0, that is, the baseline view The plane it lies in is the same as the plane containing the RGB 2D image. Let... , ,right , Four pairs of values ​​are taken uniformly to generate the rotation function. : ; use For the reference view Perform a rotation operation to obtain four other rotated views.

[0036] like Figure 4 As shown, for multi-view projection, the reference view and the projected view obtained by rotation transformation are normalized and CLAHE contrast enhancement is performed, and bilateral filtering is used to reduce noise in the image while preserving edge information; then the segmentation module divides each view into non-overlapping patches according to a fixed size.

[0037] The five views mentioned above are input into the preprocessing module for normalization, CLAHE contrast enhancement, and bilateral filtering to reduce noise while preserving edge information. The output shape of this module is... The tensor. The segmentation module divides each view into non-overlapping patches of size 8×8. Each view is divided into 784 patches, and the 5 views have a total of 3920 patches. The output shape is... The tensor.

[0038] like Figure 4 As shown, the extraction of multi-view projection features using a lightweight convolutional neural network includes: sequentially passing the segmented patch through a first convolutional layer, a max pooling layer, a second convolutional layer, a max pooling layer, a third convolutional layer, an average pooling layer, and an L2 normalization layer for feature extraction; After feature extraction, the feature map is reconstructed by view alignment. The rotation function is used to align all rotated projected view features to the reference view coordinate system. After alignment, the position code of each position is added to the feature. The weight is calculated based on the feature variance of each view, and then a weighted average is performed to achieve cross-view fusion. The fused feature vector is reconstructed into a feature map, thus completing the feature aggregation of multi-view projection features.

[0039] The feature extraction module is a lightweight CNN module with only 7 layers, and its output... The feature map shows the dimensions. The view alignment module reassembles the feature map as follows: The system aligns all features to the reference view coordinate system using a known rotation matrix. The feature aggregation module adds the positional encoding of each location to the features, calculates weights based on the feature variance of each view, and then performs a weighted average to achieve cross-view fusion. The fused feature vector is then recombined into... Feature diagram of dimensions.

[0040] Data preprocessing and feature extraction were performed on all training samples in batches to obtain multi-view image features for all samples, which were then stored in a multi-view image feature library. .

[0041] The above establishes an RGB image feature library Point cloud feature library Multi-view image feature library The steps can be performed in parallel or sequentially, depending on the computing power and parallelism of the device.

[0042] In the initial construction of the feature library, all features of the samples are stored in the feature library. As the feature library is supplemented, its size will increase. In order to improve the inference efficiency of the feature library, the fast Bayesian core set sampling method is used to maintain the size of the feature library, and the size of each feature library is limited to 3,000,000 features.

[0043] As a feasible method to control the size of the feature library, when the number of features in the feature library reaches the upper limit of the feature library, the fast Bayesian core set sampling method can be used to reduce it to less than 1 / 3 of the upper limit.

[0044] In the defect detection phase, the same hardware is used to acquire RGB image data and 3D point cloud data of the wheel hub to be inspected, generating multi-view data. Feature extraction is performed on each of the three types of data using the methods described above. The extracted features are then retrieved and compared with features in the feature library, and a defect score is assigned. The scoring scheme for all features is consistent, let... For point cloud feature library, For RGB image feature library, For multi-view image feature library, For any feature library, This is a type of sample data for testing samples. From The feature set extracted from the dataset is used to calculate the feature scores according to the following formula: ; In the formula, The feature score for a specific patch of the sample. For any feature in a feature library, for In a single patch feature, for For each Patch feature, iterate through the feature library. M , to match the feature library Features with minimal differences Recorded as ,exist Of all the patch features, with The most different features Recorded as .

[0045] After obtaining the score of the sample patch feature, it is based on a predefined threshold. Determine if a defect exists. If two or more features indicate a defect in the sample, the defect is directly identified. If only one feature indicates a defect, manual intervention (such as visual inspection) is introduced to determine the final result. If a defect is identified, the sample feature is stored in a defect database; if no defect is identified, the feature needs to be stored in three different feature databases.

[0046] threshold Based on the characteristic distribution of normal samples, in this embodiment, the sum of the mean and twice the standard deviation is used as the threshold. Patch features with characteristic values ​​less than or equal to the sum of the mean and twice the standard deviation are considered normal samples, while Patch features with characteristic values ​​greater than the sum of the mean and twice the standard deviation are considered defective samples.

[0047] For patch blocks identified as defects, upsampling is performed to upsample the patch block features to the size of the original image, thereby locating the defects in the original image. For patch blocks requiring manual intervention, the same method is used for location, thus improving the efficiency of manual identification.

[0048] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A wheel hub defect detection method based on visual multi-feature, characterized in that, Includes the following steps: Data acquisition: RGB images and 3D point clouds of the wheel hub are acquired simultaneously using a color camera and a 3D profilometer; Data preprocessing: Project the 3D point cloud onto multiple views to obtain multi-view projection, and downsample the acquired RGB image and 3D point cloud; Feature extraction: A deep convolutional neural network is used to extract 3D point cloud features; a residual neural network is used to extract RGB image features; a lightweight convolutional neural network is used to extract multi-view projection features, and then feature aggregation is performed on the multi-view projection features; Feature library construction: Store the 3D point cloud features, RGB image and multi-view projection features contained in each training sample into the corresponding feature library; Defect identification: Following the above steps of data acquisition, data preprocessing, and feature extraction, three features of the test sample are obtained. Feature scores are calculated based on the feature library, and defect determination is made based on the defect threshold, which is the sum of the mean of the feature scores and twice the standard deviation. Test samples identified as defects are segmented and located, and the features of defective test samples are stored in the defect library. The features of test samples identified as normal are stored in the feature library, and the size of the feature library is controlled by the fast Bayesian core set sampling method. The extraction of 3D point cloud features using a deep convolutional neural network includes: The down-sampled three-dimensional point cloud is divided into voxel grids, the features of each voxel grid are aggregated, each voxel grid outputs one feature vector, the dimension is 3, and the entire point cloud is gridized and output as , is the number of non-empty voxels; The voxel mesh is converted into a dense tensor, and the overall geometric features are extracted from it using 3D convolution, outputting a geometric feature tensor. With a non-empty voxel as the center, search the neighborhood voxels within a radius of , form a patch; The features of each patch are used as input to the spatial attention mechanism network, and spatial attention is applied to each patch. Max pooling aggregation is performed to obtain patch features of the specified dimensions.

2. The wheel hub defect detection method based on visual multi-features according to claim 1, characterized in that: The process of projecting a 3D point cloud onto multiple views to obtain a multi-view projection includes: projecting the 3D point cloud onto the same image physical coordinate system as the 2D RGB image based on the external parameter matrix and internal parameter matrix of the color camera to obtain a reference projection view; using the plane where the reference view is located as the reference plane, performing a rotation transformation on the reference view using a rotation function to obtain a rotated projection view, generating an even number of rotated projection views of the point cloud on different planes, with the absolute value of the rotation angle not exceeding pi / 12. The data preprocessing also includes: for multi-view projection, normalizing the reference view and the projected view obtained by rotation transformation and enhancing the contrast of CLAHE, and using bilateral filtering to reduce noise in the image while preserving edge information; then the segmentation module divides each view into non-overlapping patches according to a fixed size.

3. The visual multi-feature based wheel hub defect detection method of claim 1, wherein: The data preprocessing of the RGB image includes using the nearest neighbor interpolation algorithm to downsample the R, G, and B channels respectively, and then using height threshold filtering, normal vector consistency filtering, and connected component analysis to remove low-profile workbench background, workbench surface background, and isolated noise. Finally, the downsampled image is subjected to a bicubic interpolation algorithm.

4. The visual multi-feature based wheel hub defect detection method of claim 1, wherein: The data preprocessing of the 3D point cloud includes downsampling using the volumetric centroid method, followed by height threshold filtering, normal vector consistency filtering, and connected component analysis to remove low-profile workbench background, workbench surface background, and isolated noise points.

5. The visual multi-feature based wheel hub defect detection method of claim 3, wherein: The extraction of RGB image features using a residual neural network includes: The RGB image after data preprocessing is subjected to two convolution processes to increase the number of input channels; Then it enters the first level, which contains two multi-scale branches. After feature extraction from the two branches, it outputs high-resolution features after feature concatenation, channel compression, and residual connection. Then, it enters the second level, uses convolutional layers for downsampling, and then goes through three multi-scale branches to obtain medium-resolution features; Next, we move to the third level, where we use convolutional layers for downsampling, and then go through four multi-scale branches to obtain low-resolution features. The feature maps of the second and third levels are upsampled to the same size as those of the first level and stitched together to form a feature map. The number of channels is then compressed through the fusion module. Finally, after downsampling and reshaping, a patch-level feature map is obtained.

6. The visual multi-feature based wheel hub defect detection method of claim 2, wherein: The method of extracting multi-view projection features using a lightweight convolutional neural network includes: extracting features from the segmented patch by sequentially passing it through a first convolutional layer, a max pooling layer, a second convolutional layer, a max pooling layer, a third convolutional layer, an average pooling layer, and an L2 normalization layer; After feature extraction, the feature map is reconstructed by view alignment. The rotation function is used to align all rotated projection view features to the reference view coordinate system. After alignment, the position code of each position is added to the feature. The weight is calculated based on the feature variance of each view, and then a weighted average is performed to achieve cross-view fusion. The fused feature vector is reconstructed into a feature map, thus completing the feature aggregation of multi-view projection features.

7. The visual multi-feature based wheel hub defect detection method of claim 1, wherein: The feature score is calculated based on the feature library, including: for a point cloud feature library, for an RGB image feature library, for a multi-view image feature library, for any kind of feature library, for a sample data of a test sample, for a feature set extracted from The feature score is calculated according to the following formula: ； In the formula, The feature score for a specific patch of the sample. For any feature in a feature library, for In a single patch feature, for For each Patch feature, iterate through the feature library. M , to match the feature library Features with minimal differences Recorded as ,exist Of all the patch features, with The most different features Recorded as .

8. The visual multi-feature based wheel hub defect detection method of claim 1, wherein: The defect determination based on the defect threshold includes: determining whether there is a defect based on the three-dimensional point cloud features, RGB image features, and multi-view projection features of the test sample according to the defect threshold; if two or more features indicate that the sample has a defect, it is directly determined to have a defect; if only one feature indicates that there is a defect, manual intervention is introduced to identify and determine the final result; if it is determined to be a defect, the features of the test sample are stored in the defect database; if it is determined not to be a defect, the features of the test sample are stored in the three feature databases.

9. A visual multi-feature based wheel hub defect detection system characterized in that, The wheel hub defect detection method based on visual multi-features as described in any one of claims 1-8 is applicable, comprising: Data acquisition module: used to simultaneously acquire RGB images and 3D point clouds of the wheel hub based on a color camera and a 3D profilometer; Data preprocessing module: used to project 3D point clouds onto multiple views to obtain multi-view projection, and to downsample the acquired RGB images and 3D point clouds; Feature extraction module: used to extract 3D point cloud features using deep convolutional neural networks; extract RGB image features using residual neural networks; extract multi-view projection features using lightweight convolutional neural networks, and then perform feature aggregation on the multi-view projection features; The defect identification module is used to call the data acquisition module, data preprocessing module, and feature extraction module to obtain three features of the test sample, calculate feature scores based on the feature library module, and determine defects based on a defect threshold, which is the sum of the mean of the feature scores and twice the standard deviation. Test samples identified as defective are segmented and located, and the features of defective test samples are stored in the defect library module. Features of test samples identified as normal are stored in the feature library module. Feature library module: Used to store the three features contained in the training samples and the test samples identified as normal, and the feature library size is controlled by the fast Bayesian core set sampling method. Defect Library Module: Used to store the three features contained in test samples identified as defects; The extraction of 3D point cloud features using a deep convolutional neural network includes: The down-sampled three-dimensional point cloud is divided into voxel grids, the features of each voxel grid are aggregated, each voxel grid outputs one feature vector, the dimension is 3, and the entire point cloud is gridized and output as , is the number of non-empty voxels; The voxel mesh is converted into a dense tensor, and the overall geometric features are extracted from it using 3D convolution, outputting a geometric feature tensor. With a non-empty voxel as the center, search the neighborhood voxels within a radius of , form a Patch; The features of each patch are used as input to the spatial attention mechanism network, and spatial attention is applied to each patch. Max pooling aggregation is performed to obtain patch features of the specified dimensions.

Images