Multi-modal fusion type classification and feature detection method for automobile hubs
Through multimodal fusion technology, combined with industrial cameras and laser profile measuring instruments, the hub model and feature detection is used to use image processing and network models to solve the problems of low efficiency and poor accuracy of traditional wheel hub detection, and achieve efficient and accurate automated detection .
Patent Information
- Application Number
- CN202410942070.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-08
AI Technical Summary
Classification and feature detection of traditional wheel hub models rely on manual labor, with high cost, low efficiency, misjudgment and misjudgment, and due to subjective factors, it is impossible to achieve high automation and intelligence.
Using multimodal fusion method, combined with industrial cameras and laser profile measuring instruments, hub model classification and feature detection are performed through image processing and network model, and using mean filtering, edge detection, least squares circle fitting and full convolutional neural network for Data processing realizes the fusion of two-dimensional image and three-dimensional contour measurement information.
It improves the efficiency and accuracy of wheel hub model classification, ensures high accuracy of measurement of hub center hole size, reduces labor costs and time costs, and is suitable for the automation inspection needs of industrial production lines.
Smart Images

Figure CN118918362A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of multimodal fusion and deep learning, and in particular to a multimodal fusion automobile wheel hub model classification and feature detection method based on an industrial camera and a line laser profile scanner. Background Art
[0002] With the rapid development of my country's automobile industry in recent years, the automobile industry has become an important pillar of my country's national economy. Wheels are an indispensable part of automobiles. In the industrial production process, traditional production lines mainly rely on manual model classification and feature detection, which is costly, inefficient, and is affected by subjective factors, leading to misjudgment and missed judgment. Long-term repetitive work leads to employee fatigue and lack of concentration, affecting the accuracy of detection.
[0003] In wheel model classification, the front image of the wheel is captured by an industrial camera. How to select a suitable image algorithm to achieve fast and accurate wheel model classification? The classification result is also used in the subsequent feature detection process.
[0004] In wheel hub feature detection, the detection targets mainly include the wheel hub center hole size and valve hole positioning. The traditional method of using two-dimensional images to measure the aperture has errors in the measurement results due to interference from optical system errors, the non-parallelism of the camera optical axis and the workpiece axis, etc. The size of the wheel hub center hole will directly determine whether the wheel hub can be installed on the axle. The accurate positioning of the wheel hub valve hole can effectively guide the drilling equipment to process the valve hole.
[0005] Therefore, an automated and intelligent wheel hub model classification and feature detection technology solution is of great significance for wheel hub processing, warehouse management, and cost reduction and efficiency improvement. This solution cleverly realizes automated and intelligent wheel hub model classification and feature detection based on the multimodal fusion of two-dimensional image information and three-dimensional contour measurement information. Summary of the invention
[0006] In order to solve the shortcomings of the existing technical solutions, the purpose of the present invention is to provide a multi-modal fusion automobile wheel model classification and feature detection method, which is achieved through the following technical solutions:
[0007] The present invention discloses a multi-modal fusion automobile wheel hub model classification and feature detection method, comprising:
[0008] Build a wheel hub model classification and feature detection device, including a group of photoelectric switches, an industrial camera located above the wheel hub corresponding to the center of the wheel hub to be measured, a laser profile measuring instrument located below the wheel hub corresponding to the center hole of the wheel hub to be measured, and a turntable for placing the laser profile measuring instrument;
[0009] An industrial camera is used to obtain a front image of the wheel hub, the image is subjected to mean filtering and median filtering to obtain a filtered image, and the filtered image is subjected to grayscale stretching to obtain a grayscale stretched image of the wheel hub;
[0010] The edge detection algorithm is used to perform edge detection on the grayscale stretched image to obtain the wheel hub edge information image;
[0011] The normalized cross-correlation algorithm is used to match the wheel edge information image with the wheel template database to calculate the wheel model;
[0012] Read the wheel database according to the wheel model to obtain the registered ideal aperture size, calculate the working parameters of the wheel feature center hole size detection device, mainly including the working distance that needs to be compensated by the displacement platform because the laser profiler's measurement distance is too small, the turntable speed and the laser profiler sampling frequency. According to the working parameters, the displacement platform moves and the turntable rotates to drive the laser profiler to scan the wheel center hole to obtain the three-dimensional scanning data of the wheel center hole;
[0013] The least squares circle fitting algorithm is used to process the 3D scanning data of the hub center hole to calculate the characteristic center hole size of the hub;
[0014] Input the front image of the wheel hub into the network model, and use the network model encoder network to extract the wheel hub feature information;
[0015] The wheel feature information is input into the semantic segmentation subnet, and the network model decoder network is used to calculate the wheel feature valve hole segmentation prediction result.
[0016] As a further improvement, the photoelectric switch described in the present invention is used to detect whether the wheel hub enters or leaves the specified position to achieve wheel hub positioning, the industrial camera is used to capture the front image of the wheel hub, and the laser profile measuring instrument is used to rotate and scan the center hole of the wheel hub through the rotational displacement of the turntable to obtain three-dimensional scanning data.
[0017] As a further improvement, the laser profile measuring instrument of the present invention is 50 mm to 70 mm away from the center hole of the wheel hub to be measured.
[0018] As a further improvement, the wheel hub model classification and feature detection device described in the present invention also includes a strip light source arranged on the side of the industrial camera, corresponding to the wheel hub, and located above the wheel hub to be measured, and the strip light source is used to improve the imaging quality.
[0019] As a further improvement, the present invention provides that the laser profile measuring instrument is rotated according to the working parameters to scan the center hole of the wheel hub to obtain the three-dimensional scanning data of the center hole of the wheel hub, specifically:
[0020] First, use the photoelectric switch to detect that the wheel hub enters the specified position to control the transmission equipment to stop, use the industrial camera to capture the front image of the wheel hub, use the image processing algorithm to obtain the wheel hub model, obtain the working parameters according to the wheel hub model, control the displacement platform to move the specified working distance according to the working parameters to ensure that the working range of the laser profiler includes the center hole of the wheel hub, determine the turntable speed ω and the laser profiler sampling frequency f according to the working parameters, and start the laser profiler to scan after the turntable rotates at a constant speed and stabilizes, and obtain the three-dimensional scanning data of the center hole of the wheel hub. The initial position is recorded as the origin, then
[0021] l i =Δ i +a
[0022]
[0023] where Δ i The distance from the center hole position to the edge of the X-axis display of the profile scanner on each profile scanned. This distance is obtained by a laser profiler. a is the distance from the X-axis display edge of the profile scanner to the rotation center. This distance is fixed, where l i is the distance from the center hole position of each contour to the rotation center, θ i is the angle that each contour rotates relative to the initial position, l i and θ i These are the two parameters of the circle equation in polar coordinates, and in rectangular coordinates they are
[0024]
[0025] According to the coordinate data obtained by scanning, the least squares circle fitting method is used to process the data and calculate the size of the hub center hole.
[0026] As a further improvement, the least squares circle fitting described in the present invention is to convert the nonlinear problem into a linear problem through a reasonable approximation, specifically:
[0027] For a standard circle, assuming its center coordinates are (a, b) and its radius is r, then we have
[0028] (x) 2 +(yb) 2 =r 2
[0029] Expand
[0030] x 2 +y 2 +Ax+By+C=0
[0031] Where A=-2a, B=-2b, C=a2 +b 2 -r 2 , so by finding A, B, C, we can find the specific values of the center and radius. If the point set obtained by the laser profile measuring instrument is (x i ,y i ),i∈1,2,3,…,N, and record the distance between each point and the center of the circle as Then the point (x i ,y i ) to the center of the circle and the square of the radius of the fitted circle is:
[0032]
[0033] make Because F(A,B,C)≥0, this function must have a minimum value not less than 0. We can find the extreme point by taking partial derivatives of A, B, and C respectively and making the equation equal to 0. We can compare the obtained extreme points and select the minimum point. The equation group is as follows:
[0034]
[0035] Solving the simultaneous equations, we get the coordinates of the center of the circle after fitting:
[0036] As a further improvement, the network model described in the present invention is a fully convolutional neural network, which adopts a combination of downsampling layers and upsampling layers, uses MobileNetV3 as the backbone network, and is matched with a pyramid pooling module and a segmentation subnet to form the entire network structure.
[0037] As a further improvement, the present invention inputs the front image of the wheel hub into the network model, uses the network model encoder network to extract the wheel hub feature information; inputs the wheel hub feature information into the semantic segmentation subnet, and uses the network model decoder network to calculate the valve hole segmentation prediction result, specifically:
[0038] The front image of the wheel hub taken by the industrial camera is input into the network model, and the MobileNetV3 network is used as the feature encoding layer to extract the wheel hub feature information. The wheel hub feature information is input into the pyramid pooling module to integrate and process the wheel hub feature information at different levels to obtain the multi-scale global information of the wheel hub. The segmentation subnet decoder part uses the multi-scale global information of the wheel hub and jump connections to restore and calculate the prediction results of the wheel hub valve hole segmentation. The loss function calculation is then used to guide the update of the model parameters, optimize the wheel hub valve hole segmentation prediction results, and finally display and save the segmented results.
[0039] Compared with the prior art, the present invention has the following beneficial effects:
[0040] 1. Wheel model classification takes into account both efficiency and accuracy: Selecting appropriate image algorithms achieves fast and accurate wheel model classification through noise reduction, image enhancement and template matching.
[0041] 2. High-precision measurement of the hub center hole size: Due to the special three-dimensional morphology and high-precision requirements of the hub, the use of traditional cameras cannot meet the requirements. The design combines the two-dimensional image information and the three-dimensional contour measurement information in a multimodal manner, and the hub model information calculated by the image algorithm is used for the subsequent detection of the hub feature center hole. This solves the problem of being unable to achieve high automation and intelligence due to the variety of hub models and the limited working distance of the laser contour measuring instrument. Appropriate light sources, industrial cameras, lenses and laser contour measuring instruments are selected, and a device is built to scan and obtain data to ensure measurement accuracy.
[0042] 3. The network model has fast detection speed and accurate results: The backbone network uses MobileNetV3, which is small in size and has a small number of parameters. It is suitable for inheritance in the PLC system of the production line and is convenient for the subsequent registration and training of wheel parameters. The wheel pictures taken on the production line are used for data enhancement as a data set for network model training. The data types are rich and sufficient, which meets the data set requirements for training high-precision networks. The detection results of this network model are accurate.
[0043] 4. Perfect encapsulation of the host computer and strong human-computer interaction: QT is a one-stop solution for application development, which can be bound to the Python language to realize the development of graphical interface applications. After inserting the QT plug-in in VSCode, the UI interface design can be realized, and the wheel registration, camera management and output results can be realized by writing event functions such as buttons and labels. Users only need to click the button to start the detection. The algorithm can output the results by itself and the user decides whether to save the detection result image. It has perfect functions and strong human-computer interaction.
[0044] 5. Taking into account both efficiency and precision, it has obvious advantages in industrial production: the pre-established wheel data template library and the trained network model are encapsulated, and the interactive interface is used to implement the wheel detection provided by the user. In this process, the accuracy of the detection can be guaranteed, while the labor cost and time cost are reduced, which is convenient for assembly line work in industrial production and adapts to the needs of industrial production. It has obvious advantages in industrial production. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 This is a schematic diagram of the laser profile measuring instrument of the present invention scanning and calculating the size of the hub center hole.
[0046] Figure 2 It is a data flow chart of the method of the present invention. DETAILED DESCRIPTION
[0047] The technical solution of the present invention is further described below by means of specific embodiments in conjunction with the accompanying drawings:
[0048] The purpose of the present invention is to propose a multimodal fusion automobile wheel model classification and feature detection method in view of the poor accuracy and slow detection speed of the existing wheel model classification and feature detection.
[0049] Figure 1 The principle of calculating the center hole size by using the laser profiler to scan and obtain the 3D scanning data of the hub center hole. First, O represents the center of the hub center hole to be measured, O' represents the center of the turntable, Δ i The distance from the center hole position to the edge of the X-axis display of the profile scanner on each profile scanned. This distance is obtained by a laser profiler. a is the distance from the X-axis display edge of the profile scanner to the rotation center, which is fixed. The initial position is recorded as the origin, then
[0050] l i =Δ i +a
[0051]
[0052] Among them l i is the distance from the center hole position of each contour to the rotation center, θ i is the angle that each contour rotates relative to its initial position. i and θ i These are the two parameters of the circle equation in polar coordinates. In rectangular coordinates, we have
[0053]
[0054] According to the coordinate data obtained by scanning, the least squares circle fitting method is used to process the data and calculate the size of the hub center hole.
[0055] Figure 2 It is the work flow chart of the present invention.
[0056] The specific implementation method of the present invention is as follows:
[0057] Step 1. Before the detection, build a wheel hub model classification and feature detection device, including a group of photoelectric switches, a transmission device, an industrial camera, a strip light source, a laser profiler, a displacement platform and a turntable. The photoelectric switches are arranged on both sides of the front end of the transmission device to detect whether the wheel hub enters or leaves the specified position to realize the wheel hub positioning. The wheel hub to be measured is placed on the transmission device for movement. In this embodiment, the transmission device is a conveyor belt. The industrial camera is arranged above the center of the wheel hub to be measured to capture the front image of the wheel hub. The strip light source is arranged above the center of the wheel hub to be measured to improve the imaging quality. The laser profiler is arranged below the center hole of the wheel hub to be measured. The laser profiler is installed on the turntable. The turntable is installed on the displacement platform. The laser profiler is 50mm to 70mm away from the center hole of the wheel hub to be measured.
[0058] Step 2: After the photoelectric switch detects that the wheel hub moves to the specified position, the transmission equipment is controlled to stop. The white LED strip light source is used for high-angle front lighting to highlight the contrast between the wheel hub edge and the background. The front image of the wheel hub is captured by an industrial camera. Due to the influence of dust and mechanical vibration on the actual production line, the front image of the wheel hub has obvious noise. According to the image quality, a 3*3, 5*5 or 7*7 window size is selected to perform mean filtering and median filtering on the image to obtain a filtered image and reduce image noise. Since the image lacks mechanical structure in the area where the image has a high grayscale difference, it is not conducive to subsequent image segmentation. Then, the spatial domain method grayscale stretching is used to enhance the filtered image to obtain a grayscale stretching image, so that the image grayscale value is more evenly distributed in the entire grayscale range, and the foreground and background are better distinguished. The Canny edge detection algorithm is used to perform edge detection on the grayscale stretching image to obtain the wheel hub edge information image to ensure the continuity and high contrast of the image edge. The normalized cross-correlation algorithm is used to match the wheel hub edge information image with the wheel hub data template library to calculate the wheel hub model.
[0059] Step three, obtain the working parameters through the wheel template database according to the wheel model. Due to the limitation of the laser profile measuring range of 15mm, it is impossible to fix the laser profile measuring instrument to measure different center hole sizes of different wheel models. It is necessary to control the displacement platform to move the specified working distance for compensation according to the working parameters to ensure that the working range of the laser profile measuring instrument includes the wheel center hole. The turntable speed ω and the laser profile measuring instrument sampling frequency f are determined according to the working parameters. In order to prevent the problem of uneven sampling of the laser profile measuring instrument caused by acceleration when the turntable starts and stops, the laser profile measuring instrument is turned on after the turntable rotates at a uniform speed and stabilizes to perform sampling at equal time intervals to obtain the three-dimensional scanning data of the wheel center hole. The coordinate data is obtained based on the scan, and the least squares circle fitting method is used to process the data to calculate the size of the wheel center hole.
[0060] Step 4. Use the fully convolutional neural network to build the network model, adopt a combination of downsampling layers and upsampling layers, use MobileNetV3 as the backbone network, and match the pyramid pooling module and segmentation subnet to form the entire network structure. The data set uses the front image of the wheel hub captured by an industrial camera on the actual wheel hub production line. The front image of the wheel hub is enhanced by random rotation, scaling, cropping, flipping, translation, affine transformation, and noise addition to build a data set. It is divided into training set, cross set, and validation set according to appropriate proportions, and the training set and cross set are labeled using the LabelMe tool written in Python. The network model is trained on the training set and cross set to obtain the network training results. The network model parameters in onnx format are obtained from the network training results and saved as working parameters to complete the construction of the network model.
[0061] Step 5. Input the front image of the wheel hub taken by the industrial camera into the trained network model, use the MobileNetV3 network as the feature encoding layer to extract the wheel hub feature information, input the wheel hub feature information into the pyramid pooling module to integrate and process the wheel hub feature information at different levels to obtain the multi-scale global information of the wheel hub, and the segmentation subnet decoder part uses the multi-scale global information of the wheel hub and jump connections to restore and calculate the wheel hub valve hole segmentation prediction results, and then use the loss function calculation to guide the update of the model parameters, optimize the wheel hub valve hole segmentation prediction results, and finally display and save the segmented results.
[0062] Step 6. The wheel hub picture actually taken by the industrial camera, the wheel hub valve hole segmentation prediction result, the wheel hub model and the wheel hub center hole size are displayed through the UI interface designed and implemented by QT to facilitate human-computer interaction and intuitive acquisition of results.
[0063] Finally, it should be noted that the above examples are only specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and there are many variations. All variations that can be directly derived or associated with the content disclosed by a person skilled in the art should be considered as the protection scope of the present invention.
Claims
1. A multi-modal fusion automobile wheel model classification and feature detection method, characterized in that: include: Build a wheel hub model classification and feature detection device, including a group of photoelectric switches, an industrial camera located above the wheel hub corresponding to the center of the wheel hub to be measured, a laser profile measuring instrument located below the wheel hub corresponding to the center hole of the wheel hub to be measured, and a turntable for placing the laser profile measuring instrument; An industrial camera is used to obtain a front image of the wheel hub, the image is subjected to mean filtering and median filtering to obtain a filtered image, and the filtered image is subjected to grayscale stretching to obtain a grayscale stretched image of the wheel hub; The edge detection algorithm is used to perform edge detection on the grayscale stretched image to obtain the wheel hub edge information image; The normalized cross-correlation algorithm is used to match the wheel edge information image with the wheel template database to calculate the wheel model; Read the wheel database according to the wheel model to obtain the registered ideal aperture size, calculate the working parameters of the wheel feature center hole size detection device, mainly including the working distance that needs to be compensated by the displacement platform because the laser profiler's measurement distance is too small, the turntable speed and the laser profiler sampling frequency. According to the working parameters, the displacement platform moves and the turntable rotates to drive the laser profiler to scan the wheel center hole to obtain the three-dimensional scanning data of the wheel center hole; The least squares circle fitting algorithm is used to process the 3D scanning data of the hub center hole to calculate the characteristic center hole size of the hub; Input the front image of the wheel hub into the network model, and use the network model encoder network to extract the wheel hub feature information; The wheel feature information is input into the semantic segmentation subnet, and the network model decoder network is used to calculate the wheel feature valve hole segmentation prediction result.
2. The multimodal fusion automobile wheel model classification and feature detection method according to claim 1 is characterized in that: The photoelectric switch is used to detect whether the wheel hub enters or leaves the specified position to achieve wheel hub positioning, the industrial camera is used to capture the front image of the wheel hub, and the laser profile measuring instrument is used to rotate and scan the center hole of the wheel hub to obtain three-dimensional scanning data through the rotation displacement of the turntable.
3. The multimodal fusion automobile wheel model classification and feature detection method according to claim 1 or 2, characterized in that: The wheel hub model classification and feature detection device also includes a strip light source arranged on the side of the industrial camera, corresponding to the wheel hub, and located above the center of the wheel hub to be measured. The strip light source is used to improve the imaging quality.
4. The multimodal fusion automobile wheel model classification and feature detection method according to claim 1 is characterized in that: The laser profile measuring instrument is rotated according to the working parameters to scan the center hole of the wheel hub to obtain the three-dimensional scanning data of the center hole of the wheel hub, specifically: First, use the photoelectric switch to detect that the wheel hub enters the specified position to control the transmission equipment to stop, use the industrial camera to capture the front image of the wheel hub, use the image processing algorithm to obtain the wheel hub model, obtain the working parameters according to the wheel hub model, control the displacement platform to move the specified working distance according to the working parameters to ensure that the working range of the laser profiler includes the center hole of the wheel hub, determine the turntable speed ω and the laser profiler sampling frequency f according to the working parameters, and start the laser profiler to scan after the turntable rotates at a constant speed and stabilizes, and obtain the three-dimensional scanning data of the center hole of the wheel hub. The initial position is recorded as the origin, then l i =D i +a where Δ i The distance from the center hole position to the edge of the X-axis display of the profile scanner on each profile scanned. This distance is obtained by a laser profiler. a is the distance from the X-axis display edge of the profile scanner to the rotation center. This distance is fixed. i is the distance from the center hole position of each contour to the rotation center, θ i is the angle that each contour rotates relative to the initial position, l i and θ i These are the two parameters of the circle equation in polar coordinates, and in rectangular coordinates they are According to the coordinate data obtained by scanning, the least squares circle fitting method is used to process the data and calculate the size of the hub center hole.
5. The multimodal fusion automobile wheel model classification and feature detection method according to claim 1 is characterized in that: The least squares circle fitting is to transform the nonlinear problem into a linear problem through reasonable approximation, specifically: For a standard circle, assuming its center coordinates are (a, b) and its radius is r, then we have (x-a) 2 +(y-b) 2 =r 2 Expand x 2 +y 2 +Ax+By+C=0 Where A=-2a, B=-2b, C=a 2 +b 2 -r 2 , so by finding A, B, C, we can find the specific values of the center and radius. If the point set obtained by the laser profile measuring instrument is (x i ,y i ),i∈1,2,3,…,N, and record the distance between each point and the center of the circle as Then the point (x i ,y i ) to the center of the circle and the square of the radius of the fitted circle is: make Because F(A,B,C)≥0, this function must have a minimum value not less than 0. We can find the extreme point by taking partial derivatives of A, B, and C respectively and making the equation equal to 0. We can compare the obtained extreme points and select the minimum point. The equation group is as follows: Solving the simultaneous equations, we get the coordinates of the center of the circle after fitting:
6. The multimodal fusion automobile wheel model classification and feature detection method according to claim 1 is characterized in that: The network model is a fully convolutional neural network, which adopts a combination of downsampling layers and upsampling layers, uses MobileNetV3 as the backbone network, and is matched with a pyramid pooling module and a segmentation subnet to form the entire network structure.
7. The multimodal fusion automobile wheel model classification and feature detection method according to claim 1 or 2 or 4 or 5 or 6, characterized in that: The front image of the wheel hub is input into the network model, and the wheel hub feature information is extracted using the network model encoder network; the wheel hub feature information is input into the semantic segmentation subnet, and the valve hole segmentation prediction result is calculated using the network model decoder network, specifically: The front image of the wheel hub taken by the industrial camera is input into the network model, and the MobileNetV3 network is used as the feature encoding layer to extract the wheel hub feature information. The wheel hub feature information is input into the pyramid pooling module to integrate and process the wheel hub feature information at different levels to obtain the multi-scale global information of the wheel hub. The segmentation subnet decoder part uses the multi-scale global information of the wheel hub and jump connections to restore and calculate the prediction results of the wheel hub valve hole segmentation. The loss function calculation is then used to guide the update of the model parameters, optimize the wheel hub valve hole segmentation prediction results, and finally display and save the segmented results.
Citation Information
Patent Citations
Blank hub drilling center accurate positioning method based on machine vision
CN106023186A
Automobile hub air hole position detection device and azimuth angle detection device and method
CN112833789A
Metal hub size measuring device and method based on three-dimensional line laser scanner
CN113701648A
Hub real-time classification and air valve hole positioning method based on deep learning
CN114758178A
Automobile hub on-line size detection system and detection method thereof
CN116124026A
Cited By
Hub detection method and system
CN120778884A
Engineering supervision intelligent individual soldier system
CN121352242A
