Automobile general assembly production line assembly target positioning method based on vision measurement

Abstract

The invention discloses an automobile general assembly production line assembly target positioning method based on vision measurement, and belongs to the technical field of machine vision detection, and the technical key points are that the method comprises the following steps: 1, making a data set, taking a picture of an automobile door handle, enhancing the data, marking the enhanced picture, and dividing the data set; step 2, constructing a YOLOv8CBAM detection model, adding all the c2f modules into a CBAM module to obtain a new C2fCBAM module, and obtaining the YOLOv8CBAM detection model; 3, the data set obtained in the step 1 is substituted into the YOLOv8CBAM detection model obtained in the step 2, training and testing are carried out, and a detection model of the automobile door handle is obtained; and 4, pixel coordinates are obtained after the target is detected, the base coordinate system coordinates of the target are calculated through the camera internal reference matrix K, the camera rotation matrix R, the translation vector t and the DH parameters of the mechanical arm which are obtained through camera calibration, and the problem of obtaining the position information of the assembled vehicle on the automobile final assembly production line is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision detection, and in particular to an assembly target positioning method for an automobile assembly line based on vision measurement. Background Art

[0002] During the final assembly of passenger vehicles, most automakers, both domestically and internationally, especially in China, still install parts while the vehicles are stationary on the production line. With the rapid development of the automotive manufacturing industry, production line automation and intelligentization have become key trends in the industry. The emergence of seventh-axis mobile robots has made mobile assembly possible on automotive final assembly lines. During mobile assembly, assembly robots must obtain the position of the vehicle being assembled on the production line, following it while maintaining relative stationary position. This ensures precise installation of parts, thereby maintaining production efficiency and product quality. Current vehicle-following solutions involve deploying robot tracks parallel to the final assembly line. When a vehicle on the final assembly line enters its operating range, a sensor system is triggered, controlling the robot track to synchronize with the final assembly line while maintaining relative stationary position. The assembly robot then installs parts on the vehicle.

[0003] As an important perception technology, machine vision has been widely used in industrial automation due to its low cost and high information content. It can provide visual perception capabilities for industrial robots. Visual inspection technology can be used to measure the position of vehicles being assembled on the production line. However, the large size of assembled vehicles makes direct position detection difficult. Car door handles are a stable feature of the vehicle's exterior. Machine vision can be used to detect the position of door handles and thus obtain the position of the assembled vehicle.

[0004] Currently, commonly used machine vision inspection algorithms rely on traditional image processing methods, such as frame difference, background difference, and background Gaussian modeling. Frame difference detects moving objects by calculating pixel differences between adjacent frames. If the change in pixel value exceeds a certain threshold, the pixel is considered a moving object. This method is highly sensitive to moving objects and cannot detect stationary objects. However, since the vehicles being assembled on the production line are too large relative to the visual measurement field of view, pixel differences between adjacent frames are difficult to detect. Background difference builds a background model and compares the current frame with it. The difference is the target. This method relies heavily on the accuracy of the background model, which requires regular updates. Dynamic scenes, such as lighting changes or background movement, can render the model ineffective, making it difficult to meet real-world requirements. Furthermore, the large size of the assembled vehicles often obscures the background in the camera's field of view, preventing information from being captured regarding background differences. Therefore, this method is not suitable for automotive production lines. Background Gaussian modeling builds a Gaussian distribution model for each pixel, treating pixels that do not conform to the background distribution as foreground objects. This consumes significant computational resources.

[0005] When using these methods to detect vehicle positions on an automotive production line, if a door handle appears in the camera's field of view, the entire field of view will be the door handle and the door, making methods based on background differences ineffective. To address this issue, there is an urgent need to develop a visual measurement-based method for detecting and localizing assembly objects on automotive assembly lines to overcome the shortcomings of current practical applications. Summary of the Invention

[0006] In view of the shortcomings of the prior art, an embodiment of the present invention aims to provide an assembly target positioning method for an automobile assembly line based on visual measurement, so as to solve the problems in the above-mentioned background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A method for positioning an assembly target in an automobile assembly line based on visual measurement comprises the following steps:

[0009] Step 1: Create a dataset by taking photos of car door handles and enhancing the data. Enhancement methods include rotation, flipping, scaling, and adding noise. Label the enhanced images and divide the dataset into groups.

[0010] Step 2: Build the YOLOv8_CBAM detection model. Add all the c2f modules to the CBAM module to obtain a new C2f_CBAM module and obtain the YOLOv8_CBAM detection model.

[0011] Step 3: The dataset obtained in step S1 is fed into the YOLOv8_CBAM detection model obtained in step S2 for training and testing to obtain a detection model for car door handles.

[0012] Step 4: After detecting the target, the pixel coordinates are obtained. The base coordinate system coordinates of the target are calculated using the camera intrinsic parameter matrix K, camera rotation matrix R, translation vector t, and DH parameters of the robotic arm obtained through camera calibration.

[0013] As a further solution of the present invention, the data set in step one is manually taken of one thousand photos of car door handles under different lighting and angles, and then data enhancement is performed by rotation, scaling, brightness, noise, etc. to obtain three thousand photos, which are then labeled and finally the data set is established.

[0014] As a further solution of the present invention, the data set is divided into a training set, a validation set and a test set in a ratio of 8:1:1.

[0015] As a further solution of the present invention, the CBAM module in step 2 consists of two parts: channel attention and spatial attention. CBAM enhances the expressiveness of features through channel weighting capabilities, controls the model to focus on the key features of the car door handle, and suppresses irrelevant features.

[0016] As a further solution of the present invention, step four includes applying the trained detection model to detect the car door handle, obtaining the pixel coordinates of the door handle, performing depth estimation on the camera to obtain the depth Z, and converting the pixel coordinate system coordinates into camera coordinate system coordinates in combination with the camera's intrinsic parameter matrix K.

[0017] As a further solution of the present invention, step four also includes converting the camera coordinate system coordinates into the robot arm end coordinate system coordinates through the rotation matrix R and the translation vector t.

[0018] As a further solution of the present invention, step four also includes writing a robot arm DH parameter table, and converting the robot arm end coordinate system coordinates into the robot arm base coordinate system coordinates by using the robot arm DH parameters to determine the target position.

[0019] In summary, the embodiments of the present invention have the following beneficial effects compared with the prior art:

[0020] The present invention uses the YOLOv8_CBAM detection model to detect automobile door handles. The Map value of the detected automobile door handles is 0.823, and the positioning accuracy error is within 0.6 cm. This realizes the acquisition of position information of the assembled vehicles on the production line. It has practical significance for the mobile assembly of automobile final assembly lines, and has important practical value and broad application prospects.

[0021] In order to more clearly illustrate the structural features and effects of the present invention, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Flowchart of an embodiment of the invention.

[0023] Figure 2 This is a structural diagram of the YOLOv8_CBAM detection model in an embodiment of the invention.

[0024] Figure 3 This is a graph of the YOLOv8_CBAM model test results in an embodiment of the invention.

[0025] Figure 4 This is a diagram of the marking frame and center point of the car door handle in an embodiment of the invention.

[0026] Figure 5 This is a diagram showing the door handle positioning results in an embodiment of the invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0028] The specific implementation of the present invention is described in detail below with reference to specific embodiments.

[0029] In one embodiment, a method for locating an assembly target on an automobile assembly line based on visual measurement is provided. Figures 1 to 5 , including the following steps:

[0030] Step 1: Create a dataset by taking photos of car door handles and enhancing the data. Enhancement methods include rotation, flipping, scaling, and adding noise. Label the enhanced images and divide the dataset into groups.

[0031] Step 2: Build the YOLOv8_CBAM detection model. Add all the c2f modules to the CBAM module to obtain a new C2f_CBAM module and obtain the YOLOv8_CBAM detection model.

[0032] Step 3: The dataset obtained in step S1 is fed into the YOLOv8_CBAM detection model obtained in step S2 for training and testing to obtain a detection model for car door handles.

[0033] Step 4: After detecting the target, the pixel coordinates are obtained. The base coordinate system coordinates of the target are calculated using the camera intrinsic parameter matrix K, camera rotation matrix R, translation vector t, and DH parameters of the robotic arm obtained through camera calibration.

[0034] For further information, see Figures 1 to 5 In step 1, the dataset is manually taken of one thousand photos of car door handles under different lighting and angles, and then three thousand photos are obtained by data enhancement through rotation, scaling, brightness, noise, etc., and then labeled, and finally the dataset is established.

[0035] For further information, see Figures 1 to 5 The dataset is divided into training set, validation set and test set in a ratio of 8:1:1.

[0036] For further information, see Figures 1 to 5 The CBAM module in step 2 consists of two parts: channel attention and spatial attention. CBAM enhances the feature expression ability through channel weighting capability, controls the model to focus on the key features of the car door handle, and suppresses irrelevant features.

[0037] For further information, see Figures 1 to 5 The step 4 includes applying the trained detection model to detect the car door handle, obtaining the pixel coordinates of the door handle, performing depth estimation on the camera to obtain the depth Z, and converting the pixel coordinate system coordinates into the camera coordinate system coordinates in combination with the camera's intrinsic parameter matrix K.

[0038] For further information, see Figures 1 to 5 , the step 4 also includes converting the camera coordinate system coordinates into the robot arm end coordinate system coordinates through the rotation matrix R and the translation vector t.

[0039] For further information, see Figures 1 to 5 , the step 4 also includes writing out the robot arm DH parameter table, and converting the robot arm end coordinate system coordinates into the robot arm base coordinate system coordinates by using the robot arm DH parameters, thereby determining the target position.

[0040] In this embodiment, a data set is created by taking photos of car door handles, annotating the pictures, and dividing the data set into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0041] 2. Build the YOLOv8_CBAM detection model. Building on the YOLOv8 basic network structure, the CBAM module is integrated into all C2f modules. The CBAM module consists of two components: channel attention and spatial attention. It adaptively adjusts the weights of different channels and spatial locations in the feature map, allowing the model to focus more on key areas of the target.

[0042] 3. Train and test the detection model. Train and test the detection model to obtain the detection model of the car door handle.

[0043] 4. Use the trained detection model to detect the car door handle and obtain the pixel coordinates of the door handle. Use depth estimation calculation and the camera intrinsic parameter matrix K to convert the pixel coordinate system coordinates into the camera coordinate system coordinates. Use the rotation matrix R and translation vector t to convert the camera coordinate system coordinates into the robot end coordinate system coordinates. Use the robot arm DH parameter table to convert the robot end coordinate system coordinates into the robot arm base coordinate system coordinates to achieve target recognition and positioning.

[0044] The dataset used in this paper is self-made. Photos of car door handles are taken under different lighting, angles, and distances. The pixels are unified and then data enhancement is performed by methods such as rotation, scaling, and adding noise. The data are then annotated using Labelimg. Finally, the annotated dataset is divided into training set, test set, and validation set in a ratio of 8:1:1.

[0045] The YOLOv8_CBAM detection model structure is as follows Figure 2 As shown in the figure, all the C2f modules in the three parts of YOLOv8 are replaced by C2f_CBAM modules, as shown in the red dotted box in the figure.

[0046] The C2f_CBAM module integrates the original C2f module into the CBAM module. CBAM is an attention mechanism module for convolutional neural networks. It combines channel attention and spatial attention to effectively improve the performance of the model.

[0047] Set the path of the dataset and input the training set images into the YOLOv8_CBAM car door handle detection model, i.e., the YOLOv8_CBAM.yaml configuration file. Set the corresponding hyperparameters for training and adjust the number of training rounds through the loss curve to ensure model convergence and obtain the optimal model weight file. Load the optimal model weight file and input the test set images into the network to obtain the category, confidence, and bounding box of the car door handle to detect the car door handle. The detection model test results are as follows: Figure 3 shown.

[0048] A visual system is built on a four-axis robotic arm, and the structure of the camera and the robotic arm is like eyes in the hand.

[0049] Calibrate the camera to obtain the intrinsic parameter matrix, rotation matrix R and translation vector.

[0050] Internal parameter matrix where f x =7319.7429,f y=7318.3187, respectively, the focal length in the x and y directions, c x =1059.8872, c y =1319.5558, is the coordinate of the principal point of the image, i.e. the optical center, and s is the axis tilt parameter, which is usually 0.

[0051] Rotation Matrix

[0052] Translation vector t = [-0.1170 0.0937 1.3391]

[0053] Then estimate the camera depth using the following formula:

[0054]

[0055] Where Z is the camera depth, D x =25.9cm, D y =4.9cm, is the actual size of the object, d x =1215,d y =230, is the size of the object in the image, calculated by the pixel distance in the image.

[0056] D x d x and f x Substitute D y d y and f y Substituting in the calculation results, both Z results are approximately 156 cm.

[0057] Place the actual car door handle within the camera's field of view, and take photos at four positions in the robot arm's base coordinate system: (35.5, -32, -40.6), (46.4, -42, -40.6), (39.8, -23.8, -40.6), and (23, -42.6, -40.6).

[0058] by Figure 4 The intersection of the diagonals of the door handle annotation box is used as the target point to obtain its corresponding pixel coordinates (u, v).

[0059] Substitute the pixel coordinates into the following formula to obtain the image coordinates (x, y);

[0060] x=(u-u0)×D x

[0061] y=(v-v0)×D y

[0062] The coordinates of the image centers u0 and v0 in the pixel coordinate system are the same as the optical center c of the camera. x 、cy consistent.

[0063] Substitute the image coordinates (x, y) into the following formula to obtain the camera coordinate system coordinates (X c , Y c , Z c )

[0064]

[0065] Set u0=c x , v0=c y Substitute and simplify to get the camera coordinate system coordinate P C =(X c , Y c , Z c ):

[0066]

[0067] Z c =Z

[0068] The standard DH parameter table of the four-axis robot arm used is established, as shown in Table 1:

[0069] Table 1 Standard DH parameters of the robotic arm

[0070]

[0071] Determine that the manipulator base coordinate system {0} is the fixed reference coordinate system of the manipulator. According to the DH rule, establish the coordinate system {i} for each joint, and use the DH parameters to calculate the transformation matrix A from the coordinate system {i-1} to {i}; let the camera coordinate system be {C}, the manipulator end coordinate system be {E}, and the manipulator base coordinate system be {0}; then the transformation matrix between the camera coordinate system {C} and the manipulator end coordinate system {E} is

[0072] Substitute the above R, t matrix into get

[0073] Calculate the transformation matrix of the end coordinate system {E} relative to the base coordinate system {0} according to the DH parameters of the robot arm Convert the camera coordinate system {C} to the base coordinate system {0} through matrix multiplication:

[0074]

[0075] The transformation matrix A from coordinate system {i-1} to {i} i for:

[0076]

[0077] Substitute i=1, 2, 3, 4 to obtain A1, A2, A3 and A4 respectively.

[0078] Transformation matrix of the robot end coordinate system {E} relative to the base coordinate system {0} for:

[0079]

[0080] Substituting A1, A2, A3 and A4 into

[0081] Then as well as Substitution get

[0082] Then the coordinate P in the camera coordinate system is C , and its final coordinate P0 in the manipulator base coordinate system is:

[0083]

[0084] The YOLOv8_CBAM detection model is used to detect the four pictures taken above. After coordinate transformation calculation, the positioning results are as follows: Figure 5 As shown;

[0085] The first point has an X-axis error of 0.43cm, a Y-axis error of 0.57cm, and a Z-axis error of 0.4cm;

[0086] The second point has an X-axis error of 0.45cm, a Y-axis error of 0.37cm, and a Z-axis error of 0.4cm;

[0087] The third point has an X-axis error of 0.45cm, a Y-axis error of 0.46cm, and a Z-axis error of 0.4cm;

[0088] The fourth point has an X-axis error of 0.59 cm, a Y-axis error of 0.48 cm, and a Z-axis error of 0.4 cm.

[0089] All experimental results show that the method for detecting and positioning assembly targets in an automobile assembly line based on visual measurement proposed in the present invention can more efficiently identify targets and has a smaller positioning error, which is within 0.6 cm.

[0090] The working principle of the present invention is:

[0091] First, we use machine vision to measure the door handles. We then deeply fuse the Convolutional Block Attention Module (CBAM) with the C2f module of YOLOv8 to form the C2f_CBAM module. This module then generates the YOLOv8_CBAM detection model. This model is then used to identify the door handles and obtain their pixel coordinates.

[0092] Then, the coordinates P of the recognition target in the camera coordinate system are calculated through coordinate system transformation C First, the distance Z between the target and the camera is obtained by depth calculation, and then the pixel coordinate system coordinates are converted into the image coordinate system coordinates through the formula, and then the image coordinate system coordinates are converted into the camera coordinate system coordinates P through the camera intrinsic parameter matrix K C , and then through Convert the camera coordinate system to the robot arm coordinate system, Obtained through the DH parameter table, The camera's rotation and translation matrix R and t are obtained, and finally The coordinates of the target in the robot arm base coordinates can be obtained.

[0093] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for positioning assembly targets in an automobile assembly line based on visual measurement, characterized in that: The following steps are involved: Step 1: Create a dataset by taking photos of car door handles and enhancing the data. Enhancement methods include rotation, flipping, scaling, and adding noise. Label the enhanced images and divide the dataset into groups. Step 2: Build the YOLOv8_CBAM detection model. Add all the c2f modules to the CBAM module to obtain a new C2f_CBAM module and obtain the YOLOv8_CBAM detection model. Step 3: The dataset obtained in step S1 is fed into the YOLOv8_CBAM detection model obtained in step S2 for training and testing to obtain a detection model for car door handles. Step 4: After detecting the target, the pixel coordinates are obtained. The base coordinate system coordinates of the target are calculated using the camera intrinsic parameter matrix K, camera rotation matrix R, translation vector t, and DH parameters of the robotic arm obtained through camera calibration.

2. The method for locating assembly targets in an automobile assembly line based on visual measurement according to claim 1, characterized in that: The data set in step 1 is a set of one thousand manually taken photos of car door handles under different lighting and angles, and then three thousand photos are obtained by data enhancement through rotation, scaling, brightness, noise, etc., and then labeled, and finally the data set is established.

3. The method for locating assembly targets in an automobile assembly line based on visual measurement according to claim 2, characterized in that: The dataset is divided into training set, validation set and test set in a ratio of 8:1:

1.

4. The method for locating assembly targets in an automobile assembly line based on visual measurement according to claim 3, characterized in that: The CBAM module in step 2 consists of two parts: channel attention and spatial attention. CBAM enhances the expressiveness of features through channel weighting, controls the model to focus on the key features of the car door handle, and suppresses irrelevant features.

5. The method for locating assembly targets in an automobile assembly line based on visual measurement according to claim 1, characterized in that: The step 4 includes applying the trained detection model to detect the car door handle to obtain the pixel coordinates of the door handle, performing depth estimation on the camera to obtain the depth Z, and converting the pixel coordinate system coordinates into the camera coordinate system coordinates in combination with the camera's intrinsic parameter matrix K.

6. The method for locating assembly targets in an automobile assembly line based on visual measurement according to claim 5, characterized in that: The step 4 also includes converting the camera coordinate system coordinates into the robot arm end coordinate system coordinates through the rotation matrix R and the translation vector t.

7. The method for positioning assembly targets in an automobile assembly line based on visual measurement according to claim 6, characterized in that: The step 4 also includes writing out the robot arm DH parameter table, and converting the robot arm end coordinate system coordinates into the robot arm base coordinate system coordinates by using the robot arm DH parameters to determine the target position.

Images