Detection system for the surface related to the object under test

Abstract

An embodiment of the present invention discloses a detection system for a surface related to a measured object, including a top detection component, a limiting device, a left detection component, and a right detection component, and comprising the following steps: After the positioning system is powered on and started up, first execute the system startup initialization process to detect the hardware status; if all hardware is normally connected, proceed to the next step, otherwise alarm for abnormal connection of relevant hardware and prompt the user to perform maintenance on the relevant hardware; confirm the position of the punching robot, if the position of the punching robot is not at the zero position, perform a zeroing operation on it according to the zeroing trajectory; read relevant detection parameters during the operation of the system; after the system startup initialization is completed, the positioning system executes different processes according to different operations of the user: if the user selects single-station operation, execute the single-station manual operation process; if the user selects full-station operation, the positioning system will execute the full-station automatic operation process.

Description

Technical Field

[0001] The present invention belongs to the technical field of the production of autonomous vehicles, and particularly relates to a detection system for related surfaces of an object to be measured. Background Art

[0002] An autonomous vehicle is an intelligent vehicle that realizes driverless operation through a computer system. An autonomous vehicle relies on the collaborative cooperation of artificial intelligence, vision computing, radar, monitoring devices, and the global positioning system, enabling the computer to automatically and safely operate a motor vehicle without the active operation of a human.

[0003] With the rise of autonomous vehicle technology, more and more enterprises have invested in the research and development of autonomous vehicles. Currently, technologies such as artificial intelligence, vision monitoring and computing, and radar have become mature. By effectively integrating these technologies, an autonomous driving system can be realized. Installing the autonomous driving system into a manually driven vehicle for vehicle modification can convert a manually driven vehicle into an autonomous vehicle. After long-term testing, the autonomous driving system can achieve safe and reliable autonomous driving on the learned lanes.

[0004] The autonomous driving system includes many hardware components such as radar and vision systems. Currently, after a vehicle modification factory manually drills holes at designated installation positions on the vehicle body of a manually driven vehicle, the above-mentioned hardware devices are then installed. During the installation process, if manual drilling is carried out, there are various problems such as low precision, low efficiency, and poor repeatability. Summary of the Invention

[0005] In view of the above existing technical problems, the present invention provides a detection system for related surfaces of an object to be measured, which is used to detect each surface of the vehicle body, so as to realize high-precision positioning and drilling of the drilling positions on the vehicle body for modifying an autonomous vehicle.

[0006] To solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A detection system for related surfaces of an object to be measured includes a top detection component, a limiting device, a left detection component, and a right detection component. It uses a related curved surface positioning and drilling system based on machine vision to realize the positioning function of the related surface. When the object to be detected runs to the positioning system, positioning and drilling at the related surface positioning are realized through the laser sensor positioning method, the machine vision positioning method, and the positional relationship of the related surface, including the following steps:

[0008] After the positioning system is powered on and booted, it first executes the system boot initialization process to detect the hardware status, including the connection status detection of the top detection component, the limit device, the left detection component, and the right detection component; if all hardware is normally connected, the next operation is performed, otherwise an alarm for abnormal connection of relevant hardware is given to prompt the user to perform maintenance on the relevant hardware; confirm the position of the punching robot, if the position of the punching robot is not at the zero position, perform the zeroing operation on it according to the zeroing trajectory; read the relevant detection parameters during the operation of the system.

[0009] After the system boot initialization is completed, the positioning system executes different processes according to different operations of the user:

[0010] If the user selects single-station operation, the single-station manual operation process is executed;

[0011] If the user selects full-station operation, the positioning system will execute the full-station automatic operation process; during the execution of the full-station automatic operation process, if an operation that requires an emergency pause occurs, through the emergency pause, the current positioning and punching operation is stopped, and the punching robot is zeroed.

[0012] Preferably, the top detection component includes a top right detection camera, a first top right light source, a second top right light source, a top right sensor left-right adjustment device, a top right sensor front-back adjustment device, a top right distance measurement sensor, a top left detection camera, a first top left light source, a second top left light source, a top left sensor left-right adjustment device, a top left sensor front-back adjustment device, a top left distance measurement sensor, a top front sensor left-right adjustment device, a top front sensor front-back adjustment device, and a top front distance measurement sensor.

[0013] Preferably, the limit device includes front-back limit of the right front wheel, left-right limit of the right front wheel, front-back limit of the left front wheel, left-right limit of the left front wheel, left-right limit of the right rear wheel, front-back limit of the right rear wheel, front-back limit of the left rear wheel, and left-right limit of the left rear wheel.

[0014] Preferably, the right detection component includes a right front distance measurement sensor, a right front sensor front-back adjustment device, a right front sensor up-down adjustment device, a right front detection camera, a first right front light source, a second right front light source, a right rear distance measurement sensor, a right rear sensor front-back adjustment device, a right rear sensor up-down adjustment device, a right rear detection camera, a first right rear light source, and a second right rear light source.

[0015] Preferably, the left detection component includes a left front distance measurement sensor, a front-back adjustment device for the left front sensor, an up-down adjustment device for the left front sensor, a left front detection camera, a first left front light source, a second left front light source, a left rear distance measurement sensor, an up-down adjustment device for the left rear sensor, a front-back adjustment device for the left rear sensor, a left rear detection camera, a first left rear light source, and a second left rear light source.

[0016] Preferably, the single-station manual operation process includes distortion calibration, N-point calibration of the punching robot, magnification acquisition, template production, and real-time positioning operation. The specific implementation process is as follows:

[0017] If the user selects "distortion calibration", the camera distortion calibration process will be executed. By executing the camera distortion calibration process, distortion calibration files at different angles of the feature plane will be obtained, which can be used for image distortion correction when the feature plane and the camera plane are at different angles.

[0018] If the user selects "N-point calibration of the punching robot", the N-point calibration process of the punching robot will be executed. By executing the N-point calibration process of the punching robot, an N-point calibration file will be obtained to realize the conversion of image coordinates to physical coordinates in the coordinate system of the punching robot.

[0019] If the user selects "magnification acquisition", the magnification acquisition process will be executed. By executing the magnification acquisition process, a first-order polynomial relationship function between the height of the feature plane and the image magnification will be obtained. According to this function, the conversion of image coordinates at different heights of the feature plane will be realized, so as to obtain accurate physical coordinates of the feature points.

[0020] If the user selects "template production", the template production process will be executed. By executing the template production process, the position information of the image processing feature points obtained by the camera and the position information of the punching points on the punching surface will be obtained, and based on this, the position relationship between the feature points and the punching points will be obtained.

[0021] If the user selects "real-time positioning", the single-station real-time positioning process will be executed. Through the single-station real-time positioning process, the current selected station will be real-time positioned to obtain the position information of the punching points, and then the punching robot will be controlled to perform punching operations.

[0022] Preferably, the distortion calibration process includes the following steps:

[0023] Control the checkerboard calibration board to the initial position. Place the calibration board near the detection height of the detection feature points on the object to be measured, that is, the vehicle. After placement, control the laser sensor to collect height information at each position.

[0024] According to the height information of the laser sensor obtained, calculate the offset angle between the calibration board and the camera plane.

[0025] If the angle meets the requirements, trigger the camera to take pictures and collect images;

[0026] Perform distortion calibration on the collected images to generate a distortion calibration file;

[0027] After generating the distortion calibration file, load the distortion calibration file for the images collected by the camera to perform distortion correction.

[0028] Preferably, the magnification acquisition includes the magnification at ±10 cm from the standard distance h of the measured surface of the vehicle body from the laser sensor, and a fitting function of magnification and distance is obtained by fitting.

[0029] Preferably, the magnification at ±10 cm from the standard distance h of the measured surface of the vehicle body from the laser sensor, and the fitting function of magnification and distance includes: the specific measurement process is to measure from h - 10 cm to h + 10 cm, with a measurement interval of 5 mm each time. First, control the calibration plate to the position of h - 10 cm, then obtain the distance from the upper laser sensor to the calibration plate, and use the angle calculation method in the distortion calibration process to obtain the left-right deflection angle θ1′ and the front-back deflection angle θ2′ of the calibration plate;

[0030] Judge whether θ1′ and θ2′ are stable. If not, it means that the calibration plate has not come to a complete stop yet. Then judge whether the number of measurements exceeds the set value. If so, end the current magnification acquisition and wait for the user to operate. Otherwise, continue to obtain the distance from the laser sensor to the calibration plate and calculate the angle;

[0031] If θ1′ and θ2′ are stable, then judge whether θ1′ and θ2′ are within the required range. When the camera and the measured surface are completely perpendicular, after the collected images are corrected for distortion to remove the distortion effect of the lens itself, there is no distortion at each position of the image. Therefore, in order to remove the distortion of the angle introduced on the magnification, it is required that θ1′ and θ2′ are around 0°. If the condition is met, trigger the camera to take pictures to obtain images and save them. If θ1′ and θ2′ are not within the required range, adjust the angle of the calibration plate and then obtain θ1′ and θ2′ again;

[0032] Judge whether the images at all the above height positions have been collected. If so, process the images corresponding to all heights to obtain the pixel pitch of the checkerboard at the same position on the calibration plate. The checkerboard position is selected as the checkerboard in the middle of the image because the distortion at the middle position of the image is the smallest and the pixel pitch error obtained is the smallest. Then, according to the actual physical size of the checkerboard, the proportional relationship between the pixel pitch and the physical size is the image magnification at the calculated height. If the images at all height positions have not been collected yet, adjust the height of the calibration plate to collect images at the next height.

[0033] After obtaining the magnification ratios at various heights, a first-order polynomial fitting is performed according to the corresponding heights to obtain the fitting coefficients k and b, that is, m = k * h4 + b, where m is the magnification ratio and h4 is the distance value measured by the laser sensor, and the fitting coefficients k and b are obtained and stored.

[0034] Preferably, in the full-station automatic operation process, the single-station real-time positioning process is sequentially executed at a total of 6 stations on the left and right sides of the top, front side, and rear side of the vehicle body. Among them, the single-station real-time positioning process is first executed at the left and right stations on the top of the vehicle body. After completing the single-station real-time positioning process on both sides of the vehicle body top, the three angles of the vehicle body relative to the template-made vehicle body in the three-dimensional direction can be obtained. Among them, the left and right deflection angles and the rotation angle within the camera plane can be used as the rotation angles required in the side single-station real-time positioning process.

[0035] The present invention has the following beneficial effects:

[0036] (1) By using machine vision technology combined with high-precision sensors, the positional relationship of positioning points on different curved surfaces is determined, thereby realizing high-precision positioning of the vehicle body's featureless curved surface and the punching operation of the punching robot.

[0037] (2) Through the visual positioning of the feature surface and the rotation angles of the vehicle body feature surface in the three-dimensional direction, combined with the relative positional relationship between the feature points and punching points in the feature surface, the position and punching posture of the punching points are obtained in the real-time positioning detection, and finally the punching robot is controlled to perform the punching operation.

[0038] (3) In real-time detection, the approaching method of the laser sensor to the camera plane position during template making.

[0039] Coarse measurement by the laser sensor and rough visual positioning. In real-time positioning detection, the height of the measured feature surface is obtained through the laser sensor, and the rotation angle and position offset of the feature points relative to the feature points during template making are obtained through the visual system. Based on the rough visual positioning result, the laser sensor is controlled to adjust the measurement position on the camera plane so that it runs to the measurement position during template making, and then the fine measurement of the laser sensor is performed. Then, an accurate height value is obtained through the fine measurement result of the laser sensor, and the fine positioning of the feature information of the feature surface is performed visually.

[0040] (4) A high-precision positioning method for the vehicle body curved surface during planar three-dimensional rotation and translation.

[0041] Obtain the accurate height value of the measured feature surface through a laser sensor. Use the corresponding image distortion correction file for image distortion correction at different heights. Obtain the fitting function of height and magnification through the magnification fitting at different heights. Calculate the magnification according to the height during real-time detection and positioning. Convert the pixel coordinates of the feature points on the image of the measured surface in real-time detection to the coordinates under the standard of the pixel coordinates of the template-making image through the magnification. Obtain the N-point calibration file through the N-point calibration of the punching robot and the camera. After the coordinate transformation of the collected vehicle body feature image through the N-point calibration, the translation amount of the feature points in the physical position can be obtained. Obtain the rotation angle of the three-dimensional plane, and calculate the position of the punching points on the vehicle body surface according to the angle and the translation amount of the feature points.

[0042] (5) Realize multi-station unmanned intervention positioning and punching on multiple different surfaces of the vehicle body.

[0043] First, perform the feature positioning of the vehicle body top surface. Obtain the rotation angle of the vehicle body feature through the vehicle body top surface feature image. Obtain the left and right rotation angles of the feature surface in the three-dimensional space through the laser sensors at the left and right stations on the roof. Obtain the front and rear rotation angles of the vehicle body through the laser sensors at the front and rear stations on the roof. Thus, the three rotation angles of the vehicle body in the three-dimensional space can be obtained, and then the positioning of the corresponding featureless punching points on the top surface can be realized. Then, through the three-dimensional rotation angle, perform the feature surface positioning of the side surface, and then obtain the position and punching posture of the featureless punching points on the side surface. Description of the Drawings

[0044] Figure 1 It is the overall operation flow chart of the detection system for the relevant surfaces of the measured object in the embodiment of the present invention;

[0045] Figure 2 It is the structural schematic diagram of the top detection component and the limit component of the detection system for the relevant surfaces of the measured object in the embodiment of the present invention;

[0046] Figure 3 It is the structural schematic diagram of the right detection component of the detection system for the relevant surfaces of the measured object in the embodiment of the present invention;

[0047] Figure 4 It is the structural schematic diagram of the left detection component of the detection system for the relevant surfaces of the measured object in the embodiment of the present invention;

[0048] Figure 5 It is the single-station manual operation flow chart of the detection system for the relevant surfaces of the measured object in the embodiment of the present invention;

[0049] Figure 6 It is the camera distortion calibration flow chart of the detection system for the relevant surfaces of the measured object in the embodiment of the present invention;

[0050] Figure 7Full-station automatic operation flowchart of the detection system for the related surface of the object to be measured in the embodiment of the present invention;

[0051] Figure 8 Schematic diagram of the deflection angle of the detection system for the related surface of the object to be measured in the embodiment of the present invention;

[0052] Figure 9 Schematic diagram of the workpiece surface coordinate system established by the punching robot in the embodiment of the present invention. Detailed implementation manners

[0053] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0054] A detection system for the related surface of an object to be measured in an embodiment of the present invention includes a top detection component, a limiting device, a left detection component, and a right detection component. The related surface positioning and punching system based on machine vision is used to implement the positioning function of the related surface. When the object to be detected runs to the positioning system, the positioning punching at the related surface positioning is realized through the laser sensor positioning method, the machine vision positioning method, and the position relationship of the related surface. See Figure 1 , including the following steps:

[0055] After the positioning system is powered on and started, first execute the system startup initialization process to detect the hardware status, including the connection status detection of the top detection component, the limiting device, the left detection component, and the right detection component; if all hardware is normally connected, proceed to the next step, otherwise an alarm for abnormal connection of related hardware will be given to prompt the user to perform maintenance on the related hardware; confirm the position of the punching robot. If the position of the punching robot is not at the zero position, perform the zeroing operation according to the zeroing trajectory; read the relevant detection parameters during the operation of the system;

[0056] After the system startup initialization is completed, the positioning system executes different processes according to different operations of the user:

[0057] If the user selects single-station operation, execute the single-station manual operation process;

[0058] If the user selects full-station operation, the positioning system will execute the full-station automatic operation process; during the execution of the full-station automatic operation process, if an operation that requires an emergency pause occurs, through the emergency pause, stop the current positioning punching operation and zero the punching robot.

[0059] Through the detection system for the related surface of the object to be measured set above, first transport the vehicle body to the assembly line flat plate. The vehicle body is successively transported to the flat plate assembly line with a limit device through a crane. When the vehicle body runs to the visual detection position through the assembly line, the flat plate is controlled by the limit device installed below to stop, and the stop signal is acquired by the visual system, and then the user operation selection is carried out. The visual system performs single-station operation or full-station operation according to the user's selection. Before the single-station and full-station real-time positioning operations, the preparation processes for positioning and punching at each station must be carried out first, which are the distortion calibration process, the N-point calibration process, the magnification acquisition process, and the template production process. In the single-station real-time positioning process and the full-station real-time positioning process, through the feature point positions in the template production, adjust the position of the laser sensor to make it approach the position during template production, so as to obtain accurate laser sensor values. The preparation processes for positioning and punching at each station respectively use the detection components corresponding to each station to acquire and process the images and the measured values of the laser sensor. In the single-station real-time positioning process and the full-station real-time positioning process, first control the laser sensor in the top detection component to approach the position to obtain accurate laser sensor values, and then calculate the deflection angle of the vehicle body on the non-camera surface. Then, calculate the deflection angle of the vehicle body on the camera surface by obtaining the features of the image through the camera and the image features during template production. The corresponding deflection angles are used for processing during the distortion correction and image coordinate conversion at each station; through the images acquired by the cameras of the detection components at each station, further calculate the physical positions of the feature points. According to the feature information of the images during template production and real-time positioning and the obtained three-dimensional deflection angle information of the vehicle body, accurately calculate the position information and punching posture of the punching points, and then realize accurate positioning and punching.

[0060] In an embodiment of the present invention, a detection method for the related surface of the object to be measured set above is realized through the detection of the related surface of the object to be measured. The detection of the related surface of the object to be measured includes a top detection component, a limit device, a left detection component, and a right detection component. First, transport the vehicle body to the assembly line flat plate. The vehicle body is successively transported to the flat plate assembly line with a limit device through a crane, ensuring that the position deviation of the vehicle body transported to the flat plate each time is within ±5 cm and the angle deviation is within ±5°.

[0061] Further, in an embodiment of the present invention, refer to Figures 2 to 4, the top detection component includes a top right detection camera 1, a first top right light source 2, a second top right light source 3, a top right sensor left - right adjustment device 4, a top right sensor front - back adjustment device 5, a top right distance measurement sensor 6, a top left detection camera 7, a first top left light source 8, a second top left light source 9, a top left sensor left - right adjustment device 10, a top left sensor front - back adjustment device 11, a top left distance measurement sensor 12, a top front sensor left - right adjustment device 13, a top front sensor front - back adjustment device 14, and a top front distance measurement sensor 15. The top left detection camera 7 is arranged above the left roof position of the rear row seats of the vehicle, with the camera facing vertically downwards. The top left detection camera 7 is approximately above the roof position, and the effective measurement height is about 100 cm; the first top left light source 8 and the second top left light source 9 are relatively perpendicular. Among them, the direction of the first top left light source 8 is parallel to the front - back direction of the vehicle, and the second top left light source 9 is parallel to the left - right direction of the vehicle. The illumination directions of the first top left light source 8 and the second top left light source 9 point to the vicinity of the camera's field - of - view center, and the height of the light source is about 30 cm from the roof. The top left distance measurement sensor 12 is at a height of 20 cm from the roof. The top left sensor left - right adjustment device 10 controls the top left sensor front - back adjustment device 11 to achieve left - right movement, and the top left sensor front - back adjustment device controls the top left distance measurement sensor 12 to achieve front - back movement. It is equivalent to that the top left distance measurement sensor 12 can move left - right and front - back on the roof plane, and the overall sensor movement adjustment mechanism is located near the top of the vehicle tail on the left - hand side vision system. The top right detection camera 1 is approximately above the right roof position of the rear row seats of the vehicle, and the effective measurement height is about 100 cm, with the camera facing vertically downwards. The first top right light source 2 and the second top right light source 3 are relatively perpendicular. Among them, the direction of the first top right light source 3 is parallel to the front - back direction of the vehicle, and the second top right light source 2 is parallel to the left - right direction of the vehicle. Both are installed parallel to the roof, and the illumination direction points to the vicinity of the camera's field - of - view center, and the height of the light source is about 30 cm from the roof. The top left distance measurement sensor 12 is at a height of 20 cm from the roof. The top right sensor left - right adjustment device 4 controls the top right sensor front - back adjustment device 5 to achieve left - right movement, and the top right sensor front - back adjustment device controls the top right distance measurement sensor 6 to achieve front - back movement. It is equivalent to that the top right distance measurement sensor 6 can move left - right and front - back on the roof plane, and the overall sensor movement adjustment mechanism is located near the top of the vehicle tail on the right - hand side vision system. The top front distance measurement sensor 15 is at a height of 20 cm from the roof. The top front sensor left - right adjustment device 13 controls the top front sensor front - back adjustment device 14 to achieve left - right movement, and the top front sensor front - back adjustment device 14 controls the top front distance measurement sensor 15 to achieve front - back movement. It is equivalent to that the top front distance measurement sensor 15 can move left - right and front - back on the roof plane, and the overall sensor movement adjustment mechanism is located near the right roof of the front row of the vehicle.

[0062] The limiting device includes front - rear limit 16 for the right front wheel, left - right limit 17 for the right front wheel, front - rear limit 18 for the left front wheel, left - right limit 19 for the left front wheel, left - right limit 20 for the right rear wheel, front - rear limit 21 for the right rear wheel, front - rear limit 22 for the left rear wheel, and left - right limit 23 for the left rear wheel. Among them, the front - rear limit 16 for the right front wheel and the front - rear limit 18 for the left front wheel are respectively located at the front parts of the left and right tires at the front of the vehicle, restricting the vehicle from moving forward. The front - rear limit 21 for the right rear wheel and the front - rear limit 22 for the left rear wheel are respectively located at the rear parts of the rear tires of the vehicle, restricting the vehicle from moving backward. The left - right limit 17 for the right front wheel and the left - right limit 20 for the right rear wheel are respectively located on the side surfaces of the front and rear tires on the right side of the vehicle, restricting the vehicle from moving to the right. The left - right limit 19 for the left front wheel and the left - right limit 23 for the left rear wheel are respectively located on the side surfaces of the front and rear tires on the left side of the vehicle, restricting the vehicle from moving to the left.

[0063] The right detection component includes a right front ranging sensor 24, a right front sensor front-back adjustment device 25, a right front sensor up-down adjustment device 26, a right front detection camera 27, a first right front light source 28, a second right front light source 29, a right rear ranging sensor 30, a right rear sensor front-back adjustment device 31, a right rear sensor up-down adjustment device 32, a right rear detection camera 33, a first right rear light source 34, and a second right rear light source 35. The right front detection camera 27 is roughly located on the right side of the connection position between the right front door of the vehicle and the vehicle body, with an effective measurement distance of approximately 100 cm, a height from the ground greater than 80 cm, and the camera plane parallel to the front-back direction of the vehicle and facing left. The first right front light source 28 and the second right front light source 29 are relatively perpendicular. Among them, the direction of the first right front light source 28 is parallel to the front-back direction of the vehicle, and the second right front light source 29 is parallel to the up-down direction of the vehicle. Both are installed parallel to the side of the vehicle body, and the light illumination direction points to the vicinity of the center of the camera's field of view. The height of the light source is approximately 30 cm from the right side of the vehicle. The right front ranging sensor 24 is at a distance of 20 cm from the right front sheet metal of the vehicle. The right front sensor up-down adjustment device 26 controls the right front sensor front-back adjustment device 25 to achieve up-down movement, and the right front sensor front-back adjustment device 25 controls the right front ranging sensor 24 to achieve front-back movement. It is equivalent to that the right front ranging sensor 24 can perform front-back and up-down movements on the right side plane of the vehicle. The overall sensor movement adjustment mechanism is located near the front of the vehicle in the right front vision system. The right rear detection camera 33 is roughly located on the right side of the connection position between the right rear fender and the right rear bumper of the vehicle, with an effective measurement distance of approximately 100 cm, a height from the ground greater than 80 cm, and the camera plane parallel to the front-back direction of the vehicle and facing left. The first right rear light source 34 and the second right rear light source 35 are relatively perpendicular. Among them, the direction of the first right rear light source 34 is parallel to the front-back direction of the vehicle, and the second right rear light source 35 is parallel to the up-down direction of the vehicle. Both are installed parallel to the side of the vehicle body, and the light illumination direction points to the vicinity of the center of the camera's field of view. The height of the light source is approximately 30 cm from the right side of the vehicle. The right rear ranging sensor 30 is at a distance of 20 cm from the right front sheet metal of the vehicle. The right rear sensor up-down adjustment device 32 controls the right rear sensor front-back adjustment device 31 to achieve up-down movement, and the right rear sensor front-back adjustment device 31 controls the right rear ranging sensor 30 to achieve front-back movement. It is equivalent to that the right rear ranging sensor 30 can perform front-back and up-down movements on the right side plane of the vehicle. The overall sensor movement adjustment mechanism is located away from the rear of the vehicle in the right rear vision system.

[0064] The left detection component includes a left front ranging sensor 36, a left front sensor vertical adjustment device 37, a left front sensor horizontal adjustment device 38, a left front detection camera 39, a first left front light source 40, a second left front light source 41, a left rear ranging sensor 42, a left rear sensor vertical adjustment device 43, a left rear sensor horizontal adjustment device 45, a left rear detection camera 44, a first left rear light source 46, and a second left rear light source 47. The left front detection camera 39 is approximately located on the left side of the connection position between the left front door of the vehicle and the vehicle body, with an effective measurement distance of approximately 100 cm and a height from the ground greater than 80 cm. The camera surface is parallel to the vehicle's front-rear direction and faces right. The first left front light source 40 and the second left front light source 41 are perpendicular to each other. Among them, the direction of the first left front light source 40 is parallel to the vehicle's front-rear direction, and the second left front light source 41 is parallel to the vehicle's up-down direction. Both are installed parallel to the vehicle body side, and the light illumination direction points to the vicinity of the camera's field of view center. The height of the light source is approximately 30 cm from the left side of the vehicle. The left front ranging sensor 36 is 20 cm away from the right front sheet metal of the vehicle. The left front sensor vertical adjustment device 38 controls the left front sensor horizontal adjustment device 37 to achieve vertical movement, and the left front sensor horizontal adjustment device 37 controls the left front ranging sensor 36 to achieve horizontal movement. It is equivalent to that the left front ranging sensor 36 can perform horizontal and vertical movements on the left side plane of the vehicle. The overall sensor movement adjustment mechanism is located near the front of the vehicle in the left front vision system. The left rear detection camera 44 is approximately located on the left side of the connection position between the left rear fender and the left rear bumper of the vehicle, with an effective measurement distance of approximately 100 cm and a height from the ground greater than 80 cm. The camera surface is parallel to the vehicle's front-rear direction and faces right. The first left rear light source 46 and the second left rear light source 47 are perpendicular to each other. Among them, the direction of the first left rear light source 46 is parallel to the vehicle's front-rear direction, and the second left rear light source 47 is parallel to the vehicle's up-down direction. Both are installed parallel to the vehicle body side, and the light illumination direction points to the vicinity of the camera's field of view center. The height of the light source is approximately 30 cm from the right side of the vehicle. The left rear ranging sensor 42 is 20 cm away from the right front sheet metal of the vehicle. The left rear sensor vertical adjustment device 43 controls the left rear sensor horizontal adjustment device 45 to achieve vertical movement, and the left rear sensor horizontal adjustment device 45 controls the left rear ranging sensor 42 to achieve horizontal movement. It is equivalent to that the left rear ranging sensor 42 can perform horizontal and vertical movements on the left side plane of the vehicle. The overall sensor movement adjustment mechanism is located away from the rear of the vehicle in the left rear vision system.

[0065] In an embodiment of the present invention, before the system performs full-station automatic operation, single-station manual operation needs to be carried out. Through single-station manual operation, the image distortion correction file, the fitting function of the magnification and height, the coordinate conversion file between the image and the punching robot, and the positional relationship between the image feature plane and the punching points on the positioned punching surface of each station are obtained. Through the obtained files and data, automatic positioning punching operations for each station are realized. Refer toFigure 5 , shown is the flowchart of single-station manual operation. The single-station manual operation interface provides operation buttons for distortion calibration, N-point calibration of the punching robot, magnification acquisition, template making, and real-time positioning. According to the relevant operations of the user, the corresponding processes can be executed. The specific execution process is as follows:

[0066] The single-station operation process first executes the station selection process. Before entering the single-station manual operation interface, the user needs to first select the station to be operated. After completion of the selection, enter the manual operation interface.

[0067] If the user clicks the "Distortion Calibration" button, the camera distortion calibration process will be executed. By executing the camera distortion calibration process, the distortion calibration files at different angles of the feature surface will be obtained, and this file can be used for image distortion correction when the feature surface and the camera surface are at different angles.

[0068] If the user clicks the "N-point Calibration of the Punching Robot" button, the N-point calibration process of the punching robot will be executed. By executing the N-point calibration process of the punching robot, the N-point calibration file is obtained, and the conversion of image coordinates to physical coordinates in the coordinate system of the punching robot is realized.

[0069] If the user clicks the "Magnification Acquisition" button, the magnification acquisition process will be executed. By executing the magnification acquisition process, the first-order polynomial relationship function between the height of the feature surface and the image magnification is obtained. According to this function, the conversion of image coordinates at different heights of the feature surface can be realized, so as to obtain the accurate physical coordinates of the feature points.

[0070] If the user clicks the "Template Making" button, the template making process will be executed. By executing the template making process, the position information of the image processing feature points obtained by the camera and the position information of the punching points on the punching surface can be obtained, and the position relationship between the feature points and the punching points can be obtained accordingly.

[0071] If the user clicks the "Real-time Positioning" button, the single-station real-time positioning process will be executed. Through the single-station real-time positioning process, the current selected station will be real-time positioned to obtain the position information of the punching points, and then the punching robot will be controlled to perform punching operations. In the real-time positioning process, an emergency pause operation can be performed. After the user presses the "Emergency Pause" button, the emergency pause processing process will be executed.

[0072] When the system is in the single-station operation standby state, after the user presses the "Exit" button, the single-station operation can be exited.

[0073] In one embodiment of the present invention, cameras need to be installed at each station of the object to be detected for visual positioning. Since the feature surface to be detected and the camera surface will present different angles due to the different placement positions of the object to be detected, the camera distortion calibration process of this system will calibrate the distortions of various different angles of the feature surface and the camera surface. In the template production process, the corresponding distortion calibration file is called according to the measured angles of the feature surface and the camera surface for image distortion correction. Through the camera distortion calibration process, distortion calibration files with different deflection angles between the calibration plate and the camera surface can be generated. Each distortion calibration file consists of the perspective distortion of the image and the radial distortion parameters of the image corresponding to the deflection angle. These distortion files are used for image distortion correction. See Figure 6 , the camera distortion calibration process includes the following steps:

[0074] Determine whether the user clicks the "Distortion Calibration" button. If so, perform distortion calibration; otherwise, wait for the user's operation.

[0075] Control the checkerboard calibration plate to the initial position. Place the calibration plate near the detection height of the detection feature points on the object to be detected, that is, the vehicle. After placement, control the laser sensor to collect height information at each position.

[0076] According to the height information obtained by the laser sensor, calculate the offset angle between the calibration plate and the camera surface. During the distortion calibration process, 3 laser sensors are installed. One laser sensor is installed on each side of the left rear and right rear of the calibration plate, that is, the top left ranging sensor and the top right ranging sensor, and one laser sensor is installed on the right front, that is, the top front ranging sensor. The distance between the top left ranging sensor and the top right ranging sensor is denoted as H1, the distance between the top right ranging sensor and the top front ranging sensor is denoted as H2, the height obtained by the top left ranging sensor is denoted as h1, the height obtained by the top right ranging sensor is denoted as h2, and the height obtained by the top front ranging sensor is denoted as h3. The offset angles that can be obtained from the three heights include the left-right deflection angle and the front-back deflection angle with respect to the camera surface. The left-right deflection angle is denoted as θ1, and the front-back deflection angle is denoted as θ2. The calculation formulas are as follows:

[0077]

[0078] Determine whether the obtained θ1 and θ2 are stable within the set threshold range. If so, it is considered that the checkerboard calibration plate is in a stable and stationary state; otherwise, it is considered that the checkerboard calibration plate is still in a moving state. At this time, determine whether the number of angle measurements exceeds the set value. If so, end the current distortion calibration operation and prompt "The calibration plate is not stable. Please re-place the calibration plate"; otherwise, continue to measure θ1 and θ2.

[0079] Within the range of ±10°, the left - right deviation angle and the front - back deviation angle are calibrated for distortion at intervals of 1°. When calibrating the left - right deviation angle, the front - back deviation angle is maintained at 0°. Between the left - right deviation angles from -10° to 10°, each time the angle of the checkerboard calibration plate is controlled to deviate by 1° for distortion calibration. The same process is applied to the front - back deviation angle calibration. According to the current calibration process, it is judged whether the currently measured angle is within the required range. For example, when performing distortion calibration at the left - right deviation angle of -10°, it is judged whether θ1 is within the threshold range near -10° set by the system, and at the same time, it is necessary to ensure that θ2 is 0°. If the conditions are met, the angle is considered to be within the required range and the next step is processed; otherwise, it is necessary to control the movement of the checkerboard calibration plate to adjust the angle of the checkerboard calibration plate so that the calibration plate angle meets the above - mentioned angle requirements.

[0080] If the angle meets the requirements, the camera is triggered to take pictures and collect images.

[0081] Perform distortion calibration on the collected images to generate a distortion calibration file.

[0082] After generating the distortion calibration file, load the distortion calibration file for the images collected by the camera to perform distortion correction.

[0083] Obtain the horizontal pixel pitch and vertical pixel pitch of the checkerboards at the upper, lower, left, right, and middle positions on the checkerboard calibration plate respectively.

[0084] Judge whether the horizontal pixel pitch and vertical pixel pitch in the five directions are all within the threshold range near the standard value. If so, it indicates that the distortion calibration is successful, and the images at the current angle can be correctly distorted - corrected, and the generated distortion calibration file and the corresponding left - right deviation angle and front - back deviation angle of the calibration plate are stored; otherwise, it indicates that the distortion correction fails, and the distance from the laser sensor to the calibration plate is re - obtained and the calibration plate angle is calculated, and then the distortion calibration at the current angle is performed again.

[0085] If the distortion calibration at the current angle is successful, it is judged whether all the above - mentioned angles have completed the distortion calibration. If so, the distortion calibration is completed and waiting for user operation; otherwise, the calibration plate angle is adjusted to continue the distortion calibration of the next angle.

[0086] In an embodiment of the present invention, the vision system performs feature positioning to obtain the pixel coordinates of the feature points. The punching robot operates according to the coordinate system defined by the punching robot. It is necessary to let the punching robot locate the punching points according to the position of the feature points detected by the vision. Therefore, it is necessary to convert the vision coordinates into the punching robot coordinates. After unifying the two coordinate systems, the punching robot can be controlled through the relative position offset detected by the vision. This system generates an N - point calibration file through the N - point calibration process of the punching robot, and realizes the conversion between the vision coordinates and the punching robot coordinates through the N - point calibration file during real - time positioning. The N - point calibration process of the punching robot includes:

[0087] After entering the N - point calibration interface, wait for the user to perform the N - point calibration operation. If the user clicks the "N - point calibration" button, then perform the N - point calibration; otherwise, wait for the user's operation.

[0088] The punching robot creates the world coordinate system according to the characteristic surface of the vehicle body. The world coordinate system is used during the operation of the punching robot, and this coordinate system can be remade according to the actual working conditions. In this system, the roof camera surface of the vehicle body is used as the XY coordinate system, and the vertical direction is the Z - direction of the coordinate system. Remaking the world coordinate system in this way can make the visual coordinate system consistent with the XY coordinate system direction of the punching robot.

[0089] The punching robot sets the tool TCP (Tool Center Point). Set the tool TCP position of the punching robot to the marking head at the N - point calibration, that is, set the attitude calculation reference point of the punching robot to the marking head. Convert the attitude calculation reference point of the punching robot to the marking head, so that any change in the punching robot's attitude will not affect the coordinates of the marking head, and the coordinates of the marking head are also the actual position of the punching robot during the N - point calibration process.

[0090] Obtain the distance from the upper laser sensor to the marking plate, and calculate the front - rear deviation angle and left - right deviation angle of the marking plate.

[0091] Judge whether the angle is within the angle range of distortion correction. If so, control the punching robot to make N - point round dot marks within the camera's field of view; otherwise, prompt the user to adjust the calibration plate angle and wait for the user's operation.

[0092] After the punching robot finishes making N - point round dot marks, control the punching robot to exit the marking area and leave the field of view.

[0093] Control the camera to collect images, and call the distortion calibration files corresponding to the measured left - right deviation angle and front - rear deviation angle between the marking plate and the camera surface respectively for distortion correction.

[0094] Find the center - pixel coordinates of N marking points through the circle - center finding algorithm.

[0095] Based on the center - pixel coordinates of N marking points and the corresponding physical coordinates of the punching robot, perform N - point calibration to obtain the N - point calibration file. This N - point calibration file is the transformation matrix between the camera vision coordinate system and the punching robot coordinate system in the XY two - dimensional plane. Through this transformation matrix, the pixel coordinate values of the camera can be converted into the physical coordinate values of the punching robot.

[0096] After obtaining the N - point calibration file, perform N - point coordinate conversion for the current image to verify whether the generated calibration file is correct.

[0097] Perform circular search on the marked points in the currently completed image after distortion correction to find the pixel coordinates of the centers of the respective circles for N points. Call the N-point calibration file generated above to convert the pixel coordinates of the centers of the N points into the physical coordinates of the punching robot.

[0098] Compare the obtained physical coordinates of the N points with the physical coordinates of the N points recorded during the punching process of the punching robot to obtain statistical data. Determine whether the difference is within the set threshold range. If so, store the N-point calibration file and the corresponding height value measured by the laser sensor. Otherwise, prompt "Abnormal coordinate conversion after N-point calibration. Please re-perform N-point calibration" and wait for the user to operate.

[0099] In an embodiment of the present invention, due to the physical characteristics of machine vision imaging, when the measured surface is at different heights, the measured object appears larger near and smaller far in the image. When performing the conversion of the physical coordinates of the feature points, there will be a difference in the distance of the feature surface relative to the laser sensor between the template making process and the real-time positioning punching process. Therefore, the coordinate standards of the same image collected in the two processes are different, and there will be a position deviation when directly performing coordinate conversion. This system converts the feature point coordinates of the images collected in the two processes into the image coordinates at the time of N-point calibration, and then converts the image coordinates into physical coordinates. Since the image magnification factor can quantitatively represent the size of the measured object in the image at different heights, this process is converted through the magnification factors at different heights. The image magnification factor acquisition process includes the following steps:

[0100] After entering the magnification factor acquisition interface, wait for the user to perform the magnification factor acquisition operation. If the user clicks the "Magnification Factor Acquisition" button, then perform the magnification factor acquisition; otherwise, wait for the user to operate.

[0101] Combined with the tooling accuracy at the vehicle body positioning punching station and the situation of the vehicle body itself, this system will obtain the magnification factors of ±10 cm of the standard distance h between the measured surface of the vehicle body and the laser sensor, and fit to obtain the fitting function of the magnification factor and the distance. The specific measurement process is from (standard distance h - 10 cm) to (standard distance h + 10 cm), with a measurement interval of 5 mm each time. Therefore, first control the calibration plate to (standard distance h - 10 cm), and then obtain the distance from the upper laser sensor to the calibration plate. Use the angle calculation method in the distortion calibration process to obtain the left-right deviation angle θ1′ and the front-back deviation angle θ2′ of the calibration plate.

[0102] Determine whether θ1′ and θ2′ are stable. If not, it means that the calibration plate has not come to a complete stop yet. Then determine whether the number of measurements exceeds the set value. If so, end the current magnification factor acquisition and wait for the user to operate. Otherwise, continue to obtain the distance from the laser sensor to the calibration plate and calculate the angle.

[0103] If θ1′ and θ2′ remain stable, then it is judged whether θ1′ and θ2′ are within the required range. When the camera and the measured surface are completely perpendicular, after the collected image is corrected for distortion to remove the influence of the lens's own distortion, there is no distortion at each position of the image. Therefore, in order to remove the influence of the distortion introduced by the angle on the magnification, it is required that θ1′ and θ2′ be around 0° and between ±0.1°. If the condition is met, the camera is triggered to take a picture and save the image. If θ1′ and θ2′ are not within the required range, the calibration plate angle is adjusted and then θ1′ and θ2′ are obtained again.

[0104] It is judged whether the images at all the above-mentioned height positions have been collected. If so, for the images corresponding to all heights, the pixel pitch of the checkerboard at the same position on the calibration plate is obtained by processing the images. The checkerboard position is selected as the checkerboard in the middle of the image because the distortion at the middle position of the image is the smallest and the pixel pitch error obtained is the smallest. Then, according to the actual physical size of the checkerboard, the proportional relationship between the pixel pitch and the physical size is the image magnification of the calculated height. If the images at all height positions have not been collected yet, the height of the calibration plate is adjusted to collect the image at the next height.

[0105] After obtaining the magnifications at each height, a first-order polynomial fitting is performed according to the corresponding height to obtain the fitting coefficients k and b, that is, m = k*h4 + b, where m is the magnification and h4 is the distance value measured by the laser sensor. The fitting coefficients k and b are obtained and stored.

[0106] In an embodiment of the present invention, the template making process includes the following steps:

[0107] Tap the template making button to enter the template making process interface;

[0108] Judge whether the start making template button is tapped.

[0109] If not, judge whether the exit template making button is tapped. If so, exit the template making interface. If not, the system does not act and waits for the user to tap the start making template button.

[0110] If so, the system judges whether the punching robot has switched the tool TCP (Tool Center Point), that is, the system switches the punching robot TCP position from the marker head TCP to the punching head TCP. After the switching is completed, the punching robot will perform position movement and attitude adjustment in the workpiece coordinate system of the punching machine with the punching head TCP as the standard.

[0111] If not, the system first controls the punching robot to switch the tool TCP. After the switching is completed, it controls each laser sensor (including 3 on the top and 4 on the side) to obtain the distance from the vehicle body. If so, each laser sensor (including 3 on the top and 4 on the side) obtains the distance from the vehicle body.

[0112] The system determines whether the distances obtained by each laser sensor are stable, that is, whether each distance value is less than the set threshold.

[0113] If not, the system counts the number of measurements and determines whether this number exceeds the set value. If it does not exceed, the system continues to determine whether the distances obtained by each laser sensor are stable, that is, whether each distance value is less than the set threshold. If it exceeds, the system prompts that the vehicle body is not placed stably and returns to determine whether the start template-making button is clicked.

[0114] If so, according to the distances from the vehicle body obtained by each laser sensor (referring to the three sensors on the top and the two sensors on the right or left), the deflection angles of each direction of the vehicle body (the rotation angle α of the vehicle body around the Y-axis direction, the rotation angle β of the vehicle body around the X-axis direction, and the rotation angle γ of the vehicle body around the Z-axis direction) are calculated.

[0115] Determine whether the obtained deflection angles of each direction are within the set range.

[0116] If not, prompt that the position of the vehicle body is abnormal, and the user needs to re-place the vehicle body position, and return to determine whether the start template-making button is clicked for execution.

[0117] If so, execute the vehicle body feature acquisition process.

[0118] The system determines whether there is an abnormality in vehicle body feature acquisition.

[0119] If not, the system controls the punching robot to make its punching head run to the punching position; the system then records the intersection coordinates of the feature points, the straight line angles, and the punching point position coordinates respectively; saves the corresponding laser sensor position data, and returns to determine whether the start template-making button is clicked for execution.

[0120] If so, according to the abnormality type (including not finding the vehicle body feature and not finding the seam at the vehicle body feature), prompt the user to confirm the vehicle body position and end the template-making process.

[0121] Furthermore, in an embodiment of the present invention, the vehicle body feature acquisition process in template making is as follows:

[0122] The system controls the camera to collect images of the vehicle body feature surface.

[0123] According to the vehicle body deflection angle, the system calls the corresponding distortion calibration file to perform image distortion correction.

[0124] According to the vehicle body sheet metal feature, the system performs feature template matching at the sheet metal indirect seam.

[0125] The system determines whether the feature is found successfully, that is, whether the matching is successful. If not, the system prompts that the vehicle body feature is not found, please confirm the vehicle body position, and this process ends; if so, the position of the found feature is corrected, and then the two straight lines at the seam are searched respectively.

[0126] The system determines whether the search for two straight lines is successful. If not, it prompts that the joint at the vehicle body feature is not found, asks to confirm the vehicle body position, and this process ends; if so, it calculates the intersection point of the two found straight lines and performs coordinate transformation on the intersection point to obtain the physical coordinates.

[0127] In one embodiment of the present invention, the real-time positioning and punching process includes the following steps:

[0128] Tap the real-time positioning and punching button to enter the real-time positioning and punching process interface;

[0129] Determine whether the start real-time positioning button is tapped.

[0130] If not, determine whether the exit single-station operation button is tapped. If so, exit the real-time positioning and punching interface; if not, the system does not act and waits for the user to tap the start real-time positioning button.

[0131] If so, perform the vehicle body angle acquisition process.

[0132] The system determines whether the vehicle body angle acquisition is successful; if not, it prompts abnormal positioning and returns to continue determining whether the start real-time positioning button is tapped. If so, execute the vehicle body feature acquisition process.

[0133] The system determines whether the vehicle body feature acquisition is successful; if not, it prompts that the vehicle body feature is not found or the joint at the vehicle body feature is not found, asks to confirm the vehicle body position, and returns to continue determining whether the start real-time positioning button is tapped; if so, based on the feature points, punching points, deflection angles in each direction of the vehicle body in the template making and the coordinate of the feature point in the real-time positioning, calculate the position of the new punching point.

[0134] The system determines whether the vehicle body offset position is within the set range. If not, it prompts abnormal vehicle body position,

[0135] The user needs to re-place the vehicle body position and return to continue determining whether the start real-time positioning button is tapped; if so, after controlling the punching robot to adjust its posture, it runs above the new punching point, and the laser sensor continuously obtains the distance from the new punching point.

[0136] Determine whether the distance reaches the set range. If not, the punching robot continues to approach the new punching point by a certain distance and then determines whether the distance reaches the set range; if so, the system controls the punching robot to punch. After completion, return to continue determining whether the start real-time positioning button is tapped.

[0137] Furthermore, the vehicle body angle acquisition process in the real-time positioning and punching process is as follows:

[0138] Execute the sensor approaching template point process.

[0139] The system determines whether the approximation is completed; if not, the sensor approximation template point process continues to be executed. If so, the system calculates and obtains the non-camera plane deflection angle based on the distance measured by the laser sensor.

[0140] Execute the vehicle body feature acquisition process to obtain the camera plane deflection angle. Thus, the rotation angles of the vehicle body in three directions are obtained.

[0141] The above vehicle body angle acquisition process is different for two cases: real-time positioning and punching on the top of the vehicle body and real-time positioning and punching on the side of the vehicle body:

[0142] Taking the positioning and punching on the top of the vehicle body as an example, the workpiece surface coordinate system established by the punching robot is as Figure 9 shown, where the positive direction of the Y-axis is the vehicle head direction.

[0143] The non-camera plane deflection angle obtained by real-time positioning and punching on the top of the vehicle body includes the rotation angle α of the vehicle body around the Y-axis (the left-right tilt angle of the vehicle body), the rotation angle β of the vehicle body around the X-axis (the front-back tilt angle of the vehicle body), and the top camera plane deflection angle refers to the rotation angle γ of the vehicle body around the Z-axis (the rotation angle in the roof plane parallel to the top camera plane).

[0144] The camera plane deflection angle obtained by real-time positioning and punching on the side of the vehicle body refers to the rotation angle β of the vehicle body around the X-axis obtained by the side camera, and the non-camera plane deflection angle includes the rotation angle α around the Y-axis and the rotation angle γ of the vehicle body around the Z-axis obtained by the top camera.

[0145] In an embodiment of the present invention, due to the angle change at different positions in the curved surface space, there may be a position deviation when the point laser sensor collects the distance, resulting in inaccurate obtained distance. This system adopts the sensor approximation process to accurately obtain the position of the point laser sensor on the feature surface during template making. The sensor approximation template point process in the vehicle body angle acquisition process is as follows:

[0146] Each laser sensor obtains the distance from the vehicle body for rough positioning.

[0147] The system determines whether the distances obtained by each laser sensor are stable, that is, each distance value is less than the set threshold. If not, the system counts the number of measurements and determines whether this number exceeds the set value. If it does not exceed, it returns to continue rough positioning. If it exceeds, the system prompts that the vehicle body is not placed stably and ends this process.

[0148] If so, according to the distances from the vehicle body obtained by each laser sensor, the deflection angles of each direction of the vehicle body are calculated.

[0149] The system determines whether the deflection angles are all within the set range; if not, it prompts that the vehicle body position is abnormal and requires the user to re-place the vehicle body, and ends this process; if so, execute the vehicle body feature acquisition process.

[0150] Find the position difference between the target point and the template point, calculate the deviation value for the sensor to move to the template point position, and control the movement of the sensor.

[0151] After moving, the laser sensor measures the distance to the vehicle body. Based on the distances to the vehicle body obtained by them, calculate the deflection angles in each direction of the vehicle body.

[0152] The above sensor approaching template point process is different for two cases: real-time positioning and punching on the top of the vehicle body and real-time positioning and punching on the side of the vehicle body:

[0153] In real-time positioning and punching on the top of the vehicle body, each laser sensor refers to the top sensors (6, 12, 15), and the calculated deflection angles in each direction of the vehicle body include the rotation angle α of the vehicle body around the Y-axis and the rotation angle β of the vehicle body around the X-axis.

[0154] In real-time positioning and punching on the side of the vehicle body, each laser sensor includes the top sensors (6, 12) and the sensors for measuring the working distance of the camera at the corresponding positioning and punching positions.

[0155] The vehicle body feature acquisition process in real-time positioning and punching is the same as that in template making, except that "find the intersection of the two found straight lines and perform coordinate transformation on the intersection point to obtain the physical coordinates" is changed to "find the intersection of the two found straight lines, and according to the height of the real-time feature point and the height of the template feature point, convert the intersection point of the two straight lines into the pixel coordinates of the template image. And perform coordinate transformation on the intersection point to obtain the physical coordinates".

[0156] The formula for converting the intersection point (i.e., the feature point) of the two straight lines into the pixel coordinates of the template image in the vehicle body feature acquisition process during real-time positioning and punching is as follows:

[0157]

[0158] where m 0 represents the magnification of the feature point in template making, which is calculated by substituting the distance value measured by the corresponding station sensor during template making into the first-order fitting function in the magnification acquisition process; m 1 represents the magnification of the feature point in real-time positioning, which is calculated by substituting the distance value measured by the corresponding station sensor after position correction during real-time positioning into the first-order fitting function in the magnification acquisition process; the point (X, Y) represents the pixel coordinates of the feature point in real-time positioning, and the point (X center , Y center ) represents the pixel coordinates of the center point of the image in real-time positioning, and the point (X', Y') represents the pixel coordinates of the feature point in real-time positioning after magnification compensation.

[0159] The point (X', Y') is converted through coordinate transformation to obtain the physical coordinate point (X 1 , Y 1 ).

[0160] Calculating the position of the new punching point in the real-time positioning punching process The specific calculation process is as follows:

[0161] Case 1: The deflection angles of the vehicle body measured during real-time positioning are all the same as those during template making;

[0162]

[0163] Among them, the point (X 0 , Y 0 ) represents the physical coordinates obtained by coordinate conversion of the pixel coordinates of the feature points in template making, and the point represents the physical coordinates of the punching point in template making.

[0164] Case 2: The deflection angles (α and β) of the vehicle body measured during real-time positioning are all the same as those during template making;

[0165]

[0166] Among them, the angle γ Δ is equal to the vehicle body rotation angle γ 1 during real-time positioning minus the vehicle body rotation angle γ 0 during template making, indicating the offset of the punching point in the X-axis direction caused by the vehicle body rotating by an angle γ Δ relative to that during template making, and indicating the offset of the punching point in the Y-axis direction caused by the vehicle body rotating by an angle γ Δ relative to that during template making.

[0167] Case 3: The deflection angle β of the vehicle body measured during real-time positioning is the same as that during template making;

[0168] (1) If the line segment formed by the punching point and the feature point during template making has no angle with the template plane;

[0169]

[0170] Among them, the angle α Δ is equal to the vehicle body rotation angle α 1 during real-time positioning minus the vehicle body rotation angle α 0 , indicating the X-axis direction offset in Case 2 multiplied by cos(α Δ ), which is the change in the X-axis direction offset in Case 2 caused by the vehicle body tilting left and right, that is, indicating that the offset becomes its projection on the template plane.

[0171] (2) If the line segment formed by the punching point and the feature point during template making has an angle η with the template plane, the punching point is lower than the feature point and the punching point slopes downward relative to the feature point (or the punching point is higher than the feature point and the punching point slopes upward relative to the feature point);

[0172]

[0173] Among them, the angle η X is equal to the projection angle of the angle formed by the line segment formed by the punching point and the feature point during template making and the template plane on the plane formed by the X-axis and the Z-axis, indicating the X-axis offset that still needs to be compensated when the real-time positioning of the vehicle body has different left-right tilt angles compared to template making and there is an angle η.

[0174] (3) If the line segment formed by the punching point and the feature point during template making has an angle η with the template plane, the punching point is higher than the feature point and the punching point slopes downward relative to the feature point (or the punching point is lower than the feature point and the punching point slopes upward relative to the feature point);

[0175] Case 4: The vehicle body deflection angle α measured during real-time positioning is the same as that during template making;

[0176] (1) If the line segment formed by the punching point and the feature point during template making has no angle with the template plane;

[0177]

[0178] Among them, the angle β Δ is equal to the vehicle body rotation angle β during real-time positioning 1 minus the vehicle body rotation angle β during template making 0 .

[0179] represents the Y-axis direction offset in Case 2 multiplied by cos(β Δ ), which is the change in the Y-axis direction offset in Case 2 caused by the forward and backward tilt of the vehicle body, that is, it represents that the offset becomes its projection on the template plane.

[0180] (2) If the line segment formed by the punching point and the feature point during template making has an angle η with the template plane, the punching point is lower than the feature point and the punching point slopes downward relative to the feature point (or the punching point is higher than the feature point and the punching point slopes upward relative to the feature point);

[0181]

[0182] Among them, the angle η Y is equal to the projection angle of the angle formed by the line segment formed by the punching point and the feature point during template making and the template plane on the plane formed by the Y-axis and the Z-axis. It represents the Y-axis offset that needs to be compensated when there is a difference in the front-back tilt angle during real-time positioning of the vehicle body compared to when the template was made and there is an included angle η.

[0183] (3) If the line segment formed by the punching point and the feature point during template making has an included angle η with the template plane, the punching point is higher than the feature point and the punching point is tilted downward relative to the feature point (or the punching point is lower than the feature point and the punching point is tilted upward relative to the feature point);

[0184]

[0185] Case 5: The measured tilt angles α and β of the vehicle body during real-time positioning are different from those during template making;

[0186] (1) If the line segment formed by the punching point and the feature point during template making has no included angle with the template plane;

[0187]

[0188] (2) If the line segment formed by the punching point and the feature point during template making has an included angle η with the template plane, the punching point is lower than the feature point and the punching point is tilted downward relative to the feature point (or the punching point is higher than the feature point and the punching point is tilted upward relative to the feature point);

[0189]

[0190] (3) If the line segment formed by the punching point and the feature point during template making has an included angle η with the template plane, the punching point is higher than the feature point and the punching point is tilted downward relative to the feature point (or the punching point is lower than the feature point and the punching point is tilted upward relative to the feature point);

[0191]

[0192] In an embodiment of the present invention, the full-station automatic operation process completes the positioning and punching operations for all workstations to be punched in accordance with the set order. After completing the positioning and punching operations for all workstations, the punching robot is run to the zero return point according to the zero return trajectory and waits for the arrival of the next object to be detected. As Figure 7 shown, the full-station automatic operation process includes the following steps:

[0193] After entering the full-station operation interface, wait for the user to click the "Full-station operation" button. If the user clicks "Full-station operation", the vehicle body in-place judgment process is executed; otherwise, wait for the user to operate.

[0194] The vehicle body running assembly line consists of multiple large flat plates. The front and rear tires of the vehicle body and the side limit devices of the tires are installed on the large flat plates. After the vehicle bodies are sequentially placed at the limit positions, the large flat plates run forward in sequence. When running to the visual positioning station, the bottom limit device of the large flat plate triggers the limit signal. Therefore, the vehicle body in-place judgment process first determines whether the limit signal is received. If the limit signal is received, the laser sensor at the visual positioning station is controlled to continuously obtain the height value from the vehicle roof. If the height value remains stable within the set time, it is considered that the vehicle body has arrived and is stable, and the single-station real-time positioning process is executed. Otherwise, continue to obtain the height value of the laser sensor and determine whether the height value remains stable within the set time.

[0195] The system sequentially executes the single-station real-time positioning process for a total of 6 stations on the left and right sides of the top, the front side, and the rear side of the vehicle body. Among them, the left and right stations on the vehicle roof first execute the single-station real-time positioning process because the front-back deviation angle and the left-right deviation angle of the top relative to the camera plane are the smallest, and the rotation angle of the top within the camera plane can be accurately measured visually. The three-dimensional angle offset of the vehicle body relative to the template during production obtained in this way is the most accurate. After completing the single-station real-time positioning process for both sides of the vehicle roof, the three angles of the vehicle body relative to the template vehicle body in the three-dimensional direction can be obtained. Among them, the left-right deviation angle and the rotation angle within the camera plane can be used as the rotation angles required in the single-station real-time positioning process for the side. A total of 7 laser sensors are installed in this system. Among them, 3 laser sensors are installed on the vehicle roof to obtain the front-back deviation angle and the left-right deviation angle of the vehicle body camera plane. At the same time, the left and right laser sensors below the vehicle roof camera are also used to obtain the actual height of the vehicle body. One laser sensor is installed at each of the front and rear stations on the left and right sides of the vehicle body to obtain the distance of the vehicle body. Therefore, a total of 7 laser sensors are installed in this system.

[0196] Determine whether all stations have completed punching. If so, control the punching robot to return to the standby point and wait for the arrival of the next vehicle body. Otherwise, continue the real-time positioning of the vehicle body at the next station.

[0197] The surface of the detected vehicle is not a completely flat plane, and there is an arc in the sheet metal of the vehicle surface. If the position of the sensor remains unchanged, when making the template, there will be a deviation between the distance information collected by the laser sensor and the actual measurement by the laser sensor. The vehicle angle offset obtained through the distance is not the correct angle. Therefore, it is set that the camera is fixed and the laser sensor is movable.

[0198] After the execution of the laser sensor approaching process is completed, the front-back deviation angle and the left-right deviation angle of the vehicle roof can be calculated according to the laser sensor. The solution method is illustrated as follows: See Figure 8As shown in the figure, if circle 1 is the initial position, the distances measured by the left rear and right rear are HG and AC respectively, and the distance between the two sensors is AH. Circle 2 is the position for re-parking. If the sensors do not move along, the measured positions are HD and AB. In actual use, the sensors will approach the position according to the visual positioning situation. Therefore, only the distances after the sensors move along need to be calculated. The new positions of the sensors after moving along are I and J, and the measured positions are IM and JK. As shown in the figure, the angle of the vehicle does not change significantly. When making the template, the sensors do not move. At this time, the deviation angle of the vehicle body in a certain direction can be calculated through the distance relationship of the sensors; while in real-time detection, after the sensors move through the approximation method, the deviation angle of the vehicle body in a certain direction can also be calculated through the distance relationship of the sensors. The calculation formula is as follows:

[0199] θ is the included angle between AH and GC, which is the angle of the vehicle body when making the template

[0200] θ′ is the included angle between AH and MK, which is the angle of the vehicle body during real-time positioning

[0201] Through the above formula, the inclination angles of the vehicle body in the specified direction during template making and real-time positioning can be calculated. According to the above definition, the α angle of the vehicle body is the left-right inclination angle of the vehicle body, and the β angle is the front-back inclination angle of the vehicle body. Then, when calculating the left-right inclination angle of the vehicle body, θ in this formula corresponds to the α angle of the vehicle body during template making, and θ′ corresponds to the α angle of the vehicle body during real-time positioning. The deviation angle Δ of the vehicle body during real-time positioning can be obtained through (θ′ - θ), specifically α Δ , β Δ and γ Δ .

[0202] Example: Assume that the station is the camera station on the right side of the vehicle roof. According to the above picture, through the camera, the change in the characteristic position of the vehicle roof in the direction around the Y-axis (left-right direction of the vehicle body) is K. The control center controls the left-right adjustment device 4 of the top right sensor to move a displacement of K value.

[0203] Assume that the station is the camera station on the right side of the vehicle roof. According to the above picture, through the camera, the change in the characteristic position of the vehicle roof in the direction around the X-axis (front-back direction of the vehicle body) is L. The control center controls the left-right adjustment device 4 of the top right sensor to move a displacement of L value. The same applies to other stations.

[0204] 6 fixed cameras respectively collect feature point information, and 7 distance measuring sensors measure distances.

[0205] It is known that the top sensors are at the same height of 12, and the distance between 6 and 15 is H1

[0206] The top sensors are at the same height of 6, and the distance between 15 and 6 is H2

[0207] The right sensor is at the same height of 30, and the distance between 24 is H3

[0208] The left sensor is at the same height of 36, and the distance between 42 is H4

[0209] Set the measured value of the sensor template to: H 12 ,H 6 ,H 15 ,H 30 ,H 24 ,H 36 ,H 42

[0210] Set the measured value of the sensor during template making to: h 12 ,h 6 ,h 15 ,h 30 ,h 24 ,h 36 ,h 42

[0211] The measured value of the sensor after correcting the position during real-time measurement is: h 12-1 ,h 6-1 ,h 15-1 ,h 30-1 ,h 24-1 ,h 36-1 ,h 42-1 When the vehicle body is in place, start making the template. The distance values measured by 7 sensors are fed back to the control center to calculate the magnification factor.

[0212] The camera takes pictures to record the feature point information.

[0213] The vehicle body is in place again and starts real-time measurement;

[0214] The top sensor measures the distance value h 12-1 ,h 6-1 , calculate the magnification factor and send it to the control center.

[0215] The two top cameras calculate the offset angle in the Z-axis direction and the offset in the XY coordinate system plane and send the data to the control center.

[0216] The control center adjusts the sensor position according to the camera parameters.

[0217] Rely on the three top sensors to calculate the offset angles of the vehicle body in the X-axis and Y-axis directions. Thus, obtain the rotation angles of the vehicle body in the XYZ three coordinate axis directions.

[0218] Among them, the angle calculation formula during template making is as follows:

[0219] Rotation angle around the X-axis: α 0 =tan -1 (h 15 -h6 ) / H 2

[0220] Rotation angle in the Y-axis direction: β 0 = tan -1 (h 12 -h 6 ) / H 1

[0221] The angle calculation formula during real-time positioning is as follows:

[0222] Rotation angle in the X-axis direction: α 1 = tan -1 (h 15-1 -h 6-1 ) / H 2

[0223] Rotation angle in the Y-axis direction: β 1 = tan -1 (h 12-1 -h 6-1 ) / H 1

[0224] Through the calculation of (β 1 -β 0 ), β can be obtained. Δ , and through the calculation of (α 1 -α 0 ), α can be obtained. Δ .

[0225] The corrected value of the sensor position is transmitted to the control center and used as the magnification correction value for the camera.

[0226] It should be understood that the exemplary embodiments described herein are illustrative rather than restrictive. Although one or more embodiments of the present invention have been described in conjunction with the accompanying drawings, those of ordinary skill in the art should understand that various forms and details may be changed without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. A detection system for the relevant surface of a measured object, characterized in that, it includes a top detection component, a limiting device, a left-side detection component and a right-side detection component, and uses a relevant curved surface positioning and punching system based on machine vision to implement the positioning function of the relevant surface. When the object to be detected runs to the positioning system, the positioning and punching at the relevant surface positioning are realized through the laser sensor positioning method, the machine vision positioning method and the positional relationship of the relevant surface, including the following steps: After the positioning system is powered on and started, first execute the system startup initialization process to detect the hardware status, including the connection status detection of the top detection component, the limiting device, the left-side detection component and the right-side detection component; if all hardware is normally connected, proceed to the next step, otherwise give an alarm for abnormal connection of relevant hardware and prompt the user to repair the relevant hardware; confirm the position of the punching robot, if the position of the punching robot is not at the zero position, perform the zeroing operation on it according to the zeroing trajectory; read the relevant detection parameters during the operation of the system; After the system startup initialization is completed, the positioning system executes different processes according to different operations of the user: If the user selects single-station operation, then execute the single-station manual operation process; If the user selects full-station operation, the positioning system will execute the full-station automatic operation process; during the execution of the full-station automatic operation process, if an operation that requires an emergency pause occurs, through the emergency pause, stop the current positioning and punching operation and zero the punching robot.

2. The detection system for the relevant surface of a measured object according to claim 1, characterized in that, the top detection component includes a top right-side detection camera, a first top right-side light source, a second top right-side light source, a top right-side sensor left-right adjustment device, a top right-side sensor front-back adjustment device, a top right-side distance measurement sensor, a top left-side detection camera, a first top left-side light source, a second top left-side light source, a top left-side sensor left-right adjustment device, a top left-side sensor front-back adjustment device, a top left-side distance measurement sensor, a top front-side sensor left-right adjustment device, a top front-side sensor front-back adjustment device and a top front-side distance measurement sensor.

3. The detection system for the relevant surface of a measured object according to claim 1, characterized in that, the limiting device includes front-back limit for the right front wheel, left-right limit for the right front wheel, front-back limit for the left front wheel, left-right limit for the left front wheel, left-right limit for the right rear wheel, front-back limit for the right rear wheel, front-back limit for the left rear wheel and left-right limit for the left rear wheel.

4. The detection system for the relevant surface of a measured object according to claim 1, characterized in that, the right-side detection component includes a right front-side distance measurement sensor, a right front-side sensor front-back adjustment device, a right front-side sensor up-down adjustment device, a right front-side detection camera, a first right front-side light source, a second right front-side light source, a right rear-side distance measurement sensor, a right rear-side sensor front-back adjustment device, a right rear-side sensor up-down adjustment device, a right rear-side detection camera, a first right rear-side light source, a second right rear-side light source.

5. The detection system for the relevant surface of a measured object according to claim 1, characterized in that, The left detection component includes a left front distance measurement sensor, a front-back adjustment device for the left front sensor, an up-down adjustment device for the left front sensor, a left front detection camera, a first left front light source, a second left front light source, a left rear distance measurement sensor, an up-down adjustment device for the left rear sensor, a front-back adjustment device for the left rear sensor, a left rear detection camera, a first left rear light source, and a second left rear light source.

6. The detection system for the surface related to the object to be measured according to claim 1, characterized in that the single-station manual operation process includes distortion calibration, N-point calibration of the punching robot, magnification acquisition, template production, and real-time positioning operation. The specific implementation process is as follows: If the user selects "distortion calibration", the camera distortion calibration process will be executed. By executing the camera distortion calibration process, distortion calibration files at different angles of the feature surface will be obtained, and these files can be used for image distortion correction when the feature surface and the camera surface are at different angles; If the user selects "N-point calibration of the punching robot", the N-point calibration process of the punching robot will be executed. By executing the N-point calibration process of the punching robot, an N-point calibration file will be obtained to realize the conversion of image coordinates to physical coordinates in the coordinate system of the punching robot; If the user selects "magnification acquisition", the magnification acquisition process will be executed. By executing the magnification acquisition process, a first-order polynomial relationship function between the height of the feature surface and the image magnification will be obtained. According to this function, the conversion of image coordinates at different heights of the feature surface will be realized, so as to obtain accurate physical coordinates of the feature points; If the user selects "template production", the template production process will be executed. By executing the template production process, the position information of the image processing feature points obtained by the camera and the position information of the punching points on the punching surface will be obtained, and based on this, the position relationship between the feature points and the punching points will be obtained; If the user selects "real-time positioning", the single-station real-time positioning process will be executed. Through the single-station real-time positioning process, the current selected station will be real-time positioned to obtain the position information of the punching points, and then the punching robot will be controlled to perform punching operations.

7. The detection system for the surface related to the object to be measured according to claim 6, characterized in that the distortion calibration process includes the following steps: Control the checkerboard calibration plate to the initial position, place the calibration plate near the detection height of the feature points on the object to be measured, that is, on the vehicle. After placement, control the laser sensor to collect height information at each position; According to the height information of the laser sensor obtained, calculate the offset angle between the calibration plate and the camera surface; If the angle meets the requirements, trigger the camera to take pictures and collect images; Perform distortion calibration on the collected images to generate distortion calibration files; After generating the distortion calibration files, load the distortion calibration files for the images collected by the camera to perform distortion correction.

8. The detection system for the surface related to the object to be measured according to claim 6, characterized in that the magnification acquisition includes the magnification within ±10 cm of the standard distance h between the measured surface of the vehicle body and the laser sensor, and a fitting function of magnification and distance is obtained by fitting.

9. The detection system for the surface related to the object to be measured according to claim 8, characterized in that The magnification of the standard distance h between the measured surface of the vehicle body and the laser sensor within ±10 cm, and the fitting function of the magnification and the distance obtained by fitting includes: The specific measurement process is to measure from h - 10 cm to h + 10 cm, with an interval of 5 mm each time. First, control the calibration plate to the position of h - 10 cm, then obtain the distance from the upper laser sensor to the calibration plate, and use the angle calculation method in the distortion calibration process to obtain the left-right deflection angle θ1 of the calibration plate ′ and the front-back deflection angle θ2 ′ ; Judge θ1 ′ and θ2 ′ Whether they remain stable. If not, it indicates that the calibration plate has not come to a complete rest yet. Then judge whether the number of measurements exceeds the set value. If so, end the acquisition of the magnification this time and wait for the user to operate. Otherwise, continue to acquire the distance from the laser sensor to the calibration plate and calculate the angle; If θ1 ′ and θ2 ′ remain stable, then it is judged whether θ1 ′ and θ2 ′ are within the required range. When the camera and the measured surface are kept completely perpendicular, after the acquired image is corrected for distortion to remove the influence of the lens's own distortion, there is no distortion at each position of the image. Therefore, in order to remove the influence of the distortion introduced by the angle on the magnification, it is required that θ1 ′ and θ2 ′ be around 0°. If the condition is met, the camera is triggered to take a picture to obtain and save the image; if θ1 ′ and θ2 ′ are not within the required range, then after adjusting the angle of the calibration plate, θ1 ′ and θ2 ′ are obtained again; Determine whether the images at all the above-mentioned height positions have been collected. If so, process the images corresponding to all heights to obtain the pixel pitch of the checkerboard at the same position on the calibration board. The checkerboard position is selected as the checkerboard in the middle of the image because the distortion at the middle position of the image is the smallest and the pixel pitch error obtained is the smallest. Then, according to the actual physical size of the checkerboard, the proportional relationship between the pixel pitch and the physical size is the image magnification of the calculated height. If the images at all height positions have not been collected yet, adjust the height of the calibration board to collect the image at the next height; After obtaining the magnifications at each height, perform a first-order polynomial fitting according to the corresponding height to obtain the fitting coefficients k and b, that is, m = k * h4 + b, where m is the magnification and h4 is the distance value measured by the laser sensor. Store the obtained fitting coefficients k and b.

10. The detection system for the relevant surface of the object to be measured according to claim 6, characterized in that, in the full-station automatic operation process, the single-station real-time positioning process is sequentially executed at a total of 6 stations on the left and right sides of the top, the front side and the rear side of the vehicle body. Among them, the single-station real-time positioning process is first executed at the left and right stations on the top of the vehicle body. After completing the single-station real-time positioning process on both sides of the vehicle top, three angles of the vehicle body relative to the template-made vehicle body in the three-dimensional direction can be obtained. Among them, the left and right deviation angles and the rotation angle in the camera plane can be used as the rotation angles required in the single-station real-time positioning process on the side.

Images