Method for detecting and positioning an assembly of vehicles

Abstract

The embodiment of the application discloses a detection positioning method for automobile assembly, and the positioning system is powered on, and first executes a system startup initialization process, including: hardware state detection, including connection state detection of both sides of a punching robot, a vision system of each station and a laser sensor; if all the hardware is normally connected, the next operation is performed; the position of the punching robot is confirmed, and if the position of the punching robot is not at a zero position, zero operation is performed on the punching robot according to a zero trajectory; relevant detection parameters in the system running process are read; after the system startup initialization is completed, the positioning system executes different processes according to different operations of a user: if the user selects single-station operation, a single-station manual operation process is executed; if the user selects full-station operation, the positioning system will execute a full-station automatic operation process; in the execution process of the full-station automatic operation process, if an operation needing emergency pause occurs, the current positioning and punching operation is stopped through emergency pause.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automobile production, and particularly relates to a detection positioning method for automobile assembly. BACKGROUND

[0002] An automatic driving automobile is an intelligent automobile capable of realizing unmanned driving through a computer system. The automatic driving automobile relies on artificial intelligence, visual computing, radar, monitoring devices and a global positioning system to cooperate, so that the computer can automatically and safely operate a motor vehicle without human initiative.

[0003] With the rise of automatic driving automobile technology, more and more enterprises are investing in the research and development of automatic driving automobiles. At present, artificial intelligence, visual monitoring and computing, radar and other technologies are mature, and the technologies are effectively integrated to realize an automatic driving system. The automatic driving system is installed in a manually driven automobile for automobile modification, which can realize modification of a manually driven automobile into an automatic driving automobile. After long-term testing, the automatic driving system can realize safe and reliable automatic driving of a learned lane.

[0004] The automatic driving system includes radar, visual systems and many other hardware. At present, automobile modification factories manually punch holes at specified installation positions of a manually driven automobile, and then install the above hardware devices. Manual punching has various problems such as low precision, low efficiency and poor repeatability. SUMMARY

[0005] In view of the above technical problems, the application provides a detection positioning method for automobile assembly, which realizes high-precision positioning and punching of the punching position of the vehicle body of the modified automatic driving automobile, and realizes automatic and intelligent production of the entire modification line.

[0006] To solve the above technical problems, the application adopts the following technical scheme:

[0007] A detection positioning method for automobile assembly uses a related surface positioning and punching system based on machine vision to realize the positioning function of the related surface. When the detected object runs to the positioning system, the positioning and punching at the related surface positioning position are realized through the position relationship among the laser sensor positioning method, the machine vision positioning method and the related surface, and the method comprises the following steps:

[0008] After the positioning system is powered on, the system boot initialization process is first executed to detect the hardware state, including the connection state detection of the two-side punching robot, the cameras of each station and the laser sensor. If all the hardware is normally connected, the next operation is performed, otherwise the related hardware connection abnormality alarm is performed to prompt the user to repair the related hardware. The punching robot position is confirmed, and if the punching robot position is not at the zero position, the zero operation is performed according to the zero trajectory. The related detection parameters in the system running process are read.

[0009] After the system initialization is completed, the positioning system executes different processes according to different operations of the user:

[0010] If the user selects single-station operation, the single-station manual operation process is executed.

[0011] If the user selects full-station operation, the positioning system will execute the full-station automatic operation process. During the execution of the full-station automatic operation process, if an operation requiring emergency pause occurs, the current positioning and punching operation is stopped through emergency pause, and the punching robot is returned to zero.

[0012] Preferably, the single-station manual operation process is used to complete the pre-processing operation before real-time positioning of the specified station and the positioning and punching of the single station, including: a station selection process, a camera distortion calibration process, a punching robot N-point calibration process, a magnification acquisition process, a template making process, and a real-time positioning process.

[0013] Preferably, the camera distortion calibration process is used to obtain distortion calibration files at different angles of the feature surface, which are used for image distortion correction when the feature surface and the camera surface are at different angles.

[0014] Preferably, the punching robot N-point calibration process is used to obtain N-point calibration files to realize the conversion of image coordinates to physical coordinates in the punching robot coordinate system.

[0015] Preferably, the magnification acquisition process is used to obtain a first-order polynomial relationship function between the feature surface height and the magnification, and according to the relationship function, the image coordinates of the feature surface at different heights are converted to obtain accurate physical coordinates of the feature points.

[0016] Preferably, the template making process is used to obtain the position information of the image processing feature points obtained by the camera and the position information of the punching points on the punching surface, and according to this, the position relationship between the feature points and the punching points is obtained.

[0017] Preferably, the full-station automatic operation process is used to complete the positioning and punching operation of all the stations to be punched in a set order. After the positioning and punching operation of all the stations is completed, the punching robot is operated to the zero return point according to the zero return trajectory, and waits for the arrival of the next detected object.

[0018] Preferably, the six stations of the top, left and right sides of the front and rear of the vehicle body are sequentially executed in the full-station automatic operation flow, and the left and right sides of the roof are executed first in the single-station real-time positioning process, because the front and rear angles and the left and right angles of the top relative to the camera plane are the smallest, and the rotation angle of the top in the camera plane can be accurately measured by vision, and the three-dimensional angle offset of the vehicle body relative to the template for making the vehicle body is the most accurate in this way. After completing the single-station real-time positioning process of the two sides of the roof, the three angles of the vehicle body in the three-dimensional direction relative to the template for making the vehicle body are obtained, and the left and right angles and the rotation angle in the camera plane can be used as the rotation angle required in the single-station real-time positioning process of the side.

[0019] Preferably, the vehicle body is first transported to the flow line plate during the execution process, and the vehicle body is transported to the flow line plate by the crane, and the position deviation of the vehicle body transported to the plate each time is within ±5 cm, and the angle deviation is within ±5°, and when the vehicle body runs to the visual detection position through the flow line, the plate is stopped by the limiting device installed below, and the signal is acquired by the vision system, and then the user selects, and the vision system selects the single-station operation or the full-station operation according to the user's selection. Before the single-station and full-station real-time positioning operations, the preparation process of positioning and punching of each station must be carried out, which is respectively distortion calibration process, N-point calibration process, magnification acquisition process and template making process. In the single-station and full-station real-time positioning processes, the position of the laser sensor is adjusted to approach the position during the template making, so as to obtain the accurate laser sensor value. The preparation process of positioning and punching of each station uses the corresponding detection components of each station to acquire and process the image and the laser sensor measurement value. In the single-station and full-station real-time positioning processes, the laser sensor in the top detection component is controlled to approach the position, so as to obtain the accurate laser sensor value, and then the angle of the vehicle body on the non-camera plane is calculated, and then the angle of the vehicle body on the camera plane is calculated according to the image features acquired by the camera and the image features during the template making. The corresponding angle is used for processing during the distortion correction and image coordinate conversion of each station. The physical position of the feature point is further calculated according to the image features and the three-dimensional angle information of the vehicle body acquired during the template making and the real-time positioning, and the position information and the punching posture of the punching point are accurately calculated, and then the accurate positioning and punching are realized.

[0020] Preferably, the preparation process of positioning and punching of each station is carried out before the full-station operation, which is respectively distortion calibration process, N-point calibration process, magnification acquisition process and template making process.

[0021] The application has the following beneficial effects:

[0022] (1) In the system scheme, the machine vision technology is combined with high-precision sensors to determine the positional relationship of different curved surface positioning points, so as to realize high-precision positioning of the vehicle body without characteristic curved surface and punching operation of the punching robot.

[0023] The image feature positioning is performed through the machine vision technology, the position of the feature point and the rotation angle of the feature surface are obtained, and the front and rear and left and right deflection angles of the vehicle body feature surface are obtained through the multiple high-precision sensors, so as to obtain the rotation angle of the three-dimensional direction of the vehicle body feature surface.

[0024] (2) Positioning and punching of the punching robot without characteristic curved surface: through the visual positioning of the feature surface and the rotation angle of the three-dimensional direction of the vehicle body feature surface, in combination with the relative positional relationship between the feature point and the punching point in the feature surface, the position and punching posture of the punching point are obtained in the real-time positioning detection, and finally the punching robot is controlled to perform the punching operation.

[0025] (3) In the real-time detection, the laser sensor is used to approach the position of the camera surface during the template making.

[0026] The laser sensor is used for coarse measurement and visual coarse positioning. In the real-time positioning detection, the height of the measured feature surface is obtained through the laser sensor, and the rotation angle and position offset of the feature point relative to the feature point during the template making are obtained through the visual system. Through the visual coarse positioning result, the measurement position of the laser sensor on the camera surface is adjusted, so that it runs to the measurement position during the template making, and then the precise measurement of the laser sensor is performed. Through the precise measurement result (accurate height value) of the laser sensor, the precise positioning of the feature information of the feature surface is performed through the visual system.

[0027] (4) High-precision positioning method of the curved surface of the vehicle body during the three-dimensional rotation and translation of the plane.

[0028] The accurate height value of the measured feature surface is obtained through the laser sensor. The image distortion correction file corresponding to the image at different heights is used for image distortion correction. The fitting function of the height and the magnification is obtained through the magnification fitting at different heights, and the magnification is calculated according to the height during the real-time detection positioning. Through the magnification, the pixel coordinates of the feature point of the measured surface image in the real-time detection are converted into the coordinates under the pixel coordinates standard of the template making image. Through the N-point calibration of the punching robot and the camera, the N-point calibration file is obtained. After the coordinates of the collected vehicle body feature image are converted through the N-point calibration, the translation amount of the feature point in the physical position can be obtained. The rotation angle of the three-dimensional plane is obtained, and the position of the punching point of the vehicle body curved surface is calculated according to the angle and the translation amount of the feature point.

[0029] (5) Multi-station positioning and punching without human intervention for multiple different surfaces of the vehicle body.

[0030] First, the top surface feature positioning of the vehicle body is performed, the feature rotation angle of the vehicle body is acquired through the top surface feature image of the vehicle body, the left and right rotation angles of the feature surface in the three-dimensional space are acquired through the laser sensors of the left and right stations of the vehicle roof, the front and rear rotation angles of the vehicle body are acquired through the laser sensors of the front and rear stations of the vehicle roof, so that the rotation angles of the vehicle body in three directions in the three-dimensional space can be obtained, and the positioning of the corresponding featureless punching point of the top surface is realized. Through the three-dimensional rotation angle, the side surface feature positioning is performed, and then the position and punching posture of the featureless punching point of the side surface are acquired. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 The overall flowchart of the detection positioning method for the automobile assembly of the embodiment of the application is shown in the figure.

[0032] Figure 2 The structure schematic diagram of the top detection component and the limiting component of the detection positioning method for the automobile assembly of the embodiment of the application is shown in the figure.

[0033] Figure 3 The structure schematic diagram of the right side detection component of the detection positioning method for the automobile assembly of the embodiment of the application is shown in the figure.

[0034] Figure 4 The structure schematic diagram of the left side detection component of the detection positioning method for the automobile assembly of the embodiment of the application is shown in the figure.

[0035] Figure 5 The single station manual operation flowchart of the detection positioning method for the automobile assembly of the embodiment of the application is shown in the figure.

[0036] Figure 6 The full station automatic operation flowchart of the detection positioning method for the automobile assembly of the embodiment of the application is shown in the figure.

[0037] Figure 7 The angle deviation schematic diagram of the detection positioning method for the automobile assembly of the embodiment of the application is shown in the figure.

[0038] Figure 8 The workpiece surface coordinate system schematic diagram established by the punching robot of the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are some of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0040] The embodiment of the application discloses a detection and positioning method for automobile assembly. Figure 1 , comprising the following steps:

[0041] After the positioning system is powered on, a system startup initialization process is first executed, including: detecting the hardware state, including connection state detection of the two-side punching robot, the camera of each station, and the laser sensor; if all the hardware is normally connected, the next operation is performed, otherwise, relevant hardware connection exception alarm is performed, prompting the user to perform maintenance on the relevant hardware; the position of the punching robot is confirmed, if the position of the punching robot is not at the zero position, zero operation is performed on it according to the zero trajectory; relevant detection parameters in the system running process are read;

[0042] After the system startup initialization is completed, the positioning system executes different processes according to different operations of the user.

[0043] If the user selects single-station operation, a single-station manual operation process is executed; the single-station manual operation process can complete the pre-processing operation before real-time positioning of the specified station and the positioning and punching of the single station, mainly including: a station selection process, a camera distortion calibration process, an N-point calibration process of the punching robot, a magnification acquisition process, a template making process and a real-time positioning process.

[0044] If the user selects full-station operation, the positioning system will execute a full-station automatic operation process; the full-station automatic operation process completes the positioning and punching operation of all the stations to be punched in the set order, after completing the positioning and punching operation of all the stations, the punching robot is operated to the zero point according to the zero trajectory, and the arrival of the next detected object is waited for. During the execution of the full-station automatic operation process, if an operation that needs to be executed urgently is generated, the current punching operation is stopped through the emergency pause, and the punching robot is zeroed.

[0045] The automobile assembly detection and positioning method provided in the embodiment of the present application is realized by the high-precision positioning and punching system of the automobile mounting hole, wherein the high-precision positioning and punching system of the automobile mounting hole comprises a top detection component, a limiting device, a left side detection component and a right side detection component. First, the vehicle body is transported to the flow line plate, and the vehicle body is transported to the flow line plate with the limiting device by the crane in sequence, so that the position deviation of the vehicle body transported to the flow line plate is within ±5 cm and the angle deviation is within ±5°. When the vehicle body runs to the visual detection position through the flow line, the flow line is stopped by the limiting device installed below, and the stop signal is acquired by the visual system, and then the user operation selection is performed. The visual system performs single-station operation or full-station operation according to the user selection. Before the single-station and full-station real-time positioning operation, the preparation process of positioning and punching of each station must be performed, which is respectively the distortion calibration process, the N-point calibration process, the magnification acquisition process and the template making process. In the single-station real-time positioning process and the full-station real-time positioning process, the position of the laser sensor is adjusted to approach the position in the template making, so as to obtain the accurate laser sensor value. The preparation process of positioning and punching of each station uses the corresponding detection component of each station to acquire and process the image and the laser sensor measurement value. In the single-station real-time positioning process and the full-station real-time positioning process, the laser sensor in the top detection component is controlled to approach the position, so as to obtain the accurate laser sensor value, and then the deviation angle of the vehicle body on the non-camera surface is calculated. The deviation angle of the vehicle body on the camera surface is calculated by the camera to acquire the image features and the image features in the template making. The corresponding deviation angle is used for processing in the distortion correction and image coordinate conversion of each station. The physical position of the feature point is further calculated by the image acquired by the camera of each station detection component. According to the feature information of the image in the template making and real-time positioning and the acquired three-dimensional deviation angle information of the vehicle body, the position information and the punching posture of the punching point are accurately calculated, and then the precise positioning and punching are realized.

[0046] Further, in the embodiment of the present application, referring to Figures 2 to 4, the top detection component includes a top right detection camera 1, a first top right light source 2, a second top right light source 3, a top right sensor left-right adjusting device 4, a top right sensor front-rear adjusting device 5, a top right distance sensor 6, a top left detection camera 7, a first top left light source 8, a second top left light source 9, a top left sensor left-right adjusting device 10, a top left sensor front-rear adjusting device 11, a top left distance sensor 12, a top front sensor left-right adjusting device 13, a top front sensor front-rear adjusting device 14, and a top front distance sensor 15. The top left detection camera 7 is arranged above the left side roof position of the rear seats of the vehicle, the camera vertically downward, the top left detection camera 7 is located above the roof position at an effective measurement height of about 100 cm; the first top left light source 8 and the second top left light source 9 are perpendicular to each other, wherein the direction of the first top left light source 8 is parallel to the front-rear direction of the vehicle, the second top left light source 9 is parallel to the left-right direction of the vehicle, the light directions of the first top left light source 8 and the second top left light source 9 point to the center of the camera field of view, and the light source height is about 30 cm away from the roof; the top left distance sensor 12 is at a height of 20 cm from the roof, the top left sensor left-right adjusting device 10 controls the top left sensor front-rear adjusting device 11 to realize left-right movement, and the top left sensor front-rear adjusting device controls the top left distance sensor 12 to realize front-rear movement, which is equivalent to that the top left distance sensor 12 can move front-rear-left-right on the roof plane, and the sensor movement adjusting mechanism is located near the top of the vehicle tail on the left side of the vision system. The top right detection camera 1 is located above the right side roof position of the rear seats of the vehicle at an effective measurement height of about 100 cm, and the camera vertically downward. The first top right light source 2 and the second top right light source 3 are perpendicular to each other, wherein the direction of the first top right light source 3 is parallel to the front-rear direction of the vehicle, the second top right light source 2 is parallel to the left-right direction of the vehicle, and both are installed parallel to the roof, the light directions point to the center of the camera field of view, and the light source height is about 30 cm away from the roof. The top left distance sensor 6 is at a height of 20 cm from the roof, the top right sensor left-right adjusting device 4 controls the top right sensor front-rear adjusting device 5 to realize left-right movement, and the top right sensor front-rear adjusting device controls the top right distance sensor 6 to realize front-rear movement, which is equivalent to that the top right distance sensor 12 can move front-rear-left-right on the roof plane, and the sensor movement adjusting mechanism is located near the top of the vehicle tail on the right side of the vision system. The top front distance sensor 15 is at a height of 20 cm from the roof, the top front sensor left-right adjusting device 13 controls the top front sensor front-rear adjusting device 14 to realize left-right movement, and the top front sensor front-rear adjusting device 14 controls the top front distance sensor 15 to realize front-rear movement, which is equivalent to that the top front distance sensor 15 can move front-rear-left-right on the roof plane, and the sensor movement adjusting mechanism is located near the roof of the front seats of the vehicle on the right side.

[0047] The limiting device includes right front wheel front and rear limiting 16, right front wheel left and right limiting 17, left front wheel front and rear limiting 18, left front wheel left and right limiting 19, right rear wheel left and right limiting 20, right rear wheel front and rear limiting 21, left rear wheel front and rear limiting 22 and left rear wheel left and right limiting 23. Among them, the right front wheel front and rear limiting 16 and the left front wheel front and rear limiting 18 are respectively located at the front of the left and right tires of the front of the automobile, limiting the automobile from deviating forward. The right rear wheel front and rear limiting 21 and the left rear wheel front and rear limiting 22 are respectively located at the back of the tires behind the automobile, limiting the automobile from deviating backward. The right front wheel left and right limiting 17 and the right rear wheel left and right limiting 20 are respectively located at the side of the front and rear tires on the right side of the automobile, limiting the automobile from deviating to the right. The left front wheel left and right limiting 19 and the left rear wheel left and right limiting 23 are respectively located at the side of the front and rear tires on the left side of the automobile, limiting the automobile from deviating to the left.

[0048] The right side detection component includes a right front side distance sensor 24, a right front side sensor front and back adjusting device 25, a right front side sensor up and down adjusting device 26, a right front side detection camera 27, a first right front side light source 28, a second right front side light source 29, a right rear side distance sensor 30, a right rear side sensor front and back adjusting device 31, a right rear side sensor up and down adjusting device 32, a right rear side detection camera 33, a first right rear side light source 34, and a second right rear side light source 35. The right front side detection camera 27 is located at the right side of the right front door and the right side of the vehicle body link position, with an effective measurement distance of about 100 cm, a distance from the ground height of more than 80 cm, and a camera surface parallel to the front and back direction of the vehicle facing left. The first right front side light source 28 and the second right front side light source 29 are perpendicular to each other, wherein the first right front side light source 28 is parallel to the front and back direction of the vehicle, the second right front side light source 29 is parallel to the up and down direction of the vehicle, and both are installed parallel to the side of the vehicle body, with the light direction pointing to the center of the camera field of view, and the light source height is about 30 cm away from the right side of the vehicle. The right front side distance sensor 24 is 20 cm away from the right front panel of the vehicle, the right front side sensor up and down adjusting device 26 controls the right front side sensor front and back adjusting device 25 to realize up and down movement, and the right front side sensor front and back adjusting device 25 controls the right front side distance sensor 24 to realize front and back movement, which is equivalent to that the right front side distance sensor 24 can move front and back and up and down in the right side plane of the vehicle, and the sensor movement adjusting mechanism is located near the front of the vehicle. The right rear side detection camera 33 is located at the right side of the right rear fender and the right rear bumper connection position of the vehicle, with an effective measurement distance of about 100 cm, a distance from the ground height of more than 80 cm, and a camera surface parallel to the front and back direction of the vehicle facing left. The first right rear side light source 34 and the second right rear side light source 35 are perpendicular to each other, wherein the first right rear side light source 34 is parallel to the front and back direction of the vehicle, the second right rear side light source 35 is parallel to the up and down direction of the vehicle, and both are installed parallel to the side of the vehicle body, with the light direction pointing to the center of the camera field of view, and the light source height is about 30 cm away from the right side of the vehicle. The right rear side distance sensor 30 is 20 cm away from the right front panel of the vehicle, the right rear side sensor up and down adjusting device 32 controls the right rear side sensor front and back adjusting device 31 to realize up and down movement, and the right rear side sensor front and back adjusting device 31 controls the right rear side distance sensor 30 to realize front and back movement, which is equivalent to that the right rear side distance sensor 30 can move front and back and up and down in the right side plane of the vehicle, and the sensor movement adjusting mechanism is located near the tail of the vehicle.

[0049] The left side detection component includes a left front side distance measuring sensor 36, a left front side sensor front and back adjusting device 37, a left front side sensor up and down adjusting device 38, a left front side detection camera 39, a first left front side light source 40, a second left front side light source 41, a left rear side distance measuring sensor 42, a left rear side sensor up and down adjusting device 43, a left rear side sensor front and back adjusting device 44, a left rear side detection camera 45, a first left rear side light source 46 and a second left rear side light source 47. The left front side detection camera 39 is located at the left side of the left front door and the body of the vehicle, with an effective measuring distance of about 100 cm, a distance from the ground of more than 80 cm, and a camera plane parallel to the front and back direction of the vehicle and facing right. The first left front side light source 40 and the second left front side light source 41 are perpendicular to each other, wherein the first left front side light source 40 is parallel to the front and back direction of the vehicle, the second left front side light source 41 is parallel to the up and down direction of the vehicle, and both are installed parallel to the side of the vehicle body, with a light direction pointing to the center of the field of view of the camera, and a light source height of about 30 cm from the left side of the vehicle. The left front side distance measuring sensor 36 is 20 cm away from the right front panel of the vehicle, the left front side sensor up and down adjusting device 38 controls the left front side sensor front and back adjusting device 37 to realize up and down movement, and the left front side sensor front and back adjusting device 37 controls the left front side distance measuring sensor 36 to realize front and back movement, which is equivalent to that the left front side distance measuring sensor 36 can realize front and back up and down movement in the left side plane of the vehicle, and the sensor movement adjusting mechanism is located near the front of the vehicle. The left rear side detection camera 44 is located at the left side of the left rear fender and the left rear bumper of the vehicle, with an effective measuring distance of about 100 cm, a distance from the ground of more than 80 cm, and a camera plane parallel to the front and back direction of the vehicle and facing right. The first left rear side light source 46 and the second left rear side light source 47 are perpendicular to each other, wherein the first left rear side light source 46 is parallel to the front and back direction of the vehicle, the second left rear side light source 47 is parallel to the up and down direction of the vehicle, and both are installed parallel to the side of the vehicle body, with a light direction pointing to the center of the field of view of the camera, and a light source height of about 30 cm from the right side of the vehicle. The left rear side distance measuring sensor 42 is 20 cm away from the right front panel of the vehicle, the left rear side sensor up and down adjusting device 43 controls the left rear side sensor front and back adjusting device 45 to realize up and down movement, and the left rear side sensor front and back adjusting device 45 controls the left rear side distance measuring sensor 42 to realize front and back movement, which is equivalent to that the left rear side distance measuring sensor 42 can realize front and back up and down movement in the left side plane of the vehicle, and the sensor movement adjusting mechanism is located near the tail of the vehicle.

[0050] In an embodiment of the present application, before the system performs full-station automatic operation, single-station manual operation is required to obtain image distortion correction files of each station, fitting functions of magnification and height, coordinate conversion files between images and punching robots, and position relationships of image feature surfaces and punching points on the positioned punching surface. Through the obtained files and data, automatic positioning and punching operation of each station is realized.Figure 5 , as shown in a single-station manual operation flowchart, the single-station manual operation interface provides distortion calibration, punch robot N-point calibration, magnification acquisition, template making and real-time positioning operation buttons, and corresponding processes can be executed according to the user's relevant operation. The specific execution process is as follows:

[0051] The single-station operation process first executes the station selection process. Before entering the single-station manual operation interface, the user needs to select the station to be operated, and after completing the selection, enters the manual operation interface.

[0052] If the user clicks the "distortion calibration" button, the camera distortion calibration process will be executed. By executing the camera distortion calibration process, the distortion calibration file under different angles of the feature surface will be obtained, which can be used for image distortion correction when the feature surface and the camera surface are at different angles.

[0053] If the user clicks the "punch robot N-point calibration" button, the punch robot N-point calibration process will be executed. By executing the punch robot N-point calibration process, the N-point calibration file is obtained, realizing the conversion of image coordinates to physical coordinates in the punch robot coordinate system.

[0054] If the user clicks the "magnification acquisition" button, the magnification acquisition process will be executed. By executing the magnification acquisition process, the first-order polynomial relationship function between the feature surface height and the image magnification is obtained, and according to this function, the image coordinate conversion of the feature surface at different heights can be realized, thereby obtaining the accurate physical coordinates of the feature points.

[0055] If the user clicks the "template making" button, the template making process will be executed. By executing the template making process, the position information of the image processing feature points obtained by the camera and the position information of the punch points on the punch surface can be obtained, and the positional relationship between the feature points and the punch points can be obtained accordingly.

[0056] If the user clicks the "real-time positioning" button, the single-station real-time positioning process will be executed. Through the single-station real-time positioning process, the position information of the punch points will be obtained by real-time positioning of the currently selected station, and then the punch robot will be controlled to perform the punching operation. In the real-time positioning process, emergency pause operation can be performed, and after the user presses the "emergency pause" button, the emergency pause processing process will be executed.

[0057] When the system is in a single-station operation standby state, the user can exit the single-station operation by pressing the "exit" button.

[0058] In an embodiment of the present application, a camera is required to be installed at each station of the detected object for visual positioning. Since the feature surface to be detected and the camera surface will present different angles due to different placement positions of the detected object, the camera distortion calibration process of the present system will calibrate the distortion of each different angle of the feature surface and the camera surface. In the template making process, the corresponding distortion calibration file is called for image distortion correction according to the measured angle of the feature surface and the camera surface. The camera distortion calibration process can generate distortion calibration files of the calibration board and the camera surface at different angles, and each distortion calibration file is composed of the perspective distortion of the corresponding angle image and the radial distortion parameters of the image, which are used for image distortion correction. The camera distortion calibration process includes the following steps:

[0059] It is judged whether the user clicks the "distortion calibration" button. If yes, the distortion calibration is performed, otherwise the user operation is waited.

[0060] The checkerboard calibration board is controlled to the initial position. The calibration board is placed on the detected object, i.e. the detection feature point detection height of the car, and after the placement is completed, the laser sensor is controlled to collect the height information of each position.

[0061] According to the obtained laser sensor height information, the offset angle of the calibration board and the camera surface is calculated. In the distortion calibration process, three laser sensors are installed, i.e. a laser sensor is installed on the left side behind the calibration board, a laser sensor is installed on the right side behind the calibration board, and a laser sensor is installed in front of the right side. The distance between the top left distance sensor and the top right distance sensor is denoted as H1, the distance between the top right distance sensor and the top front distance sensor is denoted as H2, the height obtained by the top left distance sensor is denoted as h1, the height obtained by the top right distance sensor is denoted as h2, and the height obtained by the top front distance sensor is denoted as h3. According to the three heights, the offset angles including the left-right offset angle and the front-back offset angle with the camera surface can be obtained, wherein the left-right offset angle is denoted as θ1 and the front-back offset angle is denoted as θ2. The calculation formula is as follows:

[0062]

[0063]

[0064] It is judged whether the above obtained θ1 and θ2 are stable within the set threshold range. If yes, it is considered that the checkerboard calibration board is in a stable state, otherwise it is considered that the checkerboard calibration board is still in a moving state. At this time, it is judged whether the angle measurement times exceed the set value. If yes, the present distortion calibration operation is ended, and a prompt "calibration board is not stable, please place the calibration board again" is given, otherwise the measurement of θ1 and θ2 is continued.

[0065] In the range of ±10°, distortion calibration is performed every 1° for left-right and front-back deflection angles. When calibrating the left-right deflection angle, the front-back deflection angle is kept at 0°, and the left-right deflection angle is calibrated from -10° to 10°, with the chessboard calibration board angle offsetting 1° each time. The front-back deflection angle is calibrated in the same way. According to the current calibration process, it is determined whether the current measured angle is within the required range. For example, if the distortion calibration at -10° of the left-right deflection angle is being performed, it is determined whether θ1 is within the threshold range around -10° set by the system, and θ2 is also required to be 0°. If the conditions are met, it is considered that the angle is within the required range, and the next step is performed. Otherwise, the chessboard calibration board is controlled to move and adjust the angle of the chessboard calibration board to meet the angle requirements.

[0066] If the angle meets the requirements, the camera is triggered to take a picture and collect an image.

[0067] The collected image is subjected to distortion calibration to generate a distortion calibration file.

[0068] After generating the distortion calibration file, the image collected by the camera is loaded with the distortion calibration file for distortion correction.

[0069] The horizontal and vertical pixel spacings of the chessboard grids in the upper, lower, left, right, and middle positions of the chessboard calibration board are obtained.

[0070] It is determined whether the horizontal and vertical pixel spacings of the five positions are within the threshold range around the standard value. If so, it indicates that the distortion calibration is successful, the image at the current angle can be correctly corrected, and the generated distortion calibration file and the corresponding left-right and front-back deflection angles of the calibration board are stored. Otherwise, it indicates that the distortion correction fails, the distance from the laser sensor to the calibration board is reacquired, the angle of the calibration board is calculated, and the distortion calibration at the current angle is reperformed.

[0071] If the distortion calibration at the current angle is successful, it is determined whether all the angles have completed the distortion calibration. If so, the distortion calibration is completed, and the system waits for user operation. Otherwise, the angle of the calibration board is adjusted to continue the distortion calibration at the next angle.

[0072] In an embodiment of the present application, the visual system performs feature positioning to obtain pixel coordinates of feature points, and the punching robot operates according to a coordinate system defined by the punching robot. Therefore, the visual coordinates need to be converted into the punching robot coordinates, and the two coordinate systems are combined to control the punching robot through the relative position offset detected by the visual system. The present system generates an N-point calibration file through an N-point calibration process of the punching robot, and realizes the conversion between the visual coordinates and the punching robot coordinates in real time. The N-point calibration process of the punching robot includes:

[0073] After entering the N-point calibration interface, the user is waited for to perform an N-point calibration operation. If the user clicks the "N-point calibration" button, an N-point calibration is performed, otherwise the user is waited for to operate.

[0074] The punching robot makes a world coordinate system according to the feature surface of the vehicle body. The punching robot uses the world coordinate system during operation, and the coordinate system can be remade according to the actual working condition. The system takes the roof camera surface of the vehicle body as the XY coordinate system, and the vertical direction as the Z direction of the coordinate system. By remaking the world coordinate system in this way, the visual coordinate system and the XY coordinate system direction of the punching robot can be kept consistent.

[0075] The tool TCP (tool center point) of the punching robot is set. The tool TCP position of the punching robot is set to the N-point calibration mark head, that is, the posture calculation reference point of the punching robot is set to the mark head. The posture calculation reference point of the punching robot is converted to the mark head, so that the change of the posture of the punching robot will not affect the coordinates of the mark head. The coordinates of the mark head are also the actual position of the punching robot in the N-point calibration process.

[0076] The distance from the upper laser sensor to the marking plate is obtained, and the front-back and left-right angles of the marking plate are calculated.

[0077] It is judged whether the angle is within the angle range of distortion correction. If yes, the punching robot is controlled to mark N points in the camera field of view range, otherwise the user is prompted to adjust the angle of the calibration plate, and the user is waited for to operate.

[0078] After the punching robot marks N points, the punching robot is controlled to exit the marking area and leave the field of view range.

[0079] The camera is controlled to capture an image, and according to the measured left-right and front-back angles of the marking plate and the camera surface, distortion correction is performed by calling the distortion calibration file corresponding to the angle.

[0080] The center pixel coordinates of the N marking points are found by a center finding algorithm.

[0081] According to the center pixel coordinates of the N marking points and the corresponding physical coordinates of the punching robot, N-point calibration is performed to obtain an N-point calibration file. The N-point calibration file is a conversion matrix of the camera visual coordinate system and the punching robot coordinate system in the XY two-dimensional plane. Through the conversion matrix, the pixel coordinate value of the camera can be converted into the physical coordinate value of the punching robot.

[0082] After obtaining the N-point calibration file, N-point coordinate conversion is performed on the current image to verify whether the generated calibration file is correct.

[0083] Carry out circle search on the marking points in the image after distortion correction, and find the respective center pixel coordinates of the N points. Call the N-point calibration file generated above to convert the center pixel coordinates of the N points into physical coordinates of the punch machine.

[0084] Compare the physical coordinates of the N points obtained with the physical coordinates of the N points recorded in the punch robot marking process, and obtain statistical data. Determine whether the difference is within the set threshold range. If yes, store the N-point calibration file and the height value measured by the laser sensor. Otherwise, prompt "N-point calibration coordinate conversion is abnormal, please re-perform N-point calibration", and wait for user operation.

[0085] In an embodiment of the present application, due to the physical characteristics of machine vision imaging, the measured surface at different heights appears to be larger near and smaller far in the image. When performing physical coordinate conversion of feature points, the distance of the feature surface relative to the laser sensor will be different between the template making process and the real-time positioning and punching process. Therefore, the coordinate standards of the same image collected in the two processes are different, and direct coordinate conversion will have positional deviation. The present system converts the feature point coordinates of the images collected in the two processes to the image coordinates in the N-point calibration, and then converts the image coordinates to physical coordinates. Since the image magnification can quantitatively represent the size of the measured object at different heights, this process converts through the magnification at different heights. The image magnification acquisition process includes the following steps:

[0086] After entering the magnification acquisition interface, wait for the user to perform the magnification acquisition operation. If the user clicks the "magnification acquisition" button, perform magnification acquisition, otherwise wait for user operation.

[0087] In combination with the tool precision at the vehicle body positioning and punching station and the condition of the vehicle body itself, the present system will acquire the magnification of the measured surface of the vehicle body at a distance of ±10cm from the standard distance h of the laser sensor, and fit to obtain the magnification and distance fitting function. The specific measurement process is from (standard distance h-10cm) to (standard distance h+10cm), and each measurement is performed at an interval of 5mm. Therefore, first control the calibration plate to (standard distance h-10cm), and then acquire the distance from the upper laser sensor to the calibration plate. The left and right angle θ1' and the front and rear angle θ2' of the calibration plate are obtained by using the angle calculation method in the distortion calibration process.

[0088] Determine whether θ1' and θ2' are stable. If not, it means that the calibration plate has not completely stopped. Then determine whether the number of measurements exceeds the set value. If yes, end this magnification acquisition, and wait for user operation. Otherwise, continue to acquire the distance from the laser sensor to the calibration plate and calculate the angle.

[0089] If θ1' and θ2' remain stable, it is determined whether θ1' and θ2' are within a required range. When the camera and the measured surface remain completely perpendicular, the collected image is subjected to distortion correction to remove the distortion of the lens itself, and there is no distortion at each position of the image. Therefore, in order to remove the influence of the distortion introduced by the angle on the magnification, it is required that θ1' and θ2' are around 0°, within ±0.1°. If the condition is met, the camera is triggered to take an image and save it. If θ1' and θ2' are not within the required range, the angle of the calibration plate is adjusted, and θ1' and θ2' are obtained again.

[0090] It is determined whether the images at all height positions have been collected. If yes, the pixel spacing of the same position of the checkerboard of the calibration plate is obtained by processing all the corresponding images. The checkerboard at the middle of the image is selected because the distortion at the middle position of the image is the smallest, and the error of the obtained pixel spacing is the smallest. Then, the proportional relationship between the pixel spacing and the actual physical size of the checkerboard is obtained, which is the image magnification at the calculated height. If the images at all height positions have not been collected, the height of the calibration plate is adjusted for image collection at the next height.

[0091] After the magnifications at all heights are obtained, a first-order polynomial fitting is performed according to the corresponding height to obtain fitting coefficients k and b, i.e., m=k*h+b, where m is the magnification, and h is the distance value measured by the laser sensor. The fitting coefficients k and b are stored.

[0092] In an embodiment of the present application, the template making process includes the following steps:

[0093] The template making button is tapped to enter the template making process interface.

[0094] It is determined whether the start making template button is tapped.

[0095] If no, it is determined whether the exit template making button is tapped. If yes, the template making interface is exited. If no, the system does not act and waits for the user to tap the start making template button.

[0096] If yes, the system determines whether the punching robot has switched the tool TCP (tool center point). That is, the system switches the TCP position of the punching robot from the marker head TCP to the punching head TCP. After the switching is completed, the punching robot moves and adjusts the position in the workpiece coordinate system of the punching robot based on the punching head TCP.

[0097] If no, the system first controls the punching robot to switch the tool TCP, and then controls each laser sensor (including the top three and the four on the side) to obtain the distance from the vehicle body. If yes, each laser sensor (including the top three and the four on the side) obtains the distance from the vehicle body.

[0098] The system determines whether the distances obtained by the laser sensors remain stable, i.e., whether each distance value is less than a set threshold.

[0099] If not, the system counts the number of measurements and determines whether the number of measurements exceeds a set value. If not, the system continues to determine whether the distances obtained by the laser sensors remain stable, i.e., whether each distance value is less than a set threshold. If yes, the system prompts that the vehicle body is not placed stably, and returns to determine whether the start template making button is pressed.

[0100] If yes, the vehicle body direction deviation angles (the rotation angle α of the vehicle body around the Y-axis direction, the rotation angle β of the vehicle body around the X-axis direction, and the rotation angle γ of the vehicle body around the Z-axis direction) are calculated according to the distances from the vehicle body obtained by each laser sensor (the top three sensors and the right or left two sensors).

[0101] It is determined whether the obtained direction deviation angles are within a set range.

[0102] If not, it is prompted that the vehicle body position is abnormal, and the user needs to reposition the vehicle body. The process returns to determine whether the start template making button is pressed.

[0103] If yes, the vehicle body feature acquisition process is performed.

[0104] The system determines whether the vehicle body feature acquisition is abnormal.

[0105] If not, the system controls the punching robot to move the punching head to a punching position. The system records the coordinates of the feature point intersection, the angle of the straight line, and the coordinates of the punching point position, respectively. The corresponding laser sensor position data is saved, and the process returns to determine whether the start template making button is pressed.

[0106] If yes, according to the type of abnormality (including not finding the vehicle body feature and not finding the joint at the vehicle body feature), it is prompted to confirm the vehicle body position, and the template making process is ended.

[0107] Further, in an embodiment of the present application, the vehicle body feature acquisition process in the template making is as follows:

[0108] The system controls the camera to capture a feature surface image of the vehicle body.

[0109] According to the vehicle body deviation angles, the system calls a corresponding distortion calibration file to perform image distortion correction.

[0110] According to the vehicle body sheet metal features, the system performs sheet metal indirect joint feature template matching.

[0111] The system determines whether the feature is successfully found, i.e., whether the matching is successful. If not, it is prompted that the vehicle body feature is not found, please confirm the vehicle body position, and the process is ended. If yes, the found feature is position corrected, and the two straight lines at the joint are searched, respectively.

[0112] The system determines whether the two straight lines are successfully found. If not, it is prompted that the joint at the feature of the vehicle body is not found, the position of the vehicle body is confirmed, and the process ends. If yes, the intersection of the two straight lines is calculated, and the physical coordinates of the intersection are obtained through coordinate conversion.

[0113] In an embodiment of the present application, the real-time positioning and punching process includes the following steps:

[0114] The real-time positioning and punching button is pressed, and the real-time positioning and punching process interface is entered.

[0115] It is determined whether the start real-time positioning button is pressed.

[0116] If not, it is determined whether the exit single-station operation button is pressed. If yes, the real-time positioning and punching interface is exited. If not, the system does not act, and waits for the user to press the start real-time positioning button.

[0117] If yes, the vehicle body angle acquisition process is performed.

[0118] The system determines whether the vehicle body angle acquisition is successful. If not, it is prompted that the positioning is abnormal, and the determination of whether the start real-time positioning button is pressed is continued. If yes, the vehicle body feature acquisition process is performed.

[0119] The system determines whether the vehicle body feature acquisition is successful. If not, it is prompted that the feature of the vehicle body is not found or the joint at the feature of the vehicle body is not found, the position of the vehicle body is confirmed, and the determination of whether the start real-time positioning button is pressed is continued. If yes, the new punching point position is obtained through calculation according to the feature points in the template making, the punching points, the deflection angles of each direction of the vehicle body, and the feature point coordinates of real-time positioning.

[0120] The system determines whether the vehicle body deflection position is within the set range. If not, it is prompted that the position of the vehicle body is abnormal,

[0121] The user needs to reposition the vehicle body, and the determination of whether the start real-time positioning button is pressed is continued. If yes, the punching robot is controlled to adjust the posture, and then runs above the new punching point. The laser sensor obtains the distance from the new punching point in real time.

[0122] It is determined whether the distance is within the set range. If not, the punching robot continues to approach the new punching point by a certain distance, and then determines whether the distance is within the set range. If yes, the system controls the punching robot to punch. After the end, the determination of whether the start real-time positioning button is pressed is continued.

[0123] Further, the vehicle body angle acquisition process in the real-time positioning and punching process is as follows:

[0124] The sensor approaching template point process is performed.

[0125] The system judges whether the approximation is completed or not; if not, the sensor approximation template point process is continuously executed; if yes, the non-camera plane angle is calculated according to the distance measured by the laser sensor.

[0126] The vehicle body feature acquisition process is executed to obtain the camera plane angle. Thus, the rotation angles of the vehicle body in three directions are obtained.

[0127] The vehicle body angle acquisition process is different for the vehicle body top real-time positioning and punching and the vehicle body side real-time positioning and punching.

[0128] Taking the vehicle body top positioning and punching as an example, the workpiece plane coordinate system established by the punching robot is shown in Figure 8 The positive direction of the Y axis is the vehicle head direction.

[0129] The non-camera plane angle obtained by the vehicle body top real-time positioning and punching includes the rotation angle a of the vehicle body around the Y axis (the vehicle left-right inclination angle) and the rotation angle β of the vehicle body around the X axis (the vehicle front-rear inclination angle), and the top camera plane angle refers to the rotation angle γ of the vehicle body around the Z axis (the rotation angle in the vehicle top plane parallel to the top camera plane).

[0130] The camera plane angle obtained by the vehicle body side real-time positioning and punching refers to the rotation angle β of the vehicle body around the X axis obtained by the side camera, and the non-camera plane angle includes the rotation angle a around the Y axis and the rotation angle γ of the vehicle body around the Z axis obtained by the top camera.

[0131] In an embodiment of the present application, since the angles vary at different positions in the curved surface space, the position deviation of the point laser sensor may exist when collecting the distance, thereby causing the inaccuracy of the obtained distance. The sensor approximation process is used in the present system to accurately obtain the position of the point laser sensor on the feature plane during the template making. The sensor approximation template point process in the vehicle body angle acquisition process is as follows.

[0132] Each laser sensor obtains the distance from the vehicle body to coarsely position.

[0133] The system judges whether the distances obtained by each laser sensor remain stable, i.e., whether each distance value is less than the set threshold. If not, the system counts the measurement times and judges whether the times exceed the set value; if not, the coarse positioning is returned to continue; if yes, the system prompts that the vehicle body is not placed stably, and the process is ended.

[0134] If yes, the angles of the vehicle body in each direction are calculated according to the distances from the vehicle body obtained by each laser sensor.

[0135] The system judges whether the angles are within the set range; if not, the vehicle body position is abnormal, the user needs to re-place the vehicle body, and the process is ended; if yes, the vehicle body feature acquisition process is executed.

[0136] The position difference between the target point and the template point is calculated, the deviation of the sensor moving to the template point position is calculated, and the sensor is controlled to move.

[0137] After moving, the laser sensor measures the distance from the vehicle body, and according to the distance obtained, the deflection angle of the vehicle body in each direction is calculated.

[0138] The above sensor approach template point process is different for two cases: real-time positioning and punching on the top of the vehicle body and real-time positioning and punching on the side of the vehicle body.

[0139] In the real-time positioning and punching on the top of the vehicle body, each laser sensor is referred to as a top sensor (6, 12, 15), and the calculated deflection angle of the vehicle body in each direction includes the rotation angle a of the vehicle body around the Y-axis direction and the rotation angle β of the vehicle body around the X-axis direction.

[0140] In the real-time positioning and punching on the side of the vehicle body, each laser sensor includes a top sensor (6, 12) and a sensor for measuring the working distance of the camera corresponding to the positioning and punching position.

[0141] The vehicle feature acquisition process in real-time positioning and punching is consistent with the vehicle feature acquisition process during template making, except that "finding the intersection of the two straight lines, and converting the intersection point to physical coordinates" is changed to "finding the intersection of the two straight lines, converting the intersection point to pixel coordinates of the template image according to the height of the real-time feature point and the height of the template feature point, and converting the intersection point to physical coordinates".

[0142] The formula for converting the intersection point (i.e., the feature point) of the two straight lines in the vehicle feature acquisition process in real-time positioning and punching to the pixel coordinates of the template image is as follows:

[0143]

[0144]

[0145] where m0 represents the magnification of the feature point in template making, which is calculated by substituting the distance value measured by the corresponding station sensor during template making into the first-order fitting function in the magnification acquisition process; m1 represents the magnification of the feature point in real-time positioning, which is calculated by substituting the distance value measured by the corresponding station sensor after position correction in real-time positioning into the first-order fitting function in the magnification acquisition process; point (X, Y) represents the pixel coordinates of the feature point in real-time positioning, point (X center , Y center ) represents the pixel coordinates of the image center point in real-time positioning, and point (X′, Y′) represents the pixel coordinates of the feature point in real-time positioning after magnification compensation.

[0146] Point (X′, Y′) is converted to physical coordinates point (X1, Y1).

[0147] Calculating new punch point position in real-time positioning punch process The specific calculation process is as follows:

[0148] Case 1: The three direction deflection angles of the vehicle body measured in real-time positioning are the same as those in template making;

[0149]

[0150]

[0151] Wherein, point (X0, Y0) represents the physical coordinates of the pixel coordinates of the feature point in template making obtained through coordinate conversion, and point (X1, Y1) represents the physical coordinates of the punch point in template making.

[0152] Case 2: The vehicle body deflection angles (α and β) measured in real-time positioning are the same as those in template making;

[0153]

[0154]

[0155] Wherein, angle γ Δ is equal to the vehicle body rotation angle γ1 in real-time positioning minus the vehicle body rotation angle γ0 in template making, represents the X-axis direction offset amount of the punch point caused by the rotation of the vehicle body in real-time positioning by angle γ Δ relative to template making, represents the Y-axis direction offset amount of the punch point caused by the rotation of the vehicle body in real-time positioning by angle γ Δ relative to template making.

[0156] Case 3: The vehicle body deflection angle β measured in real-time positioning is the same as that in template making;

[0157] (1) If the line segment composed of the punch point and the feature point in template making is perpendicular to the template plane;

[0158]

[0159]

[0160] Wherein, angle α Δ is equal to the vehicle body rotation angle α1 in real-time positioning minus the vehicle body rotation angle α0 in template making, represents the X-axis direction offset amount in case 2 multiplied by cos(α Δ ), which is the change of the X-axis direction offset amount in case 2 caused by the left and right inclination of the vehicle body, i.e. it represents that the offset amount becomes its projection on the template plane.​

[0161] (2) If the line segment formed by the punch point and the feature point when the template is made has an angle η with the template plane, the punch point is lower than the feature point, and the punch point is inclined downward relative to the feature point (or the punch point is higher than the feature point, and the punch point is inclined upward relative to the feature point);

[0162]

[0163]

[0164] wherein the angle η X is equal to the projection angle of the angle formed by the line segment formed by the punch point and the feature point when the template is made and the template plane on the plane formed by the X axis and the Z axis, represents the X axis offset amount that needs to be compensated when the real-time positioning of the vehicle body is different from the left and right inclination angles when the template is made and there is an angle η.

[0165] (3) If the line segment formed by the punch point and the feature point when the template is made has an angle η with the template plane, the punch point is higher than the feature point, and the punch point is inclined downward relative to the feature point (or the punch point is lower than the feature point, and the punch point is inclined upward relative to the feature point);

[0166]

[0167]

[0168] Case 4: The vehicle body deflection angle α measured during real-time positioning is the same as when the template is made;

[0169] (1) If the line segment formed by the punch point and the feature point when the template is made has no angle with the template plane;

[0170]

[0171]

[0172] wherein the angle β Δ is equal to the vehicle body rotation angle β1 during real-time positioning minus the vehicle body rotation angle β0 when the template is made.

[0173] represents the Y axis direction offset amount in case 2 multiplied by cos(β Δ ), which is the change of the Y axis direction offset amount in case 2 caused by the forward and backward inclination of the vehicle body, i.e., it represents that the offset amount becomes its projection on the template plane.

[0174] (2) If the line segment formed by the punch point and the feature point when the template is made has an angle η with the template plane, the punch point is lower than the feature point, and the punch point is inclined downward relative to the feature point (or the punch point is higher than the feature point, and the punch point is inclined upward relative to the feature point);

[0175]

[0176]

[0177] wherein, angle η Y is the projection angle of the angle η between the line segment composed of the punch point and the feature point and the template plane on the plane composed of the Y axis and the Z axis. represents the Y axis offset that needs to be compensated when the real-time positioning of the vehicle body is different from the front and rear inclination angles during template making and there is an angle η.

[0178] (3) If the line segment composed of the punch point and the feature point during template making has an angle η with the template plane, the punch point is higher than the feature point, and the punch point is inclined downward relative to the feature point (or the punch point is lower than the feature point and the punch point is inclined upward relative to the feature point);

[0179]

[0180]

[0181] Case 5: The inclination angles α and β of the vehicle body measured during real-time positioning are different from those during template making;

[0182] (1) If the line segment composed of the punch point and the feature point during template making has no angle with the template plane;

[0183]

[0184]

[0185] (2) If the line segment composed of the punch point and the feature point during template making has an angle η with the template plane, the punch point is lower than the feature point, and the punch point is inclined downward relative to the feature point (or the punch point is higher than the feature point and the punch point is inclined upward relative to the feature point);

[0186]

[0187]

[0188] (3) If the line segment composed of the punch point and the feature point during template making has an angle η with the template plane, the punch point is higher than the feature point, and the punch point is inclined downward relative to the feature point (or the punch point is lower than the feature point and the punch point is inclined upward relative to the feature point);

[0189]

[0190]

[0191] In an embodiment of the present application, the full-station automatic operation process completes positioning and punching operation on all stations to be punched in the set order. After completing positioning and punching operation on all stations, the punching robot is operated to the zero point according to the zero return trajectory, and waits for the arrival of the next detected object. As shown in FIG. 8, the full-station automatic operation process includes the following steps: Figure 6

[0192] After entering the full-station operation interface, the user is waited to click the "full-station operation" button. If the user clicks the "full-station operation", the vehicle body positioning judgment process is executed, otherwise the user operation is waited.

[0193] The vehicle body running line is composed of multiple large plates. The front and rear tires and tire side limit devices are installed on the large plates. After the vehicle body is placed in the limit position in turn, the large plates run forward in turn. When the large plates run to the visual positioning station, the limit signal is triggered by the limit device at the bottom of the large plates. Therefore, the vehicle body positioning judgment process first judges whether the limit signal is received. If the limit signal is received, the laser sensor at the visual positioning station is controlled to obtain the height value from the vehicle roof in real time. If the height value remains stable within the set time, it is considered that the vehicle body has been positioned and stabilized, and the single-station real-time positioning process is executed, otherwise the height value of the laser sensor is continuously obtained to judge whether the height value remains stable within the set time.

[0194] The system executes the single-station real-time positioning process on the left and right sides of the top, front and rear sides of the vehicle body in turn, a total of 6 stations. The left and right sides of the top station execute the single-station real-time positioning process first, because the front and rear angles and the left and right angles of the top relative to the camera plane are the smallest, and the rotation angle of the top in the camera plane can be accurately measured by vision. In this way, the three-dimensional angle offset of the vehicle body relative to the template for making the vehicle body is the most accurate. After completing the single-station real-time positioning process of the two sides of the top, the three angles of the vehicle body in the three-dimensional direction relative to the template for making the vehicle body are obtained, among which the left and right angles and the rotation angle in the camera plane can be used as the rotation angle required in the side single-station real-time positioning process. The system is installed with a total of 7 laser sensors, among which 3 laser sensors are installed on the top of the vehicle body to obtain the front and rear angles and the left and right angles of the camera plane of the vehicle body, and among them, the left and right laser sensors below the camera on the top are also used to obtain the actual height of the vehicle body; 1 laser sensor is installed on each of the front and rear stations on the left and right sides of the vehicle body to obtain the distance of the vehicle body. Therefore, the system is installed with a total of 7 laser sensors.

[0195] It is judged whether all stations have completed punching. If yes, the punching robot is controlled to return to the standby point to wait for the arrival of the next vehicle body, otherwise the vehicle body real-time positioning of the next station is continued.

[0196] ​The surface of the car being inspected is not perfectly flat; the sheet metal has curvature. If the sensor position remains unchanged, there will be a discrepancy between the distance information collected by the laser sensor and the actual measurement when the template is made. The angle deviation of the car obtained through distance measurement will not be the correct angle. Therefore, the camera is set to be fixed, while the laser sensor is movable.

[0197] After the laser sensor approximation process is completed, the front-to-rear and left-to-right deflection angles of the vehicle roof can be calculated based on the laser sensor readings. An example of the solution method is illustrated below: See [link to example]. Figure 7 As shown, if circle 1 is the initial position, the measured distances to the left rear and right rear are HG and AC respectively, and the distance between the two sensors is AH. Circle 2 is the re-parking position. If the sensor does not follow the movement, the measured positions are HD and AB. In actual use, the sensor will approximate the position based on visual positioning, so only the distance after the sensor follows the movement needs to be calculated. The sensor follows the movement I and J, which are the new positions of the sensor, and the measured positions are IM and JK, as shown in the figure. The car angle does not change significantly. When the sensor does not move during template creation, the deflection angle of the vehicle body in a certain direction can be calculated through the sensor distance relationship. However, in real-time detection, after the sensor moves through the approximation method, the deflection angle of the vehicle body in a certain direction can be calculated again through the sensor distance relationship. The calculation formula is as follows:

[0198] θ is the angle between AH and GC, and is the vehicle body angle during template creation.

[0199] θ' is the angle between AH and MK, and is the angle of the vehicle body during real-time positioning.

[0200] The above formula can be used to calculate the vehicle's tilt angle in a specified direction during template creation and real-time positioning. According to the definitions above, the vehicle's α angle is the left-right tilt angle, and β angle is the front-back tilt angle. Therefore, when calculating the left-right tilt angle, θ in this formula corresponds to the vehicle's α angle during template creation, and θ' corresponds to the vehicle's α angle during real-time positioning. The deviation angle Δ during real-time positioning can be obtained through (θ' - θ), specifically α. Δ ,β Δ and γ Δ .

[0201] Example: Assuming the workstation is the camera workstation on the right side of the vehicle roof, according to the above image, the change in the position of the feature on the roof is calculated by the camera in the direction around the Y-axis (left and right direction of the vehicle body) as K. The control center controls the left and right adjustment device 4 of the top right sensor to move the displacement of the value K.

[0202] Suppose the station is on the right side of the roof camera station, according to the above picture display, the change of the feature position of the roof is calculated by the camera L in the X axis direction (the front and rear direction of the vehicle body), and the control center controls the right side of the roof sensor left and right adjusting device 4 to move L value displacement amount.

[0203] 6 fixed cameras collect feature point information, and 7 distance measuring sensors measure distance.

[0204] The top sensor is 12, and the distance between 6 is H1

[0205] The top sensor is 6, and the distance between 15 is H2

[0206] The right side sensor is 30, and the distance between 24 is H3

[0207] The left side sensor is 36, and the distance between 42 is H4

[0208] The measurement value of the sensor template is set to H 12 , H 6 , H 15 , H 30 , H 24 , H 36 , H 42

[0209] The measurement value of the sensor in the template making is set to h 12 , h 6 , h 15 , h 36 , h 24 , h 36 , h 42

[0210] The measurement value of the sensor after correction of the real-time measurement position is h 12-1 , h 6-1 , h 15-1 , h 30-1 , h 24-1 , h 36-1 , h 42-1 The vehicle body is in place to start making the template, and the distance measurement value of the 7 sensors is fed back to the control center to calculate the magnification.

[0211] The camera takes a picture to record the feature point information.

[0212] The vehicle body is in place again, and real-time measurement begins;

[0213] The top sensor measures the distance value h 12-1 , h 6-1 , and calculates the magnification and sends it to the control center.

[0214] The two cameras on top calculate the offset angle in Z axis direction and the XY coordinate system plane and send the data to the control center.

[0215] The control center adjusts the sensor position according to the camera parameters.

[0216] The three sensors on top calculate the offset angle of the vehicle body around the X axis and Y axis directions, and thus obtain the rotation angles of the vehicle body around the XYZ three coordinate axes.

[0217] The angle calculation formula when making the template is as follows:

[0218] Rotation angle around the X axis direction: α0=tan -1 (h 15 -h 6 ) / H2

[0219] Rotation angle around the Y axis direction: β0=tan -1 (h 12 -h 6 ) / H1

[0220] The angle calculation formula when making the template is as follows:

[0221] Rotation angle around the X axis direction: α1=tan -1 (h 15-1 -h 6-1 ) / H2

[0222] Rotation angle around the Y axis direction: β1=tan -1 (h 12-1 -h 6-1 ) / H1

[0223] Through the calculation of (β1-β0), β Δ can be obtained, and through the calculation of (α1-α0), α Δ can be obtained.

[0224] The corrected sensor position value is transmitted to the control center as the magnification correction value for the camera.

[0225] It should be understood that the example embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the present application are described in conjunction with the attached figures, it should be understood that various changes in form and detail can be made without departing from the spirit and scope of the application as defined by the appended claims.

Claims

1. A method of detecting and positioning an automobile assembly, characterized by, The positioning function of the relevant surface is realized by using a machine vision-based relevant surface positioning punching system. When the detected object runs to the positioning system, the positioning punching at the relevant surface positioning is realized by a laser sensor positioning method, a machine vision positioning method and the positional relationship of the relevant surface. The positioning system is powered on, and the system boot initialization process is performed to detect the hardware state, including the connection state detection of the two-side punching robot, the camera of each station and the laser sensor. If all the hardware is normally connected, the next operation is performed, otherwise the related hardware connection exception alarm is performed to prompt the user to repair the related hardware. The punching robot position is confirmed. If the punching robot position is not at the zero position, the zero operation is performed according to the zero trajectory. The relevant detection parameters in the system running process are read. After the system boot initialization is completed, the positioning system executes different processes according to different operations of the user. If the user selects single-station operation, the single-station manual operation process is executed. If the user selects full-station operation, the positioning system will execute the full-station automatic operation process. During the execution of the full-station automatic operation process, if an operation that needs to be executed urgently is performed, the current positioning punching operation is stopped through the emergency pause, and the punching robot is zeroed. The single-station manual operation process is used to complete the preprocessing operation before real-time positioning of the specified station and the positioning punching of the single station, including: station selection process, camera distortion calibration process, punching robot N-point calibration process, magnification acquisition process, template making process and real-time positioning process. The camera distortion calibration process is used to obtain a distortion calibration file at different angles of the feature surface, which is used for image distortion correction when the feature surface and the camera surface are at different angles. The punching robot N-point calibration process is used to obtain an N-point calibration file to realize the conversion of image coordinates to physical coordinates in the punching robot coordinate system. The magnification acquisition process is used to obtain a first-order polynomial relationship function between the feature surface height and the magnification, and according to the relationship function, the image coordinate conversion of the feature surface at different heights is realized to obtain accurate feature point physical coordinates. The template making process is used to obtain the position information of the feature points processed by the camera and the position information of the punching points on the punching surface, and the position relationship between the feature points and the punching points is obtained accordingly. The full-station automatic operation process is used to complete the positioning punching operation of all the stations to be punched in the set order. After completing the positioning punching operation of all the stations, the punching robot is operated to the zero point according to the zero trajectory, and the arrival of the next detected object is waited. In the full-station automatic operation flow, the left and right sides of the top, front and rear of the vehicle body are sequentially executed in the single-station real-time positioning flow, a total of 6 stations. The left and right sides of the vehicle top first execute the single-station real-time positioning flow, and after completing the single-station real-time positioning flow of the two sides of the vehicle top, the three angles of the vehicle body relative to the template making vehicle body in the three-dimensional direction are obtained, among which the left and right deflection angles and the rotation angle in the camera plane can be used as the required rotation angle in the side single-station real-time positioning flow.

2. The method of claim 1, wherein, ​ 3. The method of claim 1, wherein The vehicle body is first transported to the flat plate of the assembly line, and then transported to the flat plate assembly line with a limiting device by a crane, so that the position deviation of the vehicle body transported to the flat plate is within ±5 cm, and the angle deviation is within ±5°. When the vehicle body runs to the visual detection position through the assembly line, the flat plate is stopped by the limiting device installed below, and the signal is acquired by the visual system, and then the user selects the operation. The visual system performs single-station operation or full-station operation according to the user's selection. Before the single-station and full-station real-time positioning operation, the preparation process of positioning and punching of each station must be performed, which includes distortion calibration process, N-point calibration process, magnification acquisition process and template making process. In the single-station real-time positioning process and the full-station real-time positioning process, the position of the laser sensor is adjusted to approach the position during template making, so as to obtain accurate laser sensor values. The preparation process of positioning and punching of each station uses the corresponding detection components of each station to obtain and process the image and laser sensor measurement values. In the single-station real-time positioning process and the full-station real-time positioning process, the laser sensor in the top detection component is first controlled to approach the position, so as to obtain accurate laser sensor values, and then the deviation angle of the vehicle body on the non-camera surface is calculated. Then, the deviation angle of the vehicle body on the camera surface is calculated by comparing the image features obtained by the camera with the image features during template making. The corresponding deviation angles are used for processing during distortion correction and image coordinate conversion of each station. The physical positions of the feature points are further calculated based on the image features and the three-dimensional deviation angle information of the vehicle body obtained during template making and real-time positioning. The position information and punching posture of the punching point are accurately calculated, and then precise positioning and punching are realized.

4. The method of claim 3, wherein, Before the full-station operation, the preparation process of positioning and punching of each station is performed, which includes distortion calibration process, N-point calibration process, magnification acquisition process and template making process.

Images