Vehicle assembly method and device, computer equipment and storage medium

Abstract

The invention relates to a vehicle assembly method and device, computer equipment, a storage medium and a computer program product. The method comprises the steps that under the condition that a to-be-assembled vehicle is located at a target station, a vehicle image shot by a camera of the target station for the to-be-assembled vehicle is obtained, the vehicle image is obtained at the target station, and in the vehicle image, multiple assembly points of the to-be-assembled vehicle in a downstream station of the target station are determined, on the basis of the conversion relation between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream station and the camera internal reference of the camera, the image coordinates of all the assembly points in the vehicle image are converted, and the robot coordinates of all the assembly points in the base coordinate system are determined. According to the technical scheme, under the condition that the to-be-assembled vehicle reaches the downstream station, the robots are controlled to assemble the to-be-assembled vehicle according to the coordinates of the robots, the images do not need to be obtained and recognized at the downstream station, the assembling time is shortened, and the assembling efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of industrial manufacturing technology, and in particular to a vehicle assembly method, device, computer equipment, storage medium and computer program product. Background Art

[0002] With the rapid development of new energy vehicles, vision-guided robot assembly has gradually become an alternative to manual assembly.

[0003] In traditional technology, at the assembly station, the vehicle to be assembled is identified by a visual camera to obtain the identification result, and the robot is controlled to perform the corresponding assembly action based on the identification result. However, this method still has the problems of long assembly time and low assembly efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a vehicle assembly method, device, computer equipment, computer-readable storage medium and computer program product that can improve assembly efficiency in response to the above technical problems.

[0005] In the first aspect, the present application provides a vehicle assembly method. The method comprises: when the vehicle to be assembled is in a target station, obtaining a vehicle image taken by a camera of the target station for the vehicle to be assembled; in the vehicle image, determining multiple assembly points of the vehicle to be assembled in a downstream station of the target station; based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream station, and the camera internal parameters of the camera, converting the image coordinates of each assembly point in the vehicle image, and determining the robot coordinates of each assembly point in the base coordinate system; when the vehicle to be assembled arrives at the downstream station, controlling the robot to assemble the vehicle to be assembled according to each robot coordinate.

[0006] In one of the embodiments, the method further includes: when the calibration plate is located at the target station, determining the calibration camera coordinates of the calibration plate in the camera coordinate system of the camera; when the calibration plate is located at the downstream station, obtaining the calibration base coordinates of the calibration plate in the base coordinate system of the robot at the downstream station; and determining the conversion relationship between the camera coordinate system and the base coordinate system based on the calibration camera coordinates and the calibration base coordinates.

[0007] In one of the embodiments, determining the transformation relationship between the camera coordinate system and the base coordinate system based on the calibrated camera coordinates and the calibrated base coordinates includes: constructing a transformation matrix from the camera coordinate system to the base coordinate system based on the calibrated camera coordinates and the calibrated base coordinates; and performing singular value decomposition on the transformation matrix to obtain the transformation relationship between the camera coordinate system and the base coordinate system.

[0008] In one of the embodiments, based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream workstation, and the camera intrinsic parameters of the camera, the image coordinates of each assembly point in the vehicle image are converted to determine the robot coordinates of each assembly point in the base coordinate system, including: based on the camera intrinsic parameters of the camera, coordinate conversion of the image coordinates of each assembly point in the vehicle image to obtain the camera coordinates of each assembly point in the camera coordinate system; based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream workstation, coordinate conversion of each camera coordinate to obtain the robot coordinates of each assembly point in the base coordinate system.

[0009] In one of the embodiments, controlling the robot to assemble the vehicle to be assembled based on each of the robot coordinates includes: obtaining an assembly difficulty score corresponding to each of the robot coordinates; selecting a target coordinate whose assembly difficulty score is greater than a preset score among the robot coordinates; and controlling the robot to assemble the vehicle to be assembled based on each of the robot coordinates except the target coordinates.

[0010] In one of the embodiments, the method further includes: when the target workstation completes the operation, controlling a conveying device to convey the vehicle to be assembled to a downstream workstation of the target workstation; the conveying device keeps the vehicle posture of the vehicle to be assembled unchanged during the conveying process.

[0011] In a second aspect, the present application also provides a vehicle assembly device. The device comprises:

[0012] An acquisition module is used to acquire a vehicle image taken by a camera of the target station for the vehicle to be assembled when the vehicle to be assembled is in the target station; an assembly point determination module is used to determine, in the vehicle image, a plurality of assembly points of the vehicle to be assembled in a downstream station of the target station; a coordinate determination module is used to transform the image coordinates of each assembly point in the vehicle image based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream station and the camera intrinsic parameters of the camera, and determine the robot coordinates of each assembly point in the base coordinate system; an assembly module is used to control the robot to assemble the vehicle to be assembled according to the robot coordinates when the vehicle to be assembled arrives at the downstream station.

[0013] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0014] When a vehicle to be assembled is at a target station, a vehicle image taken by a camera of the target station for the vehicle to be assembled is obtained; in the vehicle image, a plurality of assembly points of the vehicle to be assembled in a station downstream of the target station are determined; based on a conversion relationship between a camera coordinate system of the camera and a base coordinate system of a robot in the downstream station, and a camera internal parameter of the camera, the image coordinates of each assembly point in the vehicle image are converted to determine the robot coordinates of each assembly point in the base coordinate system; when the vehicle to be assembled arrives at the downstream station, the robot is controlled to assemble the vehicle to be assembled according to the robot coordinates.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0016] When a vehicle to be assembled is at a target station, a vehicle image taken by a camera of the target station for the vehicle to be assembled is obtained; in the vehicle image, a plurality of assembly points of the vehicle to be assembled in a station downstream of the target station are determined; based on a conversion relationship between a camera coordinate system of the camera and a base coordinate system of a robot in the downstream station, and a camera internal parameter of the camera, the image coordinates of each assembly point in the vehicle image are converted to determine the robot coordinates of each assembly point in the base coordinate system; when the vehicle to be assembled arrives at the downstream station, the robot is controlled to assemble the vehicle to be assembled according to the robot coordinates.

[0017] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0018] When a vehicle to be assembled is at a target station, a vehicle image taken by a camera of the target station for the vehicle to be assembled is obtained; in the vehicle image, a plurality of assembly points of the vehicle to be assembled in a station downstream of the target station are determined; based on a conversion relationship between a camera coordinate system of the camera and a base coordinate system of a robot in the downstream station, and a camera internal parameter of the camera, the image coordinates of each assembly point in the vehicle image are converted to determine the robot coordinates of each assembly point in the base coordinate system; when the vehicle to be assembled arrives at the downstream station, the robot is controlled to assemble the vehicle to be assembled according to the robot coordinates.

[0019] The vehicle assembly method, device, computer equipment, storage medium and computer program product described above, when the vehicle to be assembled is at the target station, obtain a vehicle image taken by the camera of the target station for the vehicle to be assembled, obtain the vehicle image at the target station, determine multiple assembly points of the vehicle to be assembled in the downstream station of the target station in the vehicle image, based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream station, and the camera internal parameters of the camera, convert the image coordinates of each assembly point in the vehicle image, determine the robot coordinates of each assembly point in the base coordinate system, adopt the above method, when the vehicle to be assembled arrives at the downstream station, control the robot to assemble the vehicle to be assembled according to the coordinates of each robot, and there is no need to obtain and identify the image at the downstream station, thereby reducing the assembly time and improving the assembly efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A diagram showing an application environment of a vehicle assembly method in one embodiment;

[0021] Figure 2 A schematic flow chart of a vehicle assembly method in one embodiment;

[0022] Figure 3 A schematic diagram of a calibration plate in an embodiment;

[0023] Figure 4 A schematic flow chart of a dual-station hand-eye calibration method and an assembly method in one embodiment;

[0024] Figure 5 A schematic diagram of double-station calibration in one embodiment;

[0025] Figure 6 is a structural block diagram of a vehicle assembly device in one embodiment;

[0026] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0028] The vehicle assembly method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the control terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0029] The control terminal 102 can obtain a vehicle image taken by a camera of the target station for the vehicle to be assembled when the vehicle to be assembled is in the target station. The control terminal 102 can determine multiple assembly points of the vehicle to be assembled in the downstream station of the target station in the vehicle image. The control terminal 102 can convert the image coordinates of each assembly point in the vehicle image based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream station, as well as the camera internal parameters of the camera, and determine the robot coordinates of each assembly point in the base coordinate system. When the vehicle to be assembled arrives at the downstream station, the control terminal 102 can control the robot to assemble the vehicle to be assembled according to the coordinates of each robot. The control terminal 102 can obtain the vehicle image taken by the camera of the target station for the vehicle to be assembled from the server 104.

[0030] The control terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The control terminal 102 may also be a programmable logic controller, a single-chip microcomputer, a robot terminal, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0031] In one embodiment, Figure 2 As shown, a vehicle assembly method is provided, which is applied to Figure 1 The control terminal 102 in FIG. 1 is used as an example to illustrate, including:

[0032] S202: When the vehicle to be assembled is at a target workstation, a vehicle image captured by a camera at the target workstation and directed toward the vehicle to be assembled is obtained.

[0033] The vehicle to be assembled may be a vehicle or a vehicle component on an automobile automated assembly line.

[0034] The target station can be a station on the automobile automated assembly line. The target station can be a welding station, a spraying station, a tightening station, a handling station, a testing station, and a component assembly station. Among them, the welding robot at the welding station is responsible for spot welding and repair welding of the car body to ensure the consistency and quality of the welding process. The spraying robot at the spraying station is used to paint the surface of the car. The collaborative robot at the tightening station can replace manual repetitive tightening work to ensure qualified assembly and avoid worker fatigue and assembly errors.

[0035] The camera may be a camera fixedly mounted on the target station. The camera may be a visual camera, for example, a 3D structured light camera, a laser radar scanner, or a pulsed time-of-flight camera.

[0036] The vehicle image may be an image captured by a camera of the vehicle to be assembled.

[0037] The vehicle image can be obtained by fusing multiple images. Specifically, when the vehicle to be assembled is at a target workstation, the control terminal can obtain multiple images taken by a camera at the target workstation for the vehicle to be assembled. The control terminal can fuse the multiple images to obtain the vehicle image.

[0038] The camera's orientation angle can be adjusted to a certain extent. In one embodiment, when the vehicle to be assembled is at the target workstation, the control terminal can change the camera's orientation angle at the target workstation multiple times to obtain multiple camera orientation angles. The control terminal can obtain images taken by the camera at different camera orientation angles for the vehicle to be assembled. The images can correspond one-to-one to the camera orientation angles.

[0039] S204: In the vehicle image, a plurality of assembly points of the vehicle to be assembled in a downstream station of the target station are determined.

[0040] The downstream station can be the next station or the next stations of the target station on the automobile automated production line. The downstream station can also be any one of a welding station, a spraying station, a tightening station, a handling station, a testing station, and a component assembly station, depending on the setting of the automobile automated production line, and is not limited here.

[0041] The assembly point can be the image point corresponding to the actual point on the vehicle to be assembled that contacts the robot tool in the vehicle image. The actual point can be an engine mounting point, a vehicle shell spraying point, a vehicle screw mounting point, etc.

[0042] There are many ways to determine the image points of the vehicle image. Specifically, the control terminal can extract feature points from the vehicle image to determine a plurality of assembly points of the vehicle to be assembled in a downstream station of the target station.

[0043] In one embodiment, the control terminal may extract feature points from the vehicle image to obtain a plurality of feature points. The control terminal may match each feature point with preset assembly point feature information, and determine the image point of the vehicle image based on the successfully matched feature points.

[0044] S206, based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream workstation, as well as the camera internal parameters of the camera, the image coordinates of each assembly point in the vehicle image are converted to determine the robot coordinates of each assembly point in the base coordinate system.

[0045] The camera coordinate system may be a three-dimensional rectangular coordinate system with the optical center of the camera as the origin, the Z axis along the optical axis, and the X axis and the Y axis parallel to the horizontal and vertical directions of the image plane, respectively.

[0046] The robot can be an automated device used to perform operations at the downstream workstation. Different robots are set up according to the type of operation at the downstream workstation. For example, if the downstream workstation is an inspection workstation, the robot can be an inspection robot. For another example, if the downstream workstation is a transport workstation, the robot can be a transport robot.

[0047] A base coordinate system can refer to a fixed reference coordinate system used to describe the position and posture of other coordinate systems. In image processing, a base coordinate system can be a world coordinate system or a specific reference coordinate system used to integrate image information captured from different viewpoints.

[0048] The transformation relationship can represent the coordinate transformation relationship from the camera coordinate system to the base coordinate system.

[0049] Camera intrinsic parameters can be parameters that describe the internal properties of the camera, mainly including focal length, principal point coordinates, and distortion coefficients. These parameters are crucial for converting 3D world coordinates to 2D image coordinates. Camera intrinsic parameters can convert image coordinates in the image coordinate system to camera coordinates in the camera coordinate system.

[0050] The image coordinates may represent the coordinates of the assembly point in the camera coordinate system. The robot coordinates may represent the coordinates of the assembly point corresponding to the base coordinate system. The robot coordinates may be obtained by transforming the image coordinates.

[0051] There are many ways to transform coordinates. Specifically, the control terminal can transform the image coordinates of each assembly point in the vehicle image based on the camera intrinsic parameters of the camera to obtain the camera coordinates of each assembly point in the camera coordinate system. The control terminal can transform the camera coordinates based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream station to obtain the robot coordinates of each assembly point in the base coordinate system.

[0052] Specifically, the control terminal can perform coordinate conversion on the image coordinates of each assembly point in the vehicle image based on the camera intrinsic parameters of the camera to obtain the camera coordinates of each assembly point in the camera coordinate system. In the case where there is no coordinate conflict between the camera coordinates, the control terminal can perform coordinate conversion on each camera coordinate based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream station to obtain the robot coordinates of each assembly point in the base coordinate system.

[0053] S208, when the vehicle to be assembled arrives at the downstream workstation, the robots are controlled according to the coordinates of each robot to assemble the vehicle to be assembled.

[0054] The robot coordinates may represent the coordinates of the points where the robot contacts the vehicle to be assembled in the base coordinate system.

[0055] Specifically, when the vehicle to be assembled arrives at the downstream workstation, the control terminal can control the robots to assemble the vehicle to be assembled according to the coordinates of each robot.

[0056] In one embodiment, the control terminal may obtain the assembly difficulty score corresponding to each robot coordinate. The control terminal may select the target coordinates whose assembly difficulty score is greater than the preset score from each robot coordinate. The control terminal may control the robot to assemble the vehicle to be assembled based on each robot coordinate except the target coordinate.

[0057] In one embodiment, the control terminal can control the robot to be at the target coordinates and send prompt information to the operator, so that the operator can manually assemble the assembly point at the target coordinates according to the prompt information.

[0058] In the above vehicle assembly method, when the vehicle to be assembled is at the target station, a vehicle image taken by a camera at the target station for the vehicle to be assembled is obtained, and the vehicle image is obtained at the target station. In the vehicle image, multiple assembly points of the vehicle to be assembled in the downstream station of the target station are determined, and based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream station and the camera internal parameters of the camera, the image coordinates of each assembly point in the vehicle image are converted to determine the robot coordinates of each assembly point in the base coordinate system. By adopting the above method, when the vehicle to be assembled arrives at the downstream station, the robot is controlled to assemble the vehicle to be assembled according to the coordinates of each robot, and there is no need to obtain and recognize the image at the downstream station, thereby reducing the assembly time and improving the assembly efficiency.

[0059] In one embodiment, the vehicle assembly method further includes: when the calibration plate is located at the target station, determining the calibration camera coordinates of the calibration plate in the camera coordinate system of the camera; when the calibration plate is located at the downstream station, obtaining the calibration base coordinates of the calibration plate in the base coordinate system of the robot at the downstream station; and determining the conversion relationship between the camera coordinate system and the base coordinate system based on the calibration camera coordinates and the calibration base coordinates.

[0060] The calibration plate can be a tool for determining the relative position and posture of the camera and the end of the robotic arm. The calibration plate can be a plane or three-dimensional object with a regular pattern. Common types include plane calibration plates and three-dimensional calibration plates.

[0061] A flat calibration plate can be composed of a plane with a checkerboard or circular pattern. Its pattern should have high contrast, regularity, and repeatability to facilitate camera detection and feature point extraction. A 3D calibration plate consists of objects with 3D geometric shapes, such as cubes or cylinders. A 3D calibration plate can provide more spatial information and help improve calibration accuracy. For example, in some studies, the use of 3D calibration blocks combined with template matching algorithms for hand-eye calibration can complete calibration more quickly and accurately.

[0062] There may be multiple calibration points on the calibration plate. For example, Figure 3 As shown in the schematic diagram of the calibration plate, the calibration point in the calibration plate can be the center of a circle.

[0063] The calibration camera coordinates may represent the coordinates of the calibration points on the calibration plate in the camera coordinate system.

[0064] The calibration base coordinates may represent the coordinates of the calibration points on the calibration plate in the base coordinate system.

[0065] Specifically, the control terminal may determine a conversion relationship between the camera coordinate system and the base coordinate system based on the calibrated camera coordinates and the calibrated base coordinates.

[0066] In one embodiment, the control terminal may construct a transformation matrix from the camera coordinate system to the base coordinate system based on the calibrated camera coordinates and the calibrated base coordinates. The control terminal may perform singular value decomposition on the transformation matrix to obtain a transformation relationship between the camera coordinate system and the base coordinate system.

[0067] In one embodiment, the control terminal can directly extract the conversion relationship between the camera coordinate system and the base coordinate system from the conversion matrix, wherein the conversion relationship can be a rotation matrix and a translation vector between the camera coordinate system and the base coordinate system.

[0068] In this embodiment, when the calibration plate is located at the target station, the calibration camera coordinates of the calibration plate in the camera coordinate system of the camera are determined; when the calibration plate is located at the downstream station, the calibration base coordinates of the calibration plate in the base coordinate system of the robot at the downstream station are obtained; based on the calibration camera coordinates and the calibration base coordinates, the conversion relationship between the camera coordinate system and the base coordinate system is determined, which helps to obtain the robot coordinates. After the vehicle to be assembled arrives at the downstream station, there is no need to acquire and identify the image, which reduces the assembly time and improves the assembly efficiency.

[0069] In one embodiment, based on the calibrated camera coordinates and the calibrated base coordinates, the transformation relationship between the camera coordinate system and the base coordinate system is determined, including: based on the calibrated camera coordinates and the calibrated base coordinates, constructing a transformation matrix from the camera coordinate system to the base coordinate system, performing singular value decomposition on the transformation matrix, and obtaining the transformation relationship between the camera coordinate system and the base coordinate system.

[0070] The transformation matrix can represent the external parameter matrix corresponding to the camera coordinate system converted to the base coordinate system. The transformation matrix can include a rotation matrix and a translation vector. The rotation matrix R can be composed of rotation matrices around the x-axis, y-axis, and z-axis, and the translation vector T can represent the position of the camera relative to the world coordinate system (base coordinate system).

[0071] Singular Value Decomposition (SVD) is an important matrix decomposition technique that is widely used in linear algebra and numerical computing. It decomposes a matrix into the product of three matrices, which contains important information of the original matrix, such as the rank and eigenvalue of the matrix.

[0072] Specifically, the control terminal may construct a transformation matrix A from the camera coordinate system to the base coordinate system based on the calibrated camera coordinates and the calibrated base coordinates. The control terminal may perform singular value decomposition on the transformation matrix to obtain a transformation relationship between the camera coordinate system and the base coordinate system.

[0073] It should be noted that any m*n matrix A can be expressed in the form of singular value decomposition ,in and are orthogonal matrices of order m*m and n*n respectively, It is a diagonal matrix, and the diagonal elements are the singular values ​​of the A matrix, arranged in descending order. Regardless of whether A is a singular matrix or not, SVD can be decomposed, and it is almost unique. Although using SVD is more time-consuming than LU decomposition, SVD decomposition is numerically stable. The singular values ​​reflect the intrinsic properties of the matrix. By retaining larger singular values, noise can be effectively filtered.

[0074] In one embodiment, the control terminal may perform singular value decomposition on the transformation matrix to obtain a diagonal matrix. The control terminal may retain the top 10% elements in the diagonal matrix. The control terminal may reintroduce the top 10% elements into the transformation matrix to obtain a new transformation matrix, that is, a transformation relationship between the camera coordinate system and the base coordinate system.

[0075] In this embodiment, based on the calibrated camera coordinates and the calibrated base coordinates, a transformation matrix from the camera coordinate system to the base coordinate system is constructed, and the transformation matrix is ​​subjected to singular value decomposition to obtain the transformation relationship between the camera coordinate system and the base coordinate system. By retaining larger singular values, noise can be effectively filtered, thereby improving the accuracy of calibration.

[0076] In one embodiment, based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream workstation, and the camera intrinsic parameters of the camera, the image coordinates of each assembly point in the vehicle image are converted to determine the robot coordinates of each assembly point in the base coordinate system, including: based on the camera intrinsic parameters of the camera, coordinate conversion is performed on the image coordinates of each assembly point in the vehicle image to obtain the camera coordinates of each assembly point in the camera coordinate system; based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream workstation, coordinate conversion is performed on each camera coordinate to obtain the robot coordinates of each assembly point in the base coordinate system.

[0077] The camera intrinsic parameter may be a camera intrinsic parameter matrix, which includes the focal lengths in the x-axis and y-axis directions, and the position of the principal point.

[0078] Specifically, the control terminal may perform coordinate transformation on the image coordinates of each assembly point in the vehicle image based on the camera intrinsic parameter matrix to obtain the camera coordinates of each assembly point in the camera coordinate system.

[0079] In one embodiment, the control terminal may determine the inverse matrix of the camera intrinsic parameter matrix. The control terminal performs coordinate transformation on the image coordinates of each assembly point in the vehicle image according to the inverse matrix to obtain the camera coordinates of each assembly point in the camera coordinate system.

[0080] The conversion relationship between the camera coordinate system based on the camera and the base coordinate system of the robot in the downstream station can be the external parameter matrix of the camera. Specifically, the control terminal can first construct the corresponding homogeneous coordinates of each camera coordinate. The control terminal can multiply the homogeneous coordinates corresponding to the camera coordinate by the external parameter matrix for each camera coordinate to obtain the robot coordinates corresponding to the camera coordinate, that is, the robot coordinates of the assembly points corresponding to the camera coordinates in the base coordinate system.

[0081] In this embodiment, based on the camera intrinsic parameters of the camera, the image coordinates of each assembly point in the vehicle image are transformed to obtain the camera coordinates of each assembly point in the camera coordinate system. Based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream workstation, the camera coordinates are transformed to obtain the robot coordinates of each assembly point in the base coordinate system. The image coordinates are transformed to obtain the robot coordinates. After the vehicle to be assembled arrives at the downstream workstation, there is no need to acquire and identify the image, which reduces the assembly time and improves the assembly efficiency.

[0082] In one embodiment, controlling a robot to assemble a vehicle to be assembled based on each robot coordinate includes: obtaining an assembly difficulty score corresponding to each robot coordinate, selecting a target coordinate having an assembly difficulty score greater than a preset score in each robot coordinate, and controlling the robot to assemble the vehicle to be assembled based on each robot coordinate except the target coordinate.

[0083] The assembly difficulty score may represent the difficulty score of the robot action corresponding to the robot coordinates. Different robot coordinates may have different assembly difficulty scores. For example, for each robot coordinate arranged in a matrix, the coordinates near the middle have a greater assembly difficulty score than the coordinates near the outside.

[0084] The preset score may be a preset value set for each assembly difficulty score.

[0085] The target coordinates may be coordinates of a robot having an assembly difficulty score greater than a preset score.

[0086] Specifically, the control terminal may control the robot to assemble the vehicle to be assembled based on the coordinates of each robot except the target coordinates.

[0087] The control terminal can also control the robot's tool to be near the target coordinates and send prompt information to the operator, so that the operator can manually assemble the assembly point of the target coordinates according to the prompt information.

[0088] In this embodiment, the assembly difficulty score corresponding to each robot coordinate is obtained, and among each robot coordinate, the target coordinate whose assembly difficulty score is greater than the preset score is screened, and based on each robot coordinate except the target coordinate, the robot is controlled to assemble the vehicle to be assembled. Manual and robot work in collaboration, which improves assembly efficiency while improving assembly precision and accuracy.

[0089] In one embodiment, the vehicle assembly method further includes: when the target station completes the operation, controlling the conveying device to convey the vehicle to be assembled to a downstream station of the target station, and the conveying device keeps the vehicle posture of the vehicle to be assembled unchanged during the conveying process.

[0090] The target station completing the operation may indicate that the operation of the target station itself is completed, and the target station completing the operation may include a camera of the target station photographing the vehicle to be assembled.

[0091] The conveying device may refer to a device for carrying and conveying the vehicle to be assembled. The conveying device may specifically be at least one of an automated guided vehicle (AGV), a roller bed, an elevator, a travel machine, a conveyor belt, and the like.

[0092] The vehicle attitude can be the position and orientation of the vehicle to be assembled in space, usually including attitude angles such as pitch angle, roll angle and yaw angle. These angles describe the rotation state of the vehicle relative to the reference coordinate system (such as the ground coordinate system). Specifically, the invariant vehicle attitude can mean that the vehicle to be assembled does not rotate but only translates in the base coordinate system.

[0093] Specifically, the control terminal can control the conveying device to convey the vehicle to be assembled to the downstream station of the target station. When the conveying device reaches the limit position, the control terminal can determine that the vehicle to be assembled has arrived at the downstream station.

[0094] It should be noted that in order to ensure the stability and accuracy of the calibration plate during the transmission process, strict requirements must be placed on the repositioning accuracy of the wire slide. Only when the repositioning accuracy of the wire slide is high enough can a reliable basis be provided for subsequent calibration. Generally, positioning pins are used for precise positioning.

[0095] In this embodiment, when the target workstation completes the operation, the conveying device is controlled to convey the vehicle to be assembled to the downstream workstation of the target workstation. The conveying device keeps the vehicle posture of the vehicle to be assembled unchanged during the conveying process, providing a basis for the subsequent calibration process, thereby improving the accuracy of the calibration.

[0096] In one embodiment, Figure 4 As shown, a dual-station hand-eye calibration method and an assembly method are provided, comprising:

[0097] The first part, the calibration process, includes:

[0098] S401, when the calibration plate is located at a target workstation, determining calibration camera coordinates of the calibration plate in a camera coordinate system of the camera.

[0099] For example, Figure 5 In the dual-station calibration schematic diagram shown, the target station can be station one, and the downstream station can be station two.

[0100] At workstation one, you can use a 3D camera to take a picture of the calibration plate, use a 3D circle center recognition algorithm or manually record the point cloud at the center of the circle to obtain the coordinates of the four non-collinear circle centers in the camera coordinate system.

[0101] The robot at station 2 first performs tool coordinate system calibration, which refers to the solution of the pose parameters of the robot's tool coordinate system (ToolFrame) relative to the end-effector frame. The tool coordinate system is used to define the position of the TCP (tool center point) and the posture of the tool. The end-effector frame is the coordinate system that is fixed to the last connecting rod of the robot. Specifically, the calibration needle is tightly assembled to the position close to the end of the robot and the flange to complete the production of the tool, and then the four-point method or three-point five-step method is used to calibrate the conversion relationship from the tool coordinate system where the needle tip is located to the end coordinate system. Then, when the calibration plate arrives at station 2, the robot uses the calibrated TCP (needle tip) to point to the four centers of the calibration plate in turn to obtain the XYZ value of the center of the circle in the robot's base coordinate system. The needle tip is as close to the center of the circle as possible to improve the accuracy of the pointing coordinates.

[0102] Given the camera coordinates of the four points on the calibration plate and their corresponding coordinates in the robot coordinate system, the equation system Ax=b is constructed, where the A matrix represents the transformation matrix from the camera to the robot base coordinates, and its solution is obtained by singular value decomposition (SVD).

[0103] The coordinates of the four centers of the circle in the camera coordinate system and the robot base coordinate system are combined into a matrix, and then the matrix is ​​decomposed by the SVD method to obtain the rotation matrix and translation vector between the camera coordinate system and the robot base coordinate system. Through the rotation matrix and translation vector, we can transform the points in the camera coordinate system to the robot base coordinate system to achieve hand-eye calibration.

[0104] S402, when the calibration plate is located at a downstream station, obtaining calibration base coordinates of the calibration plate in a base coordinate system of a robot at the downstream station.

[0105] S403: construct a transformation matrix from the camera coordinate system to the base coordinate system based on the calibrated camera coordinates and the calibrated base coordinates.

[0106] S404, performing singular value decomposition on the transformation matrix to obtain a transformation relationship between the camera coordinate system and the base coordinate system.

[0107] The second part, the assembly process, includes:

[0108] S405 , when the vehicle to be assembled is at the target workstation, obtaining a vehicle image captured by a camera at the target workstation with respect to the vehicle to be assembled.

[0109] S406, when the target workstation completes the operation, controlling the conveying device to convey the vehicle to be assembled to a downstream workstation of the target workstation.

[0110] The conveying device keeps the vehicle posture of the vehicle to be assembled unchanged during the conveying process.

[0111] S407: In the vehicle image, determine a plurality of assembly points of the vehicle to be assembled in a downstream station of the target station.

[0112] S408 , based on the camera intrinsic parameters of the camera, coordinate transformation is performed on the image coordinates of each assembly point in the vehicle image to obtain the camera coordinates of each assembly point in the camera coordinate system.

[0113] S409, based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream workstation, coordinate conversion is performed on each camera coordinate to obtain the robot coordinates of each assembly point in the base coordinate system.

[0114] S410, when the vehicle to be assembled arrives at the downstream workstation, obtaining the assembly difficulty score corresponding to each robot coordinate.

[0115] S411, among the robot coordinates, select the target coordinates whose assembly difficulty score is greater than the preset score.

[0116] S412, based on the coordinates of each robot except the target coordinates, controlling the robot to assemble the vehicle to be assembled.

[0117] In this embodiment, when the vehicle to be assembled is at the target station, a vehicle image taken by a camera at the target station for the vehicle to be assembled is obtained, and the vehicle image is obtained at the target station. In the vehicle image, multiple assembly points of the vehicle to be assembled in the downstream station of the target station are determined, and based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream station, and the camera internal parameters of the camera, the image coordinates of each assembly point in the vehicle image are converted, and the robot coordinates of each assembly point in the base coordinate system are determined. By adopting the above method, when the vehicle to be assembled arrives at the downstream station, the robot is controlled to assemble the vehicle to be assembled according to the coordinates of each robot, and there is no need to obtain and recognize the image at the downstream station, thereby reducing the assembly time and improving the assembly efficiency.

[0118] The above method can also bring the following beneficial effects:

[0119] (1) The problem that the two workstations are far apart and cannot be calibrated by traditional methods is solved. The high repositioning accuracy of the follow-up calibration plate breaks through the limitation of the distance between the calibration plate and the robot in the traditional hand-eye calibration method. No matter how far the two workstations are apart, as long as the repositioning accuracy of the linear slide is high enough, the calibration plate can be accurately transferred between the two workstations, thus making hand-eye calibration possible.

[0120] (2) Compared with laser trackers, it does not require high equipment costs and complicated operations, and has higher practicality and economy.

[0121] (3) The structured light camera is used to take photos and record the coordinates of the circle center, which can quickly and accurately obtain the coordinates of the circle center on the calibration plate. At workstation 2, the robot uses the TCP tip to point to the center of the circle to obtain the coordinates. By precisely controlling the robot's motion and using high-precision motion controllers and sensors, the accuracy of the coordinates of the four center points in the robot's base coordinate system is ensured.

[0122] (4) The calibration relationship between the robot base coordinate system and the camera coordinate system is calculated by singular value decomposition. Compared with the traditional eye-external calibration method, the requirement for posture diversity of the calibration plate is avoided. This usually requires the robot to carry the calibration plate to perform up to dozens of posture transformations and data collection, which has high requirements on the richness of posture transformations and lacks calibration robustness. The present invention does not require rich engineering experience in the traditional calibration process to achieve stable and accurate dual-position hand-eye calibration.

[0123] (5) The impact on production is relatively small. The dual-station hand-eye calibration method of the present invention can be carried out without affecting production. The line slide can transfer the calibration plate from station one to station two while the production line is running. By using hot melt adhesive to stick the calibration plate to the appropriate position of the target (car body), the calibration plate and the car body will not be displaced during the dual-station transfer process. At station one, the camera collects the image of the calibration plate. At station two, the robot uses TCP to point and record data. The tool (calibration needle) can be installed at the appropriate position at the end of the robot, which does not interfere with production and calibrates the TCP in advance. This will not affect the production progress, but can also achieve hand-eye calibration and improve production efficiency. This flexibility makes the method of the present invention have a wider application prospect in the field of automobile assembly, and can meet the needs of enterprises for efficient, accurate and low-cost hand-eye calibration methods.

[0124] (6) Through TCP pointing and SVD calculation, a dual-station hand-eye calibration solution is realized, which effectively overcomes the disadvantage that hand-eye calibration cannot be performed directly due to the distance limitation between the hand and the eye, and provides a stable and easy-to-implement dual-station calibration solution for the manufacturing production line.

[0125] In summary, the dual-station hand-eye calibration method has important practical significance and application value. It can provide an efficient, accurate and low-cost hand-eye calibration solution for automobile assembly companies, thereby improving their production efficiency and competitiveness.

[0126] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0127] Based on the same inventive concept, the embodiment of the present application also provides a vehicle assembly device for implementing the vehicle assembly method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more vehicle assembly device embodiments provided below can refer to the limitations on the vehicle assembly method above, and will not be repeated here.

[0128] In one embodiment, Figure 6As shown, a vehicle assembly device is provided, comprising: an acquisition module 601, an assembly point determination module 602, a coordinate determination module 603 and an assembly module 604, wherein:

[0129] An acquisition module 601 is used to acquire a vehicle image taken by a camera of the target workstation with respect to the vehicle to be assembled when the vehicle to be assembled is at the target workstation;

[0130] An assembly point determination module 602 is used to determine, in the vehicle image, a plurality of assembly points of the vehicle to be assembled in a downstream station of the target station;

[0131] A coordinate determination module 603 is used to transform the image coordinates of each assembly point in the vehicle image based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream station, and the camera intrinsic parameters of the camera, and determine the robot coordinates of each assembly point in the base coordinate system;

[0132] The assembly module 604 is used to control the robots to assemble the vehicles according to the coordinates of the robots when the vehicles to be assembled arrive at the downstream workstation.

[0133] In one of the embodiments, the vehicle assembly device also includes: a calibration module, which is used to determine the calibration camera coordinates of the calibration plate in the camera coordinate system of the camera when the calibration plate is located at the target station; when the calibration plate is located at the downstream station, obtain the calibration base coordinates of the calibration plate in the base coordinate system of the robot at the downstream station; and determine the conversion relationship between the camera coordinate system and the base coordinate system based on the calibration camera coordinates and the calibration base coordinates.

[0134] In one of the embodiments, the calibration module is further used to construct a transformation matrix from the camera coordinate system to the base coordinate system based on the calibrated camera coordinates and the calibrated base coordinates; and perform singular value decomposition on the transformation matrix to obtain a transformation relationship between the camera coordinate system and the base coordinate system.

[0135] In one of the embodiments, the coordinate determination module 603 is also used to perform coordinate transformation on the image coordinates of each assembly point in the vehicle image based on the camera intrinsic parameters of the camera, so as to obtain the camera coordinates of each assembly point in the camera coordinate system; based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream workstation, perform coordinate transformation on each camera coordinate, so as to obtain the robot coordinates of each assembly point in the base coordinate system.

[0136] In one embodiment, the assembly module 604 is further used to obtain the assembly difficulty score corresponding to each robot coordinate; among each robot coordinate, filter the target coordinate whose assembly difficulty score is greater than the preset score; based on each robot coordinate except the target coordinate, control the robot to assemble the vehicle to be assembled.

[0137] In one of the embodiments, the vehicle assembly device further includes: a conveying module, which is used to control the conveying device to convey the vehicle to be assembled to a downstream station of the target station when the operation at the target station is completed; the conveying device keeps the vehicle posture of the vehicle to be assembled unchanged during the conveying process.

[0138] Each module in the above vehicle assembly device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0139] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store vehicle image data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a vehicle assembly method is implemented.

[0140] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0141] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the above method steps are implemented when the processor executes the computer program.

[0142] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method steps are implemented.

[0143] In one embodiment, a computer program product is provided, comprising a computer program, which implements the above method steps when executed by a processor.

[0144] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0145] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A vehicle assembly method, characterized in that: The method comprises: When the vehicle to be assembled is at a target workstation, obtaining a vehicle image taken by a camera at the target workstation with respect to the vehicle to be assembled; In the vehicle image, determining a plurality of assembly points of the vehicle to be assembled in a downstream station of the target station; Based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream workstation, and the camera intrinsic parameters of the camera, the image coordinates of each assembly point in the vehicle image are converted to determine the robot coordinates of each assembly point in the base coordinate system; When the vehicle to be assembled arrives at the downstream workstation, the robots are controlled according to the robot coordinates to assemble the vehicle to be assembled.

2. The method according to claim 1, characterized in that The method further comprises: When the calibration plate is located at the target station, determining the calibration camera coordinates of the calibration plate in the camera coordinate system of the camera; When the calibration plate is located at the downstream station, obtaining calibration base coordinates of the calibration plate in a base coordinate system of the robot at the downstream station; Based on the calibrated camera coordinates and the calibrated base coordinates, a conversion relationship between the camera coordinate system and the base coordinate system is determined.

3. The method according to claim 2, characterized in that The determining, based on the calibrated camera coordinates and the calibrated base coordinates, a conversion relationship between the camera coordinate system and the base coordinate system comprises: Based on the calibrated camera coordinates and the calibrated base coordinates, construct a transformation matrix from the camera coordinate system to the base coordinate system; Performing singular value decomposition on the transformation matrix to obtain a transformation relationship between the camera coordinate system and the base coordinate system.

4. The method according to claim 1, characterized in that The method of converting the image coordinates of each assembly point in the vehicle image based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream workstation and the camera intrinsic parameter of the camera to determine the robot coordinates of each assembly point in the base coordinate system includes: Based on the camera intrinsic parameters of the camera, coordinate transformation is performed on the image coordinates of each of the assembly points in the vehicle image to obtain the camera coordinates of each of the assembly points in the camera coordinate system; Based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream workstation, coordinate transformation is performed on each of the camera coordinates to obtain the respective robot coordinates of each of the assembly points in the base coordinate system.

5. The method according to claim 1, characterized in that The controlling the robot to assemble the vehicle to be assembled based on the coordinates of each robot includes: Obtaining the assembly difficulty score corresponding to each of the robot coordinates; Among the robot coordinates, selecting the target coordinates whose assembly difficulty score is greater than a preset score; Based on the robot coordinates other than the target coordinates, the robot is controlled to assemble the vehicle to be assembled.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: When the target workstation completes the operation, controlling the conveying device to convey the vehicle to be assembled to a downstream workstation of the target workstation; The conveying device keeps the vehicle posture of the vehicle to be assembled unchanged during the conveying process.

7. A vehicle assembly device, characterized in that: The device comprises: An acquisition module, used for acquiring a vehicle image taken by a camera of the target workstation with respect to the vehicle to be assembled when the vehicle to be assembled is in the target workstation; An assembly point determination module, used to determine, in the vehicle image, a plurality of assembly points of the vehicle to be assembled in a downstream station of the target station; A coordinate determination module, configured to transform the image coordinates of each assembly point in the vehicle image based on the conversion relationship between the camera coordinate system of the camera and the base coordinate system of the robot in the downstream station, and the camera intrinsic parameters of the camera, and determine the robot coordinates of each assembly point in the base coordinate system; The assembly module is used to control the robot to assemble the vehicle to be assembled according to the coordinates of each robot when the vehicle to be assembled arrives at the downstream workstation.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images