Screw hole positioning method and device and screw locking robot

Abstract

The invention relates to a screw hole positioning method and device and a screw locking robot, the screw locking robot comprises a mechanical arm, a screw locking device, a laser emitting device and a camera, the screw locking device, the laser emitting device and the camera are installed on the mechanical arm, and the emitting direction of the laser emitting device is parallel to the axial direction of the screw locking device; the method comprises the following steps: acquiring a screw hole image based on a calibrated camera, wherein the screw hole image comprises a target screw hole and a light spot formed by projecting laser emitted by a laser emitting device on the surface of a mounting piece; based on the screw hole image and the internal reference of the camera, obtaining the position deviation value of the tail end of the screw locking device and the target screw hole in the camera coordinate system; and on the basis of the position deviation value, the moving direction and the moving distance of the mechanical arm are controlled till the position deviation value is smaller than a preset threshold value, and the problem that screw positioning is inaccurate in the photovoltaic module installation process is solved.

Description

Technical Field

[0001] The present application relates to the technical field of photovoltaic equipment installation, and in particular to a screw hole positioning method, device and screw locking robot. Background Art

[0002] During the construction of photovoltaic power stations, a large number of photovoltaic modules need to be fixed to the pre-designed positions with screws. Currently, photovoltaic installation mechanical arms or mobile photovoltaic power station construction robots are generally used to assist the installation of photovoltaic modules through visual positioning technology. However, during the installation process, due to certain errors in visual positioning, when the robot performs position control and screw locking according to the identified position, inaccurate screw positioning may occur, resulting in screw installation misalignment, screw thread slippage, and even damage to photovoltaic modules, which in turn affects the installation quality and efficiency of photovoltaic modules.

[0003] With regard to the problem of inaccurate screw positioning during the installation of photovoltaic modules in the related art, no effective solution has been proposed so far. Summary of the invention

[0004] In this embodiment, a screw locking control method, device and screw locking robot are provided to solve the problem of excessive or insufficient screw locking force during photovoltaic module installation in the related art.

[0005] In a first aspect, a screw hole positioning method is provided in this embodiment, and the method is applied to a screw locking robot, the screw locking robot comprising a mechanical arm and a screw locking device, a laser emitting device and a camera mounted on the mechanical arm, the emitting direction of the laser emitting device is parallel to the axial direction of the screw locking device, and the method comprises:

[0006] Based on the calibrated camera, a screw hole image is collected, where the screw hole image includes a target screw hole and a light spot formed by the laser emitted by the laser emitting device and projected on the surface of the mounting part;

[0007] Based on the screw hole image and the internal parameters of the camera, obtaining a position deviation value between the end of the screw locking device and the target screw hole in the camera coordinate system;

[0008] Based on the position deviation value, the moving direction and moving distance of the robot arm are controlled until the position deviation value is less than a preset threshold.

[0009] In some of the embodiments, obtaining the position deviation value between the end of the screw locking device and the target screw hole in the camera coordinate system based on the screw hole image and the internal parameters of the camera includes:

[0010] Based on the screw hole image and the internal parameters of the camera, obtaining a first coordinate of the target screw hole in the camera coordinate system and a second coordinate of the light spot in the camera coordinate system;

[0011] Based on the second coordinate and the pre-acquired distance and orientation between the light spot and the projection of the end of the screw locking device, acquire a third coordinate of the end of the screw locking device in the camera coordinate system;

[0012] The position deviation value is acquired based on a difference between the third coordinate and the first coordinate.

[0013] In some embodiments, acquiring the first coordinate of the target screw hole in the camera coordinate system and the second coordinate of the light spot in the camera coordinate system based on the screw hole image and the internal parameters of the camera includes:

[0014] Performing image recognition on the screw hole image to obtain a fourth coordinate of the target screw hole in the image coordinate system and a fifth coordinate of the light spot in the image coordinate system;

[0015] Based on the intrinsic parameters and extrinsic parameters of the camera, the fourth coordinate is converted into a first coordinate in the camera coordinate system, and the fifth coordinate is converted into a second coordinate in the camera coordinate system.

[0016] In some of the embodiments, a laser radar is also installed on the robotic arm, and before the camera after calibration collects the screw hole image, the method further includes:

[0017] Collecting a calibration image based on the camera, wherein the calibration image includes a positioning mark;

[0018] Collecting a point cloud corresponding to the calibration image based on the laser radar;

[0019] The extrinsic parameters of the camera are calibrated based on the calibration image and the point cloud.

[0020] In some embodiments, calibrating the extrinsic parameters of the camera based on the calibration image and the point cloud includes:

[0021] Establishing a first conversion equation between the laser radar coordinate system and the camera coordinate system based on the external parameter;

[0022] Establishing a second conversion equation between the camera coordinate system and the image coordinate system based on the intrinsic parameters of the camera;

[0023] Based on the first conversion equation and the second conversion equation, superimposing and displaying the calibration image and the point cloud;

[0024] The external parameter is adjusted and determined so that the positioning mark in the calibration image and the positioning mark in the point cloud coincide with each other.

[0025] In some embodiments, the displaying the calibration image and the point cloud in a superimposed manner based on the first conversion equation and the second conversion equation includes:

[0026] Based on the first conversion equation and the second conversion equation, converting the point cloud from the laser radar coordinate system to the image coordinate system;

[0027] Based on the coordinate range of the calibration image, determining whether each projection point in the point cloud is on the calibration image;

[0028] The projection points on the calibration image are marked and displayed.

[0029] In a second aspect, a screw hole positioning device is provided in this embodiment, and the device is applied to a screw locking robot, the screw locking robot comprises a mechanical arm and a screw locking device, a laser emitting device and a camera mounted on the mechanical arm, the emitting direction of the laser emitting device is parallel to the axial direction of the screw locking device, and the device comprises:

[0030] An acquisition module, used for acquiring a screw hole image based on the calibrated camera, wherein the screw hole image includes a target screw hole and a light spot formed by the laser emitted by the laser emitting device projected on the surface of the mounting part;

[0031] An acquisition module, used for acquiring a position deviation value between the end of the screw locking device and the target screw hole in a camera coordinate system based on the screw hole image and the internal parameters of the camera;

[0032] The moving module is used to control the moving direction and moving distance of the robot arm based on the position deviation value until the position deviation value is less than a preset threshold.

[0033] According to the third aspect, a screw locking robot is provided in this embodiment, which includes a robotic arm and a screw locking device, a laser emitting device and a camera installed on the robotic arm, and the screw hole positioning device described in the second aspect; the emitting direction of the laser emitting device is parallel to the axial direction of the screw locking device.

[0034] In some of the embodiments, a laser radar is also installed on the robotic arm.

[0035] In some of the embodiments, the screw locking robot further includes a mobile chassis, and the mobile chassis is used to control the robotic arm to move in a set direction and distance.

[0036] Compared with the related art, the screw hole positioning method provided in the present embodiment obtains the relative position relationship between the light spot emitted by the laser emitting device and the target screw hole in the image coordinate system by collecting the screw hole image based on the calibrated camera; obtains the position deviation value between the end of the screw locking device and the target screw hole in the camera coordinate system based on the screw hole image and the internal parameters of the camera, and converts the position relationship between the light spot and the target screw hole into the position deviation between the end of the screw locking device and the target screw hole; controls the moving direction and moving distance of the robot arm based on the position deviation value until the position deviation value is less than a preset threshold value, and aligns the end of the screw locking device with the target screw hole, thereby solving the problem of inaccurate screw positioning during the installation of photovoltaic modules and improving the installation quality and efficiency of photovoltaic modules.

[0037] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 It is a hardware structure block diagram of the screw locking robot in some embodiments of the present application;

[0040] Figure 2 is a flow chart of a screw hole positioning method according to some embodiments of the present application;

[0041] Figure 3 is a flow chart of obtaining a position deviation value between an end of a screw locking device and a target screw hole in some embodiments of the present application;

[0042] Figure 4 is a flow chart of obtaining the camera coordinate system coordinates of the target screw hole and the light spot in some embodiments of the present application;

[0043] Figure 5 is a flow chart of camera extrinsic calibration in some embodiments of the present application;

[0044] Figure 6 is a flow chart of performing external parameter calibration based on images and point clouds in some embodiments of the present application;

[0045] Figure 7 is a flow chart of image point cloud overlay display in some embodiments of the present application;

[0046] Figure 8 is a flow chart of a screw hole positioning method according to some preferred embodiments of the present application;

[0047] Fig. 9 This is a structural block diagram of a screw hole positioning device in some embodiments of the present application. DETAILED DESCRIPTION

[0048] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0050] The screw hole positioning method provided in the embodiment of the present application can be executed on a screw locking robot or other movable mechanical arm. Figure 1 1 is a hardware structure diagram of a screw locking robot in some embodiments of the present application. Figure 1 As shown, the screw locking robot includes a mechanical arm 11 and a screw locking device 12, a laser emitting device 13 and a camera 14 mounted on the mechanical arm 11. The emitting direction of the laser emitting device 13 is parallel to the axial direction of the screw locking device 12. Furthermore, the screw locking robot may also include a laser radar 15.

[0051] Specifically, the laser emitting device 13 may include a laser and an optical system. The laser is used to generate a high-brightness laser beam, which forms a clear laser spot through the optical system; the optical system may include a laser emitting lens and an adjustment device for adjusting the size and shape of the laser spot so that it can form a clear mark on the surface of the photovoltaic module.

[0052] Specifically, the camera 14 may be a high-resolution camera for capturing images of the surface of the photovoltaic module, especially the position and shape of the laser spot and the screw hole. The camera lens may have high resolution and high contrast to ensure image clarity and accurately identify the details of the laser spot.

[0053] Specifically, the robot arm 11 may be a six-degree-of-freedom robot arm, and the position and angle of the screw locking device 12 may be adjusted. The screw locking device 12 is used to control the alignment of the screw hole and the installation of the screw. The screw locking device 12 may include an electric screwdriver. The laser emitting device 13 may be fixed by an integrated structural member so that the laser emitting direction is parallel to the axial direction of the screw locking device, that is, parallel to the axial direction of the electric screwdriver.

[0054] It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the screw locking robot. Figure 1 There may be more components shown in the figure, or different configurations, which are not limited in this embodiment.

[0055] In this embodiment, a screw hole positioning method is provided. Figure 2 is a flow chart of a screw hole positioning method in some embodiments of the present application, such as Figure 2 As shown, the process includes the following steps:

[0056] Step S201, capturing a screw hole image based on a calibrated camera, the screw hole image including a target screw hole and a light spot formed by a laser emitted by a laser emitting device projected onto a surface of a mounting part.

[0057] The camera calibration in this embodiment refers to the calibration of the camera's external parameters, that is, determining the position and posture of the camera relative to the reference coordinate system, including the translation vector T and the rotation matrix R. The camera of this embodiment is installed on the robotic arm, and the camera's external parameters can be determined by the relative position and posture of the camera on the robotic arm. In some embodiments, the reference coordinate system can be the body coordinate system of the screw locking robot. In other embodiments, a laser radar is also installed on the robotic arm, and the reference coordinate system can be a laser radar coordinate system.

[0058] After the external parameter calibration, the position and posture of the camera relative to the reference coordinate system no longer change, so the subsequent screw hole positioning operation is performed with the external parameter. Therefore, based on the calibrated external parameter matrix, the coordinate transformation of any coordinate point in the reference coordinate system can be performed to obtain the corresponding coordinate of the coordinate point in the camera coordinate system, eliminating the coordinate data deviation between the camera coordinate system and the reference coordinate system. The camera coordinate system can be a three-dimensional space coordinate system used to describe the spatial dimensions of the real world.

[0059] After the camera is calibrated, the laser emitting device on the robotic arm is started, and the laser emitting device projects a laser spot on the surface of the photovoltaic module to be installed to form a clear mark. It should be noted that at this time, the emission direction of the laser emitting device has been adjusted based on the pre-acquired locking plane normal vector, so that the emission direction is perpendicular to the surface of the photovoltaic module, and the axial direction of the screw locking device is parallel to the emission direction. There are many different ways to obtain the locking plane normal vector, including through laser point cloud recognition, image feature extraction and recognition, etc., which will not be repeated in this embodiment.

[0060] Since the axial direction of the screw locking device is parallel to the laser emission direction and perpendicular to the surface of the photovoltaic module, the distance and orientation between the projection of the end of the screw locking device on the surface of the photovoltaic module and the laser spot are constant and can be acquired in advance. The screw hole image is collected by the calibrated camera, and the screw hole image includes the target screw hole and the laser projected spot.

[0061] Step S202: Based on the screw hole image and the internal parameters of the camera, a position deviation value between the end of the screw locking device and the target screw hole in the camera coordinate system is obtained.

[0062] By performing image recognition on the screw hole image, the position coordinates of the target screw hole and the light spot in the image coordinate system can be obtained. This embodiment does not limit the image recognition algorithm. The image coordinate system is a two-dimensional plane coordinate system of the screw hole image. Based on the internal parameters of the camera, a coordinate conversion equation between the image coordinate system and the camera coordinate system can be established. According to the coordinate conversion equation and the position coordinates of the target screw hole and the light spot in the image coordinate system, the position coordinates of the target screw hole and the light spot in the camera coordinate system can be calculated. Combined with the distance and orientation between the end projection of the screw locking device and the light spot in step S201, the position deviation value of the end of the screw locking device and the target screw hole in the camera coordinate system can be obtained.

[0063] Step S203: Based on the position deviation value, control the moving direction and moving distance of the robot arm until the position deviation value is less than a preset threshold.

[0064] According to the position deviation value between the end of the screw locking device and the target screw hole, the robot arm is controlled to move in the direction and distance corresponding to the position deviation value, and the camera posture is not changed during the movement. After the movement action is completed, the position deviation value between the end of the screw locking device and the target screw hole is confirmed again. If it is less than the preset threshold, it is determined that the screw hole positioning is completed.

[0065] Through steps S201 to S203, the screw hole image is collected based on the calibrated camera to obtain the relative position relationship between the light spot emitted by the laser emitting device and the target screw hole in the image coordinate system; based on the screw hole image and the internal parameters of the camera, the position deviation value between the end of the screw locking device and the target screw hole in the camera coordinate system is obtained, and the position relationship between the light spot and the target screw hole is converted into the position deviation between the end of the screw locking device and the target screw hole; based on the position deviation value, the moving direction and moving distance of the robot arm are controlled until the position deviation value is less than a preset threshold value, and the end of the screw locking device is aligned with the target screw hole, thereby solving the problem of inaccurate screw positioning during the installation of photovoltaic modules and improving the installation quality and efficiency of photovoltaic modules.

[0066] In some embodiments, Figure 3 Flowchart for obtaining the position deviation value between the end of the screw locking device and the target screw hole in some embodiments of the present application, such as Figure 3 As shown, the process includes the following steps:

[0067] Step S301, based on the screw hole image and the internal parameters of the camera, obtain the first coordinates of the target screw hole in the camera coordinate system and the second coordinates of the light spot in the camera coordinate system.

[0068] Based on the camera's internal parameters, a coordinate conversion equation between the image coordinate system and the camera coordinate system can be established. According to the coordinate conversion equation and the position coordinates of the target screw hole and the light spot in the image coordinate system, the first coordinate of the target screw hole in the camera coordinate system and the second coordinate of the light spot in the camera coordinate system can be calculated.

[0069] Step S302: Acquire a third coordinate of the end of the screw locking device in the camera coordinate system based on the second coordinate and the pre-acquired distance and orientation between the light spot and the projection of the end of the screw locking device.

[0070] Since the axial direction of the screw locking device is parallel to the laser emission direction and perpendicular to the surface of the photovoltaic module, the positions of the screw locking device and the laser emitting device are fixed relative to the camera coordinate system after calibration. Therefore, in the camera coordinate system, the distance and orientation between the projection of the end of the screw locking device on the surface of the photovoltaic module and the laser spot are constant and can be obtained in advance.

[0071] According to the second coordinate of the light spot in the camera coordinate system, and the distance and orientation between the end of the screw locking device and the light spot, the third coordinate of the end of the screw locking device in the camera coordinate system is obtained. Assuming that the second coordinate of the light spot in the camera coordinate system is (x2, y2, z2), and the distance between the end of the screw locking device and the light spot is (m, -n), the third coordinate of the end of the screw locking device in the camera coordinate system is (x2+m, y2-n, z2).

[0072] Step S303: acquiring a position deviation value based on a difference between the third coordinate and the first coordinate.

[0073] Through steps S301 to S303, the first coordinate of the target screw hole in the camera coordinate system and the second coordinate of the light spot in the camera coordinate system are obtained based on the screw hole image and the internal parameters of the camera, so as to obtain the accurate positions of the target screw hole and the light spot; the third coordinate of the end of the screw locking device in the camera coordinate system is obtained based on the second coordinate and the distance and orientation between the light spot and the end projection of the screw locking device obtained in advance, and the accurate position of the end projection of the screw locking device is obtained through light spot positioning; the position deviation value is obtained based on the difference between the third coordinate and the first coordinate, so as to obtain the position deviation between the end projection of the screw locking device and the target screw hole, so as to obtain accurate data for the movement and alignment of the robot arm.

[0074] In some embodiments, Figure 4 is a flow chart of obtaining the camera coordinate system coordinates of the target screw hole and the light spot in some embodiments of the present application, such as Figure 4 As shown, the process includes the following steps:

[0075] Step S401, performing image recognition on the screw hole image to obtain the fourth coordinate of the target screw hole in the image coordinate system and the fifth coordinate of the light spot in the image coordinate system.

[0076] Through image recognition, the fourth coordinate of the target screw hole in the image coordinate system and the fifth coordinate of the light spot in the image coordinate system are obtained. This embodiment does not limit the image recognition algorithm.

[0077] Step S402: based on the intrinsic parameters of the camera, convert the fourth coordinate into a first coordinate in the camera coordinate system, and convert the fifth coordinate into a second coordinate in the camera coordinate system.

[0078] Assuming that the camera's intrinsic parameter matrix is ​​K and the fourth coordinate is (u, v), the fourth coordinate (u, v) can be converted to the normalized coordinate (x_c, y_c, z_c) in the camera coordinate system according to the intrinsic parameter matrix K. The calculation formula is as follows:

[0079]

[0080] The calculation process of the fifth coordinate is the same as that in the above embodiment.

[0081] Through steps S401 to S402, image recognition is performed on the screw hole image to obtain the fourth coordinate of the target screw hole in the image coordinate system and the fifth coordinate of the light spot in the image coordinate system, and based on the internal parameters of the camera, the fourth coordinate is converted into the first coordinate in the camera coordinate system, and the fifth coordinate is converted into the second coordinate in the camera coordinate system to obtain the accurate positions of the target screw hole and the light spot in the camera coordinate system, so as to obtain accurate coordinate data for subsequent target screw hole alignment.

[0082] In some embodiments, a laser radar is also installed on the robotic arm. Figure 5 is a flow chart of camera extrinsic calibration in some embodiments of the present application, such as Figure 5 As shown, the process includes the following steps:

[0083] Step S501: Capture a calibration image based on a camera, where the calibration image includes a positioning mark.

[0084] LiDAR measures the distance from the target to the LiDAR receiver by emitting a light beam to obtain accurate three-dimensional point cloud data. The collected point cloud data has high precision, good anti-interference, is not affected by changes in illumination, and has accurate depth information. The fusion of LiDAR and camera data can achieve information complementarity between the two sensors and obtain richer three-dimensional environmental information. External parameter calibration can obtain the transformation matrix between the LiDAR coordinate system and the camera coordinate system, and realize the mutual conversion between the LiDAR coordinate system and the camera coordinate system data.

[0085] In this embodiment, the laser radar and the camera are calibrated by the joint external parameters through the laser radar and the camera fusion perception. Before calibration, the camera first collects a calibration image with a positioning mark. The positioning mark can be a QR code or other mark with image features.

[0086] Step S502, collecting a point cloud corresponding to the calibration image based on the laser radar.

[0087] The point cloud corresponding to the calibration image is collected by laser radar, and the point cloud also includes a positioning mark. Further, the point cloud can be pre-processed by filtering.

[0088] Step S503: calibrate the extrinsic parameters of the camera based on the calibration image and the point cloud.

[0089] The camera's external parameters are calibrated based on the positioning marks in the calibration image and the positioning marks in the point cloud. Specifically, the laser radar's point cloud data and the camera's calibration image can be mapped to the same coordinate system by observation matching, and the point cloud data and the calibration image can be matched in the coordinate system. The camera's external parameters are obtained based on the matching results.

[0090] Through steps S501 to S503, by collecting a calibration image based on the camera, the calibration image includes a positioning mark, and based on the point cloud corresponding to the calibration image collected by the lidar, the position data of the same positioning mark in the lidar coordinate system and the camera coordinate system are respectively obtained; by calibrating the external parameters of the camera based on the calibration image and the point cloud, the transformation parameter matrix between the lidar coordinate system and the camera coordinate system is obtained, the data fusion between the lidar coordinate system and the camera coordinate system is realized, and the accuracy of environmental perception is improved.

[0091] In some embodiments, Figure 6 is a flowchart of performing external parameter calibration based on images and point clouds in some embodiments of the present application, such as Figure 6 As shown, the process includes the following steps:

[0092] Step S601, establishing a first conversion equation between the laser radar coordinate system and the camera coordinate system based on external parameters.

[0093] Assume that the spatial position of the point cloud in the coordinate system defined by the lidar is (x, y, z) T , the spatial position of the point cloud in the camera coordinate system is (x c ,y c , z c ) T ; The camera extrinsic parameters are [R|t], where R is the rotation matrix and t is the translation vector.

[0094] The first transformation equation for transforming the point cloud from the lidar coordinate system to the camera coordinate system is:

[0095]

[0096] Step S602: establishing a second conversion equation between the camera coordinate system and the image coordinate system based on the intrinsic parameters of the camera.

[0097] Assume K is the camera internal parameter, K∈R a×a , transform the point cloud from the camera coordinate system to the normalized plane coordinate system and project it onto the image plane, (u, v, 1) T It is the projection point of the point cloud onto the image.

[0098] The second transformation equation that transforms the point cloud from the camera coordinate system to the image coordinate system is:

[0099]

[0100] Step S603: Based on the first conversion equation and the second conversion equation, the calibration image and the point cloud are superimposed and displayed.

[0101] The point cloud is transformed from the laser radar coordinate system to the image coordinate system through the first conversion equation and the second conversion equation, and then the point cloud and the calibration image are superimposed and displayed in the image coordinate system. At this time, the external parameters of the camera can be the default values ​​before calibration.

[0102] Step S604, adjusting and determining the external parameters so that the positioning mark in the calibration image and the positioning mark in the point cloud coincide with each other.

[0103] By adjusting the camera's extrinsic matrix R, t, the positioning markers in the point cloud image coincide with the positioning markers in the camera image. The extrinsic matrix R, t at this time is the calibrated camera extrinsic parameter. Specifically, coincidence refers to the coincidence of multiple feature points of the positioning marker in the point cloud image with the corresponding multiple feature points in the camera image.

[0104] Through steps S601 to S604, the point cloud data is converted from the lidar coordinate system to the image coordinate system by establishing a first conversion equation between the lidar coordinate system and the camera coordinate system based on external parameters and establishing a second conversion equation between the camera coordinate system and the image coordinate system based on the internal parameters of the camera; based on the first conversion equation and the second conversion equation, the calibration image and the point cloud are superimposed and displayed, and the external parameters are adjusted so that the positioning mark in the calibration image and the positioning mark in the point cloud coincide with each other, and the external parameter calibration is performed when the positioning marks coincide with each other, and a data transformation relationship between the lidar coordinate system and the camera coordinate system is established, thereby improving the accuracy of screw hole positioning by the camera.

[0105] In some embodiments, Figure 7 is a flowchart of image point cloud overlay display in some embodiments of the present application, such as Figure 7 As shown, the process includes the following steps:

[0106] Step S701, based on the first conversion equation and the second conversion equation, convert the point cloud from the laser radar coordinate system to the image coordinate system.

[0107] According to the first conversion equation and the second conversion equation of steps S601 to S602 in the above embodiment, the position coordinates of each point in the point cloud are converted from the lidar coordinate system to the image coordinate system to obtain the position coordinates of each point projected into the image coordinate system.

[0108] Step S702: based on the coordinate range of the calibration image, determine whether each projection point in the point cloud is on the calibration image.

[0109] The size of the calibration image is usually expressed in the coordinate range of the pixel points. According to the maximum horizontal and vertical coordinates of the pixel points, it can be determined whether the position coordinates of each point in the point cloud are within the range of the calibration image after being projected into the image coordinate system. If it is not within the range of the calibration image, the point will not be displayed.

[0110] Step S703: Mark and display the projection points on the calibration image.

[0111] If the projection point is within the range of the calibration image, the projection point is marked on the calibration image.

[0112] Through steps S701 to S703, the point cloud is converted from the lidar coordinate system to the image coordinate system based on the first conversion equation and the second conversion equation, and based on the coordinate range of the calibration image, it is determined whether each projection point in the point cloud is on the calibration image, the projection points on the calibration image are marked and displayed, and the position coordinates of each point in the point cloud are filtered based on the coordinate range of the calibration image, and only the projection points within the range of the calibration image are displayed, thereby reducing the range of the superimposed display image and reducing the difficulty of determining the overlap of positioning marks.

[0113] The preferred embodiment of the present application is described and illustrated below. The preferred embodiment of the present application is applied to a screw locking robot, which includes a mechanical arm and a screw locking device, a laser emitting device, a laser radar and a camera installed on the mechanical arm. The emitting direction of the laser emitting device is parallel to the axial direction of the screw locking device. Figure 8 is a flowchart of screw hole positioning in some preferred embodiments of the present application, such as Figure 8 As shown, the process includes the following steps:

[0114] Step S801, the laser emitting device projects a laser spot on the surface of the photovoltaic module to form a clear mark, the edge of the spot is clear, and the distance d between the projection of the end of the screw locking device and the spot remains constant at different heights;

[0115] Step S802: The camera collects a calibration image, where the calibration image includes a positioning mark;

[0116] Step S803, the laser radar collects point cloud data corresponding to the calibration image;

[0117] Step S804, establishing a first conversion equation between the laser radar coordinate system and the camera coordinate system based on the camera extrinsic parameters;

[0118] Step S805, establishing a second conversion equation between the camera coordinate system and the image coordinate system based on the intrinsic parameters of the camera;

[0119] Step S806, based on the first conversion equation and the second conversion equation, convert the point cloud data from the laser radar coordinate system to the image coordinate system;

[0120] Step S807, based on the coordinate range of the calibration image, determining whether each projection point in the point cloud data is on the calibration image;

[0121] Step S808, marking and displaying the projection points on the calibration image;

[0122] Step S809, adjusting the camera external parameters so that the positioning mark in the calibration image and the positioning mark in the point cloud coincide with each other, and determining the external parameters when they coincide with each other as the external parameters after camera calibration;

[0123] Step S810, collecting a screw hole image based on the calibrated camera, where the screw hole image includes the target screw hole and a light spot formed by the laser emitted by the laser emitting device projected on the surface of the mounting part;

[0124] Step S811, performing image recognition on the screw hole image to obtain the coordinates of the target screw hole and the light spot in the image coordinate system;

[0125] Step S812, based on the intrinsic parameters of the camera, convert the coordinates of the target screw hole and the light spot in the image coordinate system into coordinates in the camera coordinate system;

[0126] Step S813, obtaining the coordinates of the end of the screw locking device in the camera coordinate system according to the coordinates of the target screw hole and the light spot in the camera coordinate system, and the distance and orientation between the light spot and the projection of the end of the screw locking device;

[0127] Step S814, obtaining a position deviation value between the end of the screw locking device and the target screw hole according to the coordinates of the two in the camera coordinate system;

[0128] Step S815, based on the position deviation value, controlling the moving direction and moving distance of the robot arm until the position deviation value is less than a preset threshold.

[0129] Through steps S801 to S815, the coordinate conversion equation between the laser radar coordinate system and the image coordinate system is established through the external parameters and internal parameters of the camera, and the point cloud data containing the positioning mark is collected by the laser radar, and the point cloud data is converted into the image coordinate system for superposition and display with the calibration image collected by the camera, and the external parameters are adjusted to make the positioning mark in the calibration image coincide with the positioning mark in the point cloud. When the positioning marks coincide, the external parameter calibration is performed, and the data transformation relationship between the laser radar coordinate system and the camera coordinate system is established, thereby improving the accuracy of screw hole positioning by the camera; by collecting the screw hole image based on the calibrated camera, the relative position relationship between the light spot emitted by the laser emitting device and the target screw hole in the image coordinate system is obtained; by converting the position relationship between the light spot and the target screw hole into the position deviation between the end of the screw locking device and the target screw hole, the moving direction and moving distance of the mechanical arm are controlled according to the position deviation, and the end of the screw locking device is aligned with the target screw hole, thereby solving the problem of inaccurate screw positioning during the installation of photovoltaic modules and improving the installation quality and efficiency of photovoltaic modules.

[0130] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0131] In some embodiments, the present application also provides a screw hole positioning device, which is applied to a screw locking robot. The screw locking robot includes a robotic arm and a screw locking device, a laser emitting device and a camera installed on the robotic arm. The emission direction of the laser emitting device is parallel to the axial direction of the screw locking device. The screw hole positioning device is used to implement the above-mentioned embodiments and preferred implementation modes, and those that have been explained will not be repeated here. The terms "module", "unit", "sub-unit", etc. used below can be a combination of software and / or hardware that implements predetermined functions. In some embodiments, Fig. 9 : is a structural block diagram of the screw hole positioning device of this embodiment, such as Fig. 9 As shown, the device comprises:

[0132] An acquisition module 91 is used to acquire a screw hole image based on a calibrated camera, wherein the screw hole image includes a target screw hole and a light spot formed by a laser emitted by a laser emitting device and projected onto a surface of a mounting part;

[0133] An acquisition module 92 is used to acquire a position deviation value between the end of the screw locking device and the target screw hole in the camera coordinate system based on the screw hole image and the internal parameters of the camera;

[0134] The moving module 93 is used to control the moving direction and moving distance of the robot arm based on the position deviation value until the position deviation value is less than a preset threshold.

[0135] The screw hole positioning device of this embodiment acquires the screw hole image based on the calibrated camera through the acquisition module 91, and obtains the relative position relationship between the light spot emitted by the laser emitting device and the target screw hole in the image coordinate system; acquires the position deviation value between the end of the screw locking device and the target screw hole in the camera coordinate system based on the screw hole image and the internal parameters of the camera through the acquisition module 92, and converts the position relationship between the light spot and the target screw hole into the position deviation between the end of the screw locking device and the target screw hole; controls the moving direction and moving distance of the robot arm based on the position deviation value through the moving module 93 until the position deviation value is less than the preset threshold value, and aligns the end of the screw locking device with the target screw hole, thereby solving the problem of inaccurate screw positioning during the installation of photovoltaic modules and improving the installation quality and efficiency of photovoltaic modules.

[0136] In some embodiments, the present application also provides a screw locking robot, which includes a robotic arm and a screw locking device, a laser emitting device and a camera installed on the robotic arm, and the screw hole positioning device in the above embodiments; the emitting direction of the laser emitting device is parallel to the axial direction of the screw locking device.

[0137] The screw locking robot of this embodiment projects a laser spot on the surface of the photovoltaic module to be installed through a laser emitting device, and the distance and orientation between the laser spot and the projection of the end of the screw locking device are constant, which can be used as a positioning mark of the screw locking device in the screw hole image captured by the camera; the screw hole image is captured by the camera to obtain the positional relationship between the screw hole and the light spot in the image coordinate system, and the positional relationship is converted into the camera coordinate system to obtain the positional relationship between the screw hole and the end of the screw locking device; the mechanical arm is moved according to the positional relationship to reduce the distance between the screw hole and the screw locking device, thereby realizing the positioning of the screw hole, solving the problem of inaccurate screw positioning during the installation of photovoltaic modules and improving the installation quality and efficiency of photovoltaic modules.

[0138] In some embodiments, a laser radar is also installed on the robotic arm. The laser radar is used to perform joint external parameter calibration of the laser radar and the camera by fusion perception with the camera, obtain the transformation matrix between the laser radar coordinate system and the camera coordinate system, realize the mutual conversion between the laser radar coordinate system and the camera coordinate system data, improve the accuracy of environmental perception, and improve the accuracy of screw hole positioning through the camera.

[0139] In some embodiments, the screw locking robot also includes a mobile chassis, which is used to control the robot arm to move in a set direction and distance to reduce the distance between the screw hole and the screw locking device to achieve alignment and positioning of the screw hole.

[0140] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0141] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0142] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0143] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0144] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A screw hole positioning method, characterized in that: The method is applied to a screw locking robot, which includes a mechanical arm and a screw locking device, a laser emitting device and a camera installed on the mechanical arm, wherein the emitting direction of the laser emitting device is parallel to the axial direction of the screw locking device, and the method includes: Based on the calibrated camera, a screw hole image is collected, where the screw hole image includes a target screw hole and a light spot formed by the laser emitted by the laser emitting device and projected on the surface of the mounting part; Based on the screw hole image and the internal parameters of the camera, obtaining a position deviation value between the end of the screw locking device and the target screw hole in the camera coordinate system; Based on the position deviation value, the moving direction and moving distance of the robot arm are controlled until the position deviation value is less than a preset threshold.

2. The method according to claim 1, characterized in that The step of obtaining the position deviation value between the end of the screw locking device and the target screw hole in the camera coordinate system based on the screw hole image and the internal parameter of the camera includes: Based on the screw hole image and the internal parameters of the camera, obtaining a first coordinate of the target screw hole in the camera coordinate system and a second coordinate of the light spot in the camera coordinate system; Based on the second coordinate and the pre-acquired distance and orientation between the light spot and the projection of the end of the screw locking device, acquiring a third coordinate of the end of the screw locking device in the camera coordinate system; The position deviation value is acquired based on a difference between the third coordinate and the first coordinate.

3. The method according to claim 2, characterized in that The acquiring, based on the screw hole image and the internal parameters of the camera, the first coordinate of the target screw hole in the camera coordinate system and the second coordinate of the light spot in the camera coordinate system comprises: Performing image recognition on the screw hole image to obtain a fourth coordinate of the target screw hole in the image coordinate system and a fifth coordinate of the light spot in the image coordinate system; Based on the intrinsic parameters of the camera, the fourth coordinate is converted into a first coordinate in the camera coordinate system, and the fifth coordinate is converted into a second coordinate in the camera coordinate system.

4. The method according to claim 1, characterized in that: The mechanical arm is also equipped with a laser radar. Before collecting the screw hole image based on the calibrated camera, the method further includes: Collecting a calibration image based on the camera, wherein the calibration image includes a positioning mark; Collecting a point cloud corresponding to the calibration image based on the laser radar; The extrinsic parameters of the camera are calibrated based on the calibration image and the point cloud.

5. The method according to claim 4, characterized in that The calibrating the external parameters of the camera based on the calibration image and the point cloud includes: Establishing a first conversion equation between the laser radar coordinate system and the camera coordinate system based on the external parameter; Establishing a second conversion equation between the camera coordinate system and the image coordinate system based on the intrinsic parameters of the camera; Based on the first conversion equation and the second conversion equation, superimposing and displaying the calibration image and the point cloud; The external parameter is adjusted and determined so that the positioning mark in the calibration image and the positioning mark in the point cloud coincide with each other.

6. The method according to claim 5, characterized in that The superimposing and displaying the calibration image and the point cloud based on the first conversion equation and the second conversion equation comprises: Based on the first conversion equation and the second conversion equation, converting the point cloud from the laser radar coordinate system to the image coordinate system; Based on the coordinate range of the calibration image, determining whether each projection point in the point cloud is on the calibration image; The projection points on the calibration image are marked and displayed.

7. A screw hole positioning device, characterized in that: The device is applied to a screw locking robot, which includes a mechanical arm and a screw locking device, a laser emitting device and a camera installed on the mechanical arm. The emitting direction of the laser emitting device is parallel to the axial direction of the screw locking device. The device includes: An acquisition module, used for acquiring a screw hole image based on the calibrated camera, wherein the screw hole image includes a target screw hole and a light spot formed by the laser emitted by the laser emitting device projected on the surface of the mounting part; An acquisition module, used for acquiring a position deviation value between the end of the screw locking device and the target screw hole in a camera coordinate system based on the screw hole image and the internal parameters of the camera; The moving module is used to control the moving direction and moving distance of the robot arm based on the position deviation value until the position deviation value is less than a preset threshold.

8. A screw locking robot, characterized in that: The screw locking robot includes a robotic arm and a screw locking device, a laser emitting device and a camera installed on the robotic arm, and the screw hole positioning device according to claim 7; the emitting direction of the laser emitting device is parallel to the axial direction of the screw locking device.

9. The screw locking robot according to claim 8, characterized in that: A laser radar is also installed on the mechanical arm.

10. The screw locking robot according to claim 8, characterized in that: The screw locking robot also includes a mobile chassis, and the mobile chassis is used to control the mechanical arm to move in a set direction and distance.

Images