Camera and line laser radar fusion positioning method, device and positioning system
Through the fusion positioning method of camera and line lidar and combined with external parameter calibration matrix conversion coordinates, the precise positioning of the lightning arrester screw is achieved, solving the problem of low positioning accuracy in the existing technology, and improving the efficiency and safety of automated maintenance.
Patent Information
- Application Number
- CN202411013482.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-07-26
Smart Images

Figure CN118544366B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of positioning technology, and in particular to a camera and line laser radar fusion positioning method, device, computer program product and positioning system. Background Art
[0002] High-voltage transmission lines are an important part of the power system, and lightning arresters must be installed to prevent them from being damaged by lightning outdoors. However, the inspection and replacement of lightning arresters are mainly completed by climbing poles manually, which has low construction efficiency and is difficult to ensure personal safety. Therefore, it is of great significance to automatically complete the inspection and replacement of lightning arresters.
[0003] Accurately positioning the arrester screw is the basis for realizing the automated maintenance and replacement of the arrester by robots. At present, for positioning the arrester screw, the monocular camera has a large sensing range, but the positioning accuracy is low in the outdoors, and the line laser radar has high positioning accuracy, but the sensing range is small. Both of them are difficult to achieve the positioning of the arrester screw alone. Summary of the invention
[0004] The main purpose of this application is to provide a camera and line laser radar fusion positioning method, device, computer program product and positioning system to at least solve the problem that accurate positioning is difficult to achieve in the prior art.
[0005] According to another aspect of the present application, a camera and line laser radar fusion positioning method is provided, wherein the camera and the line laser radar are located on a tooling platform of a robotic arm, and the tooling platform is located on one side of the end of the robotic arm. The method comprises: controlling the camera to capture an image of a target position, and determining a first target coordinate based on the image of the target position, wherein the first target coordinate is the coordinate of the target position in the camera coordinate system; converting the first target coordinate into a second target coordinate based on a first external parameter calibration matrix, and controlling the end of the robotic arm to move to the position where the second target coordinate is located, wherein the first external parameter calibration matrix is a matrix of the camera coordinate system and the end coordinate of the robotic arm. The method comprises the following steps: obtaining a full scan point cloud, wherein the full scan point cloud is a collection of multi-frame point clouds collected by the line laser radar during multiple movements of the end of the robotic arm along a predetermined axis, wherein the predetermined axis is an axis perpendicular to a sensing plane of the line laser radar, and the sensing plane of the line laser radar is a plane where the point cloud detected by the line laser radar is located; determining a nearest coordinate point by the point closest to the origin of the offline laser radar coordinate system in the full scan point cloud; converting the coordinates of the nearest coordinate point into positioning coordinates according to a second extrinsic parameter calibration matrix, wherein the second extrinsic parameter calibration matrix is a coordinate conversion matrix between the line laser radar coordinate system and the coordinate system of the end of the robotic arm.
[0006] Optionally, before controlling the camera to capture an image of the target position, the method further includes: establishing the line laser radar coordinate system, the camera coordinate system and the robotic arm end coordinate system, so that the directions of the X-axis, Y-axis and Z-axis of the line laser radar coordinate system, the camera coordinate system and the robotic arm end coordinate system are the same, and the origins are different, the Z-axis is perpendicular to the tooling plane, the Y-axis is perpendicular to the line laser radar sensing plane, and the tooling plane is the installation plane of the tooling platform; performing external parameter calibration on the line laser radar, the camera and the robotic arm end in pairs to obtain the first external parameter calibration matrix, the second external parameter calibration matrix and the third external parameter calibration matrix, and the third external parameter calibration matrix is the coordinate conversion matrix between the line laser radar coordinate system and the camera coordinate system.
[0007] Optionally, obtaining a full scan point cloud includes: controlling the end of the robotic arm to move multiple times along the positive direction of the predetermined axis with the second target coordinates as the starting point, and controlling the line laser radar to collect a frame of point cloud after each movement of the robotic arm end, so as to obtain multiple first point clouds; controlling the end of the robotic arm to move to the position where the second target coordinates are located; controlling the end of the robotic arm to move multiple times along the negative direction of the predetermined axis with the second target coordinates as the starting point, and controlling the line laser radar to collect a frame of point cloud after each movement of the robotic arm end, so as to obtain multiple second point clouds; and combining multiple first point clouds and multiple second point clouds into the full scan point cloud.
[0008] Optionally, determining the first target coordinates according to the image of the target position includes: determining the pixel coordinates of the target position according to the image of the target position, the pixel coordinates of the target position being the pixel coordinates of the target position in the image; converting the pixel coordinates of the target position into the first target coordinates according to an intrinsic parameter matrix, the intrinsic parameter matrix being used for coordinate conversion between the camera coordinate system and the coordinate system of the image.
[0009] Optionally, the nearest coordinate point is determined by the point closest to the origin of the offline laser radar coordinate system in the full-scan point cloud, including: converting the polar coordinates of all points in the full-scan point cloud into the coordinates of the line laser radar coordinate system to obtain multiple line laser radar coordinates; and determining the point with the smallest Z-axis coordinate among the multiple line laser radar coordinates as the nearest coordinate point.
[0010] Optionally, before converting the coordinates of the nearest coordinate point into positioning coordinates according to the second external parameter calibration matrix, the method also includes: determining points in the full scan point cloud whose distance from the nearest coordinate point is less than a predetermined distance as candidate points; calculating the average value of the coordinates of all the candidate points to obtain average coordinates; and updating the coordinates of the nearest coordinate point to the average coordinates.
[0011] Optionally, the target position is the position of the center hole of the screw of the lightning arrester. After the coordinates of the nearest coordinate point are converted into positioning coordinates according to the second external parameter calibration matrix, the method also includes: controlling the end of the robotic arm to move to the positioning coordinates to inspect or replace the lightning arrester.
[0012] According to another aspect of the present application, a camera and line laser radar fusion positioning device is provided, wherein the camera and the line laser radar are located on a tooling platform of a robotic arm, and the tooling platform is located on one side of the end of the robotic arm, and the device includes: a control unit, used to control the camera to collect an image of a target position, and determine a first target coordinate based on the image of the target position, wherein the first target coordinate is the coordinate of the target position in the camera coordinate system; a first conversion unit, used to convert the first target coordinate into a second target coordinate based on a first extrinsic parameter calibration matrix, and control the end of the robotic arm to move to the position where the second target coordinate is located, wherein the first extrinsic parameter calibration matrix is a matrix between the camera coordinate system and the coordinate system of the end of the robotic arm. A coordinate conversion matrix; an acquisition unit, used to acquire a full-scan point cloud, wherein the full-scan point cloud is a collection of point clouds collected by the line laser radar when the end of the robotic arm moves multiple times along a predetermined axis, wherein the predetermined axis is an axis perpendicular to the line laser radar sensing plane, and the line laser radar sensing plane is a plane where the point cloud detected by the line laser radar is located; a first determination unit, used to determine the nearest coordinate point of the point closest to the origin of the offline laser radar coordinate system in the full-scan point cloud; a second conversion unit, used to convert the coordinates of the nearest coordinate point into positioning coordinates according to a second external parameter calibration matrix, wherein the second external parameter calibration matrix is a coordinate conversion matrix between the line laser radar coordinate system and the coordinate system of the end of the robotic arm.
[0013] According to another aspect of the present application, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, any one of the methods described above is implemented.
[0014] According to another aspect of the present application, a positioning system is provided, comprising: a camera, a line laser radar, a robotic arm, one or more processors, a memory, and one or more programs, wherein the camera and the line laser radar are located on a tooling platform of the robotic arm, and the tooling platform is located on one side of the end of the robotic arm, and the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the methods described.
[0015] By applying the technical solution of the present application, the method determines the coordinates of the target position in the coordinate system of the end of the robot arm through camera positioning, that is, the second target coordinates, and then moves the end of the robot arm to the second target coordinates, so that the end of the robot arm moves to the vicinity of the target position. The end of the robot arm moves again near the target position for multiple times to obtain multiple point clouds to form a full-scan point cloud. The nearest point of the offline laser radar in the point cloud can be found, and the coordinates of the target position located by the fusion of the camera and the line laser radar can be determined, and then converted into the coordinates of the coordinate system of the end of the robot arm to obtain the final positioning coordinates, thereby achieving efficient and precise positioning. The end of the robot arm can be moved to the positioning coordinates to perform corresponding operations, thereby solving the problem that it is difficult to achieve precise positioning in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 A hardware structure block diagram of a mobile terminal for executing a camera and line laser radar fusion positioning method provided in an embodiment of the present application is shown;
[0017] Figure 2 A schematic diagram of a flow chart of a camera and line laser radar fusion positioning method provided according to an embodiment of the present application is shown;
[0018] Figure 3 A schematic diagram of a line laser radar coordinate system, a camera coordinate system, and a robotic arm end coordinate system provided according to an embodiment of the present application is shown;
[0019] Figure 4 A structural block diagram of a camera and line laser radar fusion positioning device provided according to an embodiment of the present application is shown.
[0020] The above drawings include the following reference numerals:
[0021] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION
[0022] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0025] As introduced in the background technology, it is difficult to achieve precise positioning in the prior art. To solve this technical problem, the embodiments of the present application provide a camera and line laser radar fusion positioning method, device, computer program product and positioning system.
[0026] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0027] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal for a camera and laser radar fusion positioning method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.
[0028] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the display method of device information in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0029] In this embodiment, a camera and line laser radar fusion positioning method running on a mobile terminal, a computer terminal or a similar computing device is provided. The camera and the line laser radar are located on the tooling platform of the robotic arm, and the above-mentioned tooling platform is located on one side of the end of the robotic arm. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.
[0030] Figure 2 FIG. 1 is a flow chart of a camera and line laser radar fusion positioning method according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:
[0031] Step S201, controlling the camera to collect an image of a target position, and determining first target coordinates according to the image of the target position, where the first target coordinates are coordinates of the target position in a camera coordinate system;
[0032] Step S202, converting the first target coordinates into second target coordinates according to a first extrinsic calibration matrix, and controlling the end of the robotic arm to move to the position where the second target coordinates are located, wherein the first extrinsic calibration matrix is a coordinate conversion matrix between the camera coordinate system and the robotic arm end coordinate system;
[0033] Step S203, obtaining a full scan point cloud, wherein the full scan point cloud is a collection of multi-frame point clouds collected by controlling the line laser radar during the process of the end of the robotic arm moving multiple times along a predetermined axis, wherein the predetermined axis is an axis perpendicular to a sensing plane of the line laser radar, and the sensing plane of the line laser radar is a plane where the point cloud detected by the line laser radar is located;
[0034] Step S204, determining the nearest coordinate point of the point closest to the origin of the offline laser radar coordinate system in the full scan point cloud;
[0035] Step S205, converting the coordinates of the above-mentioned nearest coordinate point into positioning coordinates according to the second external parameter calibration matrix, wherein the above-mentioned second external parameter calibration matrix is a coordinate conversion matrix between the above-mentioned line laser radar coordinate system and the above-mentioned robot arm end coordinate system.
[0036] In the above-mentioned camera and line laser radar fusion positioning method, the coordinates of the target position in the coordinate system of the end of the robot arm, that is, the second target coordinates, are determined by camera positioning, and then the end of the robot arm is moved to the second target coordinates, so that the end of the robot arm is moved to the vicinity of the target position. The end of the robot arm is moved near the target position for multiple times to obtain multiple point clouds to form a full-scan point cloud. The nearest point of the offline laser radar in the point cloud can be found, and the coordinates of the target position for the camera and line laser radar fusion positioning can be determined, and then converted into the coordinates of the coordinate system of the end of the robot arm to obtain the final positioning coordinates, thereby achieving efficient and precise positioning, and the end of the robot arm can be moved to the positioning coordinates to perform corresponding operations, thereby solving the problem that it is difficult to achieve precise positioning in the prior art.
[0037] In order to implement external parameter calibration, in an optional implementation manner, before controlling the camera to collect an image of the target position, the method further includes:
[0038] Step S301, establishing the above-mentioned line laser radar coordinate system, the above-mentioned camera coordinate system and the above-mentioned robot arm end coordinate system, so that the directions of the X-axis, Y-axis and Z-axis of the above-mentioned line laser radar coordinate system, the above-mentioned camera coordinate system and the above-mentioned robot arm end coordinate system are the same, and the origins are different, the above-mentioned Z-axis is perpendicular to the tooling plane, the above-mentioned Y-axis is perpendicular to the above-mentioned line laser radar sensing plane, and the above-mentioned tooling plane is the installation plane of the above-mentioned tooling platform;
[0039] Step S302, performing extrinsic calibration on the above-mentioned line laser radar, the above-mentioned camera and the above-mentioned robotic arm end in pairs, to obtain the above-mentioned first extrinsic parameter calibration matrix, the above-mentioned second extrinsic parameter calibration matrix and the third extrinsic parameter calibration matrix, wherein the above-mentioned third extrinsic parameter calibration matrix is a coordinate conversion matrix between the above-mentioned line laser radar coordinate system and the above-mentioned camera coordinate system.
[0040] In the above implementation mode, if Figure 3As shown in the figure, the robot base coordinate system, the line laser radar coordinate system and the camera coordinate system are defined respectively. Among them, the triangular area is the sensing plane of the line laser radar, which is perpendicular to the tooling plane, the center of the laser emitter is the origin, the Z axis is perpendicular to the tooling plane, and the Y axis is perpendicular to the line laser radar sensing plane. The camera and line laser radar, the end of the robot and the camera are calibrated with external parameters respectively to obtain the third external parameter calibration matrix And the first external parameter calibration matrix , then the second external parameter calibration matrix between the end of the manipulator and the line laser radar is ,Right now .
[0041] In order to achieve a comprehensive scan, in an optional implementation, the above step S203 includes:
[0042] Step S2031, controlling the end of the robotic arm to move multiple times along the positive direction of the predetermined axis with the second target coordinate as the starting point, and controlling the line laser radar to collect a frame of point cloud after each movement of the end of the robotic arm, to obtain multiple first point clouds;
[0043] Step S2032, controlling the end of the robotic arm to move to the position where the second target coordinates are located;
[0044] Step S2033, controlling the end of the robotic arm to move multiple times along the negative direction of the predetermined axis with the second target coordinate as the starting point, and controlling the line laser radar to collect a frame of point cloud after each movement of the end of the robotic arm, to obtain multiple second point clouds;
[0045] Step S2034: Combining the plurality of first point clouds and the plurality of second point clouds into the full scan point cloud.
[0046] In the above embodiment, the robot arm moves in the positive direction along the Y axis of the laser radar coordinate system, the number of steps is step, and the step length is d movey During this process, the linear laser radar posture remains unchanged, and the first point cloud of each step of the robotic arm is collected synchronously. , and multiple first point clouds Deposit in order In ; where ρ is the length of the emitted laser beam; angle is the angle between the laser beam and the normal line of the line laser radar plane, clockwise is positive and counterclockwise is negative; the line laser radar collects data from 160 points per frame. The robot returns to its original position and then moves in the opposite direction along the Y axis of the line laser radar coordinate system. The number of steps is step and the step length is d movey During this process, the linear laser radar posture remains unchanged, and the second point cloud of each step of the robotic arm is collected synchronously. , and multiple second point clouds Deposit in order In ; Sort by order and , that is, the full scan point cloud , to achieve a comprehensive scan near the second target coordinates to avoid missing the target position.
[0047] In order to obtain the positioning coordinates of the camera coordinate system, in an optional implementation, the above step S201 includes:
[0048] Step S2011, determining pixel coordinates of the target position according to the image of the target position, where the pixel coordinates of the target position are the pixel coordinates of the target position in the image;
[0049] Step S2012: converting the pixel coordinates of the target position into the first target coordinates according to an intrinsic parameter matrix, wherein the intrinsic parameter matrix is used for coordinate conversion between the camera coordinate system and the image coordinate system.
[0050] In the above implementation, the YOLOv9 target detection algorithm is used for image recognition to obtain an image containing the arrester screw target. The center coordinate of the arrester screw is P cam , according to the camera intrinsic parameter matrix, the pixel coordinate P cam Transformed to the xOy plane of the world coordinate system, the coordinates are the first target coordinates P world ; After that, we can use the external parameter matrix of the camera and line laser , the first target coordinate P world Convert to the line laser radar coordinate system, that is ; According to the external parameter matrix of the line laser radar and the end of the robotic arm , P lidar Transformed to the robot end coordinate system, that is, the second target coordinate .
[0051] In order to determine the point closest to the target position, in an optional implementation, the above step S204 includes:
[0052] Step S2041, converting the polar coordinates of all points in the full scan point cloud into the coordinates of the line laser radar coordinate system to obtain a plurality of line laser radar coordinates;
[0053] Step S2042: determine the point with the smallest Z-axis coordinate among the multiple line laser radar coordinates as the nearest coordinate point.
[0054] In the above implementation, the points in the polar coordinate system are transformed into the Cartesian coordinate system, that is, , where θ is the angle between the laser transmitter and the receiver of the line laser radar. According to the Z-axis coordinate value, points c Sort the points in from small to large and get the point closest to the origin in the Z-axis direction. , which is the point closest to the target position.
[0055] In order to improve the positioning accuracy, in an optional implementation manner, before converting the coordinates of the above-mentioned nearest coordinate point into positioning coordinates according to the second extrinsic calibration matrix, the above-mentioned method further includes:
[0056] Step S401, determining a point in the full scan point cloud whose distance to the nearest coordinate point is less than a predetermined distance as a candidate point;
[0057] Step S402, calculating the average value of the coordinates of all the candidate points to obtain the average coordinates;
[0058] Step S403, updating the coordinates of the above-mentioned nearest coordinate point to the above-mentioned average coordinates.
[0059] In the above implementation, points are extracted c The Z-axis coordinate is (z min ,z min +d near ) and store it in points near ,Right now ; Among them, z i for The Z-axis coordinate of the i-th point in . Calculate points near The average value of points avg =(x avg ,y avg ,z avg ),Right now ; where n is points near The number of midpoints can further optimize the positioning and improve the positioning accuracy.
[0060] In order to realize automatic maintenance, in an optional implementation manner, the target position is the position of the center hole of the screw of the arrester. After the coordinates of the nearest coordinate point are converted into positioning coordinates according to the second external parameter calibration matrix, the method further includes:
[0061] Step S501, controlling the end of the above-mentioned robot arm to move to the above-mentioned positioning coordinates to inspect or replace the above-mentioned lightning arrester.
[0062] In the above embodiment, according to the second external parameter calibration matrix of the line laser radar and the end of the robotic arm , points avgTransformed to the coordinate system of the end of the robotic arm, the positioning coordinates of the arrester screw are The end of the mechanical arm can be controlled to move to the positioning coordinates to inspect or replace the arrester.
[0063] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0064] The embodiment of the present application also provides a camera and line laser radar fusion positioning device. It should be noted that the camera and line laser radar fusion positioning device of the embodiment of the present application can be used to execute the camera and line laser radar fusion positioning method provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred embodiments, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.
[0065] The following is an introduction to the camera and line laser radar fusion positioning device provided in an embodiment of the present application. The camera and the line laser radar are located on a tooling platform of the robotic arm, and the tooling platform is located on one side of the end of the robotic arm.
[0066] Figure 4 is a structural block diagram of a camera and line laser radar fusion positioning device according to an embodiment of the present application. Figure 4 As shown, the device comprises:
[0067] A control unit 10, used to control the camera to collect an image of a target position, and determine a first target coordinate according to the image of the target position, wherein the first target coordinate is a coordinate of the target position in a camera coordinate system;
[0068] A first conversion unit 20 is used to convert the first target coordinates into second target coordinates according to a first extrinsic calibration matrix, and control the end of the robotic arm to move to the position where the second target coordinates are located, wherein the first extrinsic calibration matrix is a coordinate conversion matrix between the camera coordinate system and the end coordinate system of the robotic arm;
[0069] The acquisition unit 30 is used to acquire a full scan point cloud, wherein the full scan point cloud is a collection of point clouds collected by controlling the line laser radar during the process of the end of the robotic arm moving multiple times along a predetermined axis, wherein the predetermined axis is an axis perpendicular to a sensing plane of the line laser radar, and the sensing plane of the line laser radar is a plane where the point cloud detected by the line laser radar is located;
[0070] The first determination unit 40 is used to determine the nearest coordinate point of the point closest to the origin of the offline laser radar coordinate system in the full scan point cloud;
[0071] The second conversion unit 50 is used to convert the coordinates of the above-mentioned nearest coordinate point into positioning coordinates according to the second external parameter calibration matrix, and the above-mentioned second external parameter calibration matrix is a coordinate conversion matrix between the above-mentioned line laser radar coordinate system and the above-mentioned robot arm end coordinate system.
[0072] In the above-mentioned camera and line laser radar fusion positioning device, the coordinates of the target position in the coordinate system of the end of the robot arm, that is, the second target coordinates, are determined by camera positioning, and then the end of the robot arm is moved to the second target coordinates, so that the end of the robot arm is moved to the vicinity of the target position. The end of the robot arm is moved near the target position for multiple times to obtain multiple point clouds to form a full-scan point cloud. The nearest point of the offline laser radar in the point cloud can be found, and the coordinates of the target position for the fusion positioning of the camera and line laser radar can be determined, and then converted into the coordinates of the coordinate system of the end of the robot arm to obtain the final positioning coordinates, thereby achieving efficient and precise positioning, and the end of the robot arm can be moved to the positioning coordinates to perform corresponding operations, thereby solving the problem that it is difficult to achieve precise positioning in the prior art.
[0073] In order to realize external parameter calibration, in an optional implementation manner, the above-mentioned device further includes:
[0074] An establishing unit, used for establishing the line laser radar coordinate system, the camera coordinate system and the robot end coordinate system before controlling the camera to collect an image of the target position, so that the directions of the X-axis, Y-axis and Z-axis of the line laser radar coordinate system, the camera coordinate system and the robot end coordinate system are the same, and the origins are different, the Z-axis is perpendicular to the tooling plane, the Y-axis is perpendicular to the line laser radar sensing plane, and the tooling plane is the installation plane of the tooling platform;
[0075] A calibration unit is used to perform external parameter calibration on the above-mentioned line laser radar, the above-mentioned camera and the above-mentioned robotic arm end in pairs to obtain the above-mentioned first external parameter calibration matrix, the above-mentioned second external parameter calibration matrix and the third external parameter calibration matrix. The above-mentioned third external parameter calibration matrix is a coordinate conversion matrix between the above-mentioned line laser radar coordinate system and the above-mentioned camera coordinate system.
[0076] In the above implementation mode, if Figure 3 As shown in the figure, the robot base coordinate system, the line laser radar coordinate system and the camera coordinate system are defined respectively. Among them, the triangular area is the sensing plane of the line laser radar, which is perpendicular to the tooling plane, the center of the laser emitter is the origin, the Z axis is perpendicular to the tooling plane, and the Y axis is perpendicular to the line laser radar sensing plane. The camera and line laser radar, the end of the robot and the camera are calibrated with external parameters respectively to obtain the third external parameter calibration matrix And the first external parameter calibration matrix , then the second external parameter calibration matrix between the end of the manipulator and the line laser radar is ,Right now .
[0077] In order to achieve a comprehensive scan, in an optional implementation, the acquisition unit includes:
[0078] A first control module is used to control the end of the robotic arm to move multiple times along the positive direction of the predetermined axis with the second target coordinate as the starting point, and control the line laser radar to collect a frame of point cloud after each movement of the end of the robotic arm, so as to obtain multiple first point clouds;
[0079] A second control module, used for controlling the end of the robotic arm to move to the position where the second target coordinate is located;
[0080] A third control module is used to control the end of the robotic arm to move multiple times along the negative direction of the predetermined axis with the second target coordinate as the starting point, and control the line laser radar to collect a frame of point cloud after each movement of the end of the robotic arm to obtain multiple second point clouds;
[0081] A composition module is used to combine the plurality of the first point clouds and the plurality of the second point clouds into the full scan point cloud.
[0082] In the above embodiment, the robot arm moves in the positive direction along the Y axis of the laser radar coordinate system, the number of steps is step, and the step length is d movey During this process, the linear laser radar posture remains unchanged, and the first point cloud of each step of the robotic arm is collected synchronously. , and multiple first point clouds Deposit in order In ; where ρ is the length of the emitted laser beam; angle is the angle between the laser beam and the normal line of the line laser radar plane, clockwise is positive and counterclockwise is negative; the line laser radar collects data from 160 points per frame. The robot returns to its original position and then moves in the opposite direction along the Y axis of the line laser radar coordinate system. The number of steps is step and the step length is d movey During this process, the linear laser radar posture remains unchanged, and the second point cloud of each step of the robotic arm is collected synchronously. , and multiple second point clouds Deposit in order In ; Sort by order and , that is, the full scan point cloud , to achieve a comprehensive scan near the second target coordinates to avoid missing the target position.
[0083] In order to obtain the positioning coordinates of the camera coordinate system, in an optional implementation, the control unit includes:
[0084] A first determining module is used to determine the pixel coordinates of the target position according to the image of the target position, where the pixel coordinates of the target position are the pixel coordinates of the target position in the image;
[0085] The first conversion module is used to convert the pixel coordinates of the target position into the first target coordinates according to an intrinsic parameter matrix, and the intrinsic parameter matrix is used for coordinate conversion between the camera coordinate system and the image coordinate system.
[0086] In the above implementation, the YOLOv9 target detection algorithm is used for image recognition to obtain an image containing the arrester screw target. The center coordinate of the arrester screw is P cam , according to the camera intrinsic parameter matrix, the pixel coordinate P cam Transformed to the xOy plane of the world coordinate system, the coordinates are the first target coordinates P world ; After that, we can use the external parameter matrix of the camera and line laser , the first target coordinate P world Convert to the line laser radar coordinate system, that is ; According to the external parameter matrix of the line laser radar and the end of the robotic arm , P lidar Transformed to the robot end coordinate system, that is, the second target coordinate .
[0087] In order to determine the point closest to the target position, in an optional implementation manner, the first determining unit includes:
[0088] A second conversion module is used to convert the polar coordinates of all points in the full scan point cloud into the coordinates of the line laser radar coordinate system to obtain a plurality of line laser radar coordinates;
[0089] The second determination module is used to determine the point with the smallest Z-axis coordinate among the multiple line laser radar coordinates as the nearest coordinate point.
[0090] In the above implementation, the points in the polar coordinate system are transformed into the Cartesian coordinate system, that is, , where θ is the angle between the laser transmitter and the receiver of the line laser radar. According to the Z-axis coordinate value, points c Sort the points in from small to large and get the point closest to the origin in the Z-axis direction. , which is the point closest to the target position.
[0091] In order to improve the positioning accuracy, in an optional implementation manner, the above device further includes:
[0092] A second determination unit is used to determine, as a candidate point, a point in the full scan point cloud whose distance to the nearest coordinate point is less than a predetermined distance before converting the coordinates of the nearest coordinate point into positioning coordinates according to a second external parameter calibration matrix;
[0093] A calculation unit, used to calculate the average value of the coordinates of all the candidate points to obtain the average coordinates;
[0094] The updating unit is used to update the coordinates of the above-mentioned nearest coordinate point to the above-mentioned average coordinates.
[0095] In the above implementation, points are extracted c The Z-axis coordinate is (z min ,z min +d near ) and store it in points near ,Right now ; Among them, z i for The Z-axis coordinate of the i-th point in . Calculate points near The average value of points avg =(x avg ,y avg ,z avg ),Right now ; where n is points near The number of midpoints can further optimize the positioning and improve the positioning accuracy.
[0096] In order to realize automatic maintenance, in an optional implementation manner, the target position is the position of the center hole of the screw of the arrester. After the coordinates of the nearest coordinate point are converted into positioning coordinates according to the second external parameter calibration matrix, the method further includes:
[0097] Step S501, controlling the end of the above-mentioned robot arm to move to the above-mentioned positioning coordinates to inspect or replace the above-mentioned lightning arrester.
[0098] In the above embodiment, according to the second external parameter calibration matrix of the line laser radar and the end of the robotic arm , points avg Transformed to the coordinate system of the end of the robotic arm, the positioning coordinates of the arrester screw are The end of the mechanical arm can be controlled to move to the positioning coordinates to inspect or replace the arrester.
[0099] The camera and line laser radar fusion positioning device includes a processor and a memory. The control unit, the first conversion unit, the acquisition unit, the first determination unit, the second conversion unit, etc. are all stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.
[0100] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the problem of difficult to achieve accurate positioning in the prior art can be solved by adjusting kernel parameters.
[0101] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.
[0102] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the camera and line laser radar fusion positioning method.
[0103] Specifically, the camera and line laser radar fusion positioning method includes:
[0104] Step S201, controlling the camera to collect an image of a target position, and determining first target coordinates according to the image of the target position, where the first target coordinates are coordinates of the target position in a camera coordinate system;
[0105] Step S202, converting the first target coordinates into second target coordinates according to a first extrinsic calibration matrix, and controlling the end of the robotic arm to move to the position where the second target coordinates are located, wherein the first extrinsic calibration matrix is a coordinate conversion matrix between the camera coordinate system and the robotic arm end coordinate system;
[0106] Step S203, obtaining a full scan point cloud, wherein the full scan point cloud is a collection of multi-frame point clouds collected by controlling the line laser radar during the process of the end of the robotic arm moving multiple times along a predetermined axis, wherein the predetermined axis is an axis perpendicular to a sensing plane of the line laser radar, and the sensing plane of the line laser radar is a plane where the point cloud detected by the line laser radar is located;
[0107] Step S204, determining the nearest coordinate point of the point closest to the origin of the offline laser radar coordinate system in the full scan point cloud;
[0108] Step S205, converting the coordinates of the above-mentioned nearest coordinate point into positioning coordinates according to the second external parameter calibration matrix, wherein the above-mentioned second external parameter calibration matrix is a coordinate conversion matrix between the above-mentioned line laser radar coordinate system and the above-mentioned robot arm end coordinate system.
[0109] An embodiment of the present invention provides a processor, which is used to run a program, wherein the camera and line laser radar fusion positioning method is executed when the program is run.
[0110] Specifically, the camera and line laser radar fusion positioning method includes:
[0111] Step S201, controlling the camera to collect an image of a target position, and determining first target coordinates according to the image of the target position, where the first target coordinates are coordinates of the target position in a camera coordinate system;
[0112] Step S202, converting the first target coordinates into second target coordinates according to a first extrinsic calibration matrix, and controlling the end of the robotic arm to move to the position where the second target coordinates are located, wherein the first extrinsic calibration matrix is a coordinate conversion matrix between the camera coordinate system and the robotic arm end coordinate system;
[0113] Step S203, obtaining a full scan point cloud, wherein the full scan point cloud is a collection of multi-frame point clouds collected by controlling the line laser radar during the process of the end of the robotic arm moving multiple times along a predetermined axis, wherein the predetermined axis is an axis perpendicular to a sensing plane of the line laser radar, and the sensing plane of the line laser radar is a plane where the point cloud detected by the line laser radar is located;
[0114] Step S204, determining the nearest coordinate point of the point closest to the origin of the offline laser radar coordinate system in the full scan point cloud;
[0115] Step S205, converting the coordinates of the above-mentioned nearest coordinate point into positioning coordinates according to the second external parameter calibration matrix, wherein the above-mentioned second external parameter calibration matrix is a coordinate conversion matrix between the above-mentioned line laser radar coordinate system and the above-mentioned robot arm end coordinate system.
[0116] An embodiment of the present invention provides a positioning system, which includes a camera, a line laser radar, a robotic arm, a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, at least the following steps are implemented:
[0117] Step S201, controlling the camera to collect an image of a target position, and determining first target coordinates according to the image of the target position, where the first target coordinates are coordinates of the target position in a camera coordinate system;
[0118] Step S202, converting the first target coordinates into second target coordinates according to a first extrinsic calibration matrix, and controlling the end of the robotic arm to move to the position where the second target coordinates are located, wherein the first extrinsic calibration matrix is a coordinate conversion matrix between the camera coordinate system and the robotic arm end coordinate system;
[0119] Step S203, obtaining a full scan point cloud, wherein the full scan point cloud is a collection of multi-frame point clouds collected by controlling the line laser radar during the process of the end of the robotic arm moving multiple times along a predetermined axis, wherein the predetermined axis is an axis perpendicular to a sensing plane of the line laser radar, and the sensing plane of the line laser radar is a plane where the point cloud detected by the line laser radar is located;
[0120] Step S204, determining the nearest coordinate point of the point closest to the origin of the offline laser radar coordinate system in the full scan point cloud;
[0121] Step S205, converting the coordinates of the above-mentioned nearest coordinate point into positioning coordinates according to the second external parameter calibration matrix, wherein the above-mentioned second external parameter calibration matrix is a coordinate conversion matrix between the above-mentioned line laser radar coordinate system and the above-mentioned robot arm end coordinate system.
[0122] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:
[0123] Step S201, controlling the camera to collect an image of a target position, and determining first target coordinates according to the image of the target position, where the first target coordinates are coordinates of the target position in a camera coordinate system;
[0124] Step S202, converting the first target coordinates into second target coordinates according to a first extrinsic calibration matrix, and controlling the end of the robotic arm to move to the position where the second target coordinates are located, wherein the first extrinsic calibration matrix is a coordinate conversion matrix between the camera coordinate system and the robotic arm end coordinate system;
[0125] Step S203, obtaining a full scan point cloud, wherein the full scan point cloud is a collection of multi-frame point clouds collected by controlling the line laser radar during the process of the end of the robotic arm moving multiple times along a predetermined axis, wherein the predetermined axis is an axis perpendicular to a sensing plane of the line laser radar, and the sensing plane of the line laser radar is a plane where the point cloud detected by the line laser radar is located;
[0126] Step S204, determining the nearest coordinate point of the point closest to the origin of the offline laser radar coordinate system in the full scan point cloud;
[0127] Step S205, converting the coordinates of the above-mentioned nearest coordinate point into positioning coordinates according to the second external parameter calibration matrix, wherein the above-mentioned second external parameter calibration matrix is a coordinate conversion matrix between the above-mentioned line laser radar coordinate system and the above-mentioned robot arm end coordinate system.
[0128] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0129] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0130] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0131] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0133] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0134] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0135] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0136] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0137] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:
[0138] 1) In the camera and line laser radar fusion positioning method of the present application, the coordinates of the target position in the coordinate system of the end of the robot arm, that is, the second target coordinates, are determined by camera positioning, and then the end of the robot arm is moved to the second target coordinates, so that the end of the robot arm is moved to the vicinity of the target position. The end of the robot arm is moved near the target position for multiple times to obtain multiple point clouds to form a full-scan point cloud. The nearest point of the offline laser radar in the point cloud can be found, and the coordinates of the target position for the camera and line laser radar fusion positioning can be determined, and then converted into the coordinates of the coordinate system of the end of the robot arm to obtain the final positioning coordinates, thereby achieving efficient and precise positioning. The end of the robot arm can be moved to the positioning coordinates to perform corresponding operations, thereby solving the problem that it is difficult to achieve precise positioning in the prior art.
[0139] 2) In the camera and line laser radar fusion positioning device of the present application, the coordinates of the target position in the coordinate system of the end of the robot arm, that is, the second target coordinates, are determined by camera positioning, and then the end of the robot arm is moved to the second target coordinates, so that the end of the robot arm is moved to the vicinity of the target position. The end of the robot arm is moved near the target position for multiple times to obtain multiple point clouds to form a full-scan point cloud. The nearest point of the offline laser radar in the point cloud can be found, and the coordinates of the target position located by the fusion of the camera and the line laser radar can be determined, and then converted into the coordinates of the coordinate system of the end of the robot arm to obtain the final positioning coordinates, thereby achieving efficient and precise positioning. The end of the robot arm can be moved to the positioning coordinates to perform corresponding operations, thereby solving the problem of difficult to achieve precise positioning in the prior art.
[0140] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A camera and line laser radar fusion positioning method, characterized in that: The camera and the line laser radar are located on a tooling platform of the robotic arm, and the tooling platform is located on one side of the end of the robotic arm. The method includes: Controlling the camera to capture an image of a target position, and determining first target coordinates according to the image of the target position, where the first target coordinates are coordinates of the target position in a camera coordinate system; The first target coordinates are converted into second target coordinates according to a first extrinsic calibration matrix, and the end of the robotic arm is controlled to move to a position where the second target coordinates are located, wherein the first extrinsic calibration matrix is a coordinate conversion matrix between the camera coordinate system and the end coordinate system of the robotic arm; Acquire a full scan point cloud, where the full scan point cloud is a collection of multi-frame point clouds collected by the line laser radar during the process of the end of the robotic arm moving multiple times along a predetermined axis, where the predetermined axis is an axis perpendicular to a sensing plane of the line laser radar, and the sensing plane of the line laser radar is a plane where the point cloud detected by the line laser radar is located; Determine the nearest coordinate point of the point closest to the origin of the offline laser radar coordinate system in the full scan point cloud; The coordinates of the nearest coordinate point are converted into positioning coordinates according to a second external parameter calibration matrix, wherein the second external parameter calibration matrix is a coordinate conversion matrix between the line laser radar coordinate system and the robot arm end coordinate system; obtaining a full scan point cloud, including: controlling the robot arm end to move multiple times along the positive direction of the predetermined axis with the second target coordinate as the starting point, and controlling the line laser radar to collect a frame of point cloud after each movement of the robot arm end, to obtain multiple first point clouds; controlling the robot arm end to move to the position where the second target coordinate is located; controlling the robot arm end to move multiple times along the negative direction of the predetermined axis with the second target coordinate as the starting point, and controlling the line laser radar to collect a frame of point cloud after each movement of the robot arm end, to obtain multiple second point clouds; and combining multiple first point clouds and multiple second point clouds into the full scan point cloud.
2. The method according to claim 1, characterized in that Before controlling the camera to collect an image of the target position, the method further includes: Establish the line laser radar coordinate system, the camera coordinate system and the robot end coordinate system, so that the directions of the X-axis, Y-axis and Z-axis of the line laser radar coordinate system, the camera coordinate system and the robot end coordinate system are the same, and the origins are different, the Z-axis is perpendicular to the tooling plane, the Y-axis is perpendicular to the line laser radar sensing plane, and the tooling plane is the installation plane of the tooling platform; The line laser radar, the camera and the end of the robotic arm are calibrated with external parameters in pairs to obtain the first external parameter calibration matrix, the second external parameter calibration matrix and the third external parameter calibration matrix, where the third external parameter calibration matrix is a coordinate conversion matrix between the line laser radar coordinate system and the camera coordinate system.
3. The method according to claim 1, characterized in that Determining first target coordinates according to the image of the target position includes: Determine pixel coordinates of the target position according to the image of the target position, wherein the pixel coordinates of the target position are pixel coordinates of the target position in the image; The pixel coordinates of the target position are converted into the first target coordinates according to an intrinsic parameter matrix, and the intrinsic parameter matrix is used for coordinate conversion between the camera coordinate system and the image coordinate system.
4. The method according to claim 2, characterized in that: Determining the nearest coordinate point of the point closest to the origin of the offline laser radar coordinate system in the full scan point cloud includes: Convert the polar coordinates of all points in the full scan point cloud into the coordinates of the line laser radar coordinate system to obtain a plurality of line laser radar coordinates; The point with the smallest Z-axis coordinate among the multiple line laser radar coordinates is determined as the nearest coordinate point.
5. The method according to claim 1, characterized in that Before converting the coordinates of the nearest coordinate point into positioning coordinates according to the second extrinsic calibration matrix, the method further includes: Determine a point in the full scan point cloud whose distance from the nearest coordinate point is less than a predetermined distance as a candidate point; Calculate the average value of the coordinates of all the candidate points to obtain the average coordinates; The coordinates of the nearest coordinate point are updated to the average coordinates.
6. The method according to claim 1, characterized in that The target position is the position of the center hole of the screw of the arrester. After converting the coordinates of the nearest coordinate point into positioning coordinates according to the second external parameter calibration matrix, the method further includes: The end of the mechanical arm is controlled to move to the positioning coordinate to inspect or replace the lightning arrester.
7. A camera and line laser radar fusion positioning device, characterized in that: The camera and the line laser radar are located on the tooling platform of the robotic arm, and the tooling platform is located on one side of the end of the robotic arm. The device includes: A control unit, used for controlling the camera to capture an image of a target position, and determining a first target coordinate according to the image of the target position, wherein the first target coordinate is a coordinate of the target position in a camera coordinate system; a first conversion unit, configured to convert the first target coordinates into second target coordinates according to a first extrinsic calibration matrix, and control the end of the robotic arm to move to a position where the second target coordinates are located, wherein the first extrinsic calibration matrix is a coordinate conversion matrix between the camera coordinate system and the end coordinate system of the robotic arm; an acquisition unit, used for acquiring a full scan point cloud, wherein the full scan point cloud is a collection of point clouds collected by controlling the line laser radar during the process of the end of the robotic arm moving multiple times along a predetermined axis, wherein the predetermined axis is an axis perpendicular to a sensing plane of the line laser radar, and the sensing plane of the line laser radar is a plane where the point cloud detected by the line laser radar is located; A first determining unit is used to determine the nearest coordinate point of the point closest to the origin of the offline laser radar coordinate system in the full scan point cloud; A second conversion unit is used to convert the coordinates of the nearest coordinate point into positioning coordinates according to a second external parameter calibration matrix, and the second external parameter calibration matrix is a coordinate conversion matrix between the line laser radar coordinate system and the robot arm end coordinate system; the acquisition unit includes: a first control module, used to control the robot arm end to move multiple times along the positive direction of the predetermined axis with the second target coordinate as the starting point, and control the line laser radar to collect a frame of point cloud after each movement of the robot arm end, so as to obtain multiple first point clouds; a second control module, used to control the robot arm end to move to the position where the second target coordinate is located; a third control module, used to control the robot arm end to move multiple times along the negative direction of the predetermined axis with the second target coordinate as the starting point, and control the line laser radar to collect a frame of point cloud after each movement of the robot arm end, so as to obtain multiple second point clouds; a composition module, used to combine multiple first point clouds and multiple second point clouds into the full scan point cloud.
8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
9. A positioning system, characterized in that: include: A camera, a line laser radar, a robotic arm, one or more processors, a memory, and one or more programs, wherein the camera and the line laser radar are located on a tooling platform of the robotic arm, and the tooling platform is located on one side of the end of the robotic arm, and the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 6.
Citation Information
Patent Citations
Image segmentation method for object to be operated in live working scene
CN113240670A
Welding robot intelligent positioning and control method based on binocular vision and line laser sensing data fusion
CN113954085A
