Positioning method, device, robot and storage medium

Abstract

The present application relates to a positioning method, device, robot and storage medium. The method comprises: determining the calibration information of each object in the captured image according to the captured image acquired by the camera installed on the robot; acquiring virtual point cloud data corresponding to the actual point cloud data according to the actual point cloud data acquired by the laser radar installed on the robot, the position of the camera and the position of the laser radar; determining the position information of each object according to the calibration information and virtual point cloud data of each object. The method can accurately acquire the 3D position information of the target object.

Description

Technical Field

[0001] The present application relates to the field of laser radar ranging technology, and in particular to a positioning method, device, robot and storage medium. Background Art

[0002] With the continuous development of LiDAR technology, the fusion of LiDAR and camera can more accurately determine the location information of the target. The robot can detect the target object through LiDAR and camera at the same time to obtain the 3D location information of the target object.

[0003] In the related technology, both an RGB camera and a laser radar are installed on a robot. In order to obtain the 2D image information of the target object more accurately, the RGB camera is installed at a higher position, while the laser radar needs to be used for obstacle avoidance and point cloud image collection at the same time. The laser radar is installed at a lower position. The image information collected by the RGB camera is fused with the point cloud information collected by the laser radar to obtain the 3D position information of the target object.

[0004] Since the RGB camera has a blind spot, the existing technology makes it impossible to fuse the images collected by the RGB camera and the point cloud information collected by the lidar, thereby failing to obtain the 3D position information of the target object. Summary of the invention

[0005] Based on this, it is necessary to provide a positioning method, device, robot and storage medium that can accurately obtain the 3D position information of the target object in order to solve the above technical problems.

[0006] In a first aspect, the present application provides a positioning method, the method comprising: determining calibration information of each object in a captured image according to a captured image acquired by a camera disposed on a robot;

[0007] According to the actual point cloud data acquired by the laser radar set on the robot, the position of the camera and the position of the laser radar, the virtual point cloud data corresponding to the actual point cloud data is acquired;

[0008] The position information of each object is determined according to the calibration information of each object and the virtual point cloud data.

[0009] In one embodiment, determining the position information of each object according to the calibration information of each object and the virtual point cloud data includes:

[0010] Match the virtual point cloud data with the calibration information of each object respectively to determine the target object corresponding to the virtual point cloud data;

[0011] According to the actual point cloud data corresponding to the virtual point cloud data, the position information of the target object corresponding to the virtual point cloud data is determined.

[0012] In one embodiment, the calibration information includes a detection frame, and matching the virtual point cloud data with the calibration information of each object to determine the target object corresponding to the virtual point cloud data includes:

[0013] Projecting the virtual point cloud data onto the captured image, and determining the proportion of the virtual point cloud data in the detection frame of each object;

[0014] The target object corresponding to the virtual point cloud data is determined according to the proportion.

[0015] In one embodiment, determining the target object corresponding to the virtual point cloud data according to the proportion includes:

[0016] The object corresponding to the detection box whose proportion is greater than a preset threshold is determined as the target object; or

[0017] The object corresponding to the detection box with the highest proportion is determined as the target object.

[0018] In one embodiment, determining the position information of the target object corresponding to the virtual point cloud data according to the actual point cloud data of the virtual point cloud data includes:

[0019] Get the centroid of the actual point cloud data;

[0020] The position of the centroid is determined as the position information of the target object.

[0021] In one embodiment, the position of the camera includes the height of the camera, and the position of the laser radar includes the height of the laser radar; according to the actual point cloud data acquired by the laser radar set on the robot, the position of the camera and the position of the laser radar, obtaining virtual point cloud data corresponding to the actual point cloud data includes:

[0022] Create a virtual lidar based on the camera height and lidar height;

[0023] According to the actual point cloud data and the virtual laser radar, the virtual point cloud data is obtained.

[0024] In one embodiment, the method further comprises:

[0025] Filter out redundant point clouds in actual point cloud data to obtain filtered point cloud data; the redundant point clouds include other point clouds except the object;

[0026] According to the actual point cloud data obtained by the laser radar set on the robot, the position of the camera and the position of the laser radar, the virtual point cloud data corresponding to the actual point cloud data is obtained, including:

[0027] According to the filtered point cloud data, the position of the camera and the position of the laser radar, virtual point cloud data corresponding to the filtered point cloud data is obtained.

[0028] In one embodiment, filtering out redundant point clouds in actual point cloud data to obtain filtered point cloud data includes:

[0029] Get the point cloud shape of the object in the actual point cloud data;

[0030] The other point clouds except the point cloud shapes of each object in the actual point cloud data are filtered out to obtain the filtered point cloud data.

[0031] In one embodiment, the method further comprises:

[0032] Clustering the filtered point cloud data to obtain multiple point cloud clusters;

[0033] According to the filtered point cloud data, the position of the camera and the position of the laser radar, the virtual point cloud data corresponding to the filtered point cloud data is obtained, including:

[0034] According to each point cloud cluster, the position of the camera and the position of the lidar, the virtual point cloud data corresponding to each point cloud cluster is obtained.

[0035] In a second aspect, the present application further provides a positioning device, the device comprising:

[0036] A first determination module is used to determine the calibration information of each object in the captured image based on the captured image acquired by the camera disposed on the robot;

[0037] A first acquisition module is used to acquire virtual point cloud data corresponding to the actual point cloud data according to the actual point cloud data acquired by the laser radar set on the robot, the position of the camera and the position of the laser radar;

[0038] The second determination module is used to determine the position information of each object according to the calibration information of each object and the virtual point cloud data.

[0039] In a third aspect, the present application further provides a robot comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any step of the method in the first aspect when executing the computer program.

[0040] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, any step in the method in the first aspect is implemented.

[0041] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements any step of the method in the first aspect when executed by a processor.

[0042] The above positioning method, device, robot and storage medium determine the calibration information of each object in the captured image based on the captured image obtained by the camera set on the robot, obtain the virtual point cloud data corresponding to the actual point cloud data based on the actual point cloud data obtained by the laser radar set on the robot, the position of the camera and the position of the laser radar, and determine the position information of each object based on the calibration information of each object and the virtual point cloud data. The method can accurately distinguish each object in the captured image by determining the calibration information of each object in the captured image, and obtain the virtual point cloud data corresponding to the actual point cloud data based on the actual point cloud data, the position of the camera and the position of the laser radar, thereby avoiding the problem that the actual point cloud data cannot be matched with the captured image due to the height difference between the camera and the laser radar. The method determines the actual point cloud data corresponding to each object through the correspondence between the obtained virtual point cloud data and the calibration information of each object, and can accurately obtain the position information of each object in the area in front of the robot, eliminating the blind spot of camera shooting and improving the robot's perception ability of close-range objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A diagram of an application environment of a positioning method in an embodiment;

[0044] Figure 2 is a flow chart of a positioning method in one embodiment;

[0045] Figure 3 A schematic diagram of a detection frame of an object in a captured image in one embodiment;

[0046] Figure 4 A schematic diagram of a blind area for image perception in an embodiment;

[0047] Figure 5 is a flow chart of a positioning method in one embodiment;

[0048] Figure 6 is a flow chart of a positioning method in one embodiment;

[0049] Figure 7 is a flow chart of a positioning method in one embodiment;

[0050] Figure 8 is a flow chart of a positioning method in one embodiment;

[0051] Fig. 9 is a flow chart of a positioning method in one embodiment;

[0052] Fig.10 is a flow chart of a positioning method in one embodiment;

[0053] Fig.11 A schematic diagram of actual point cloud data in an embodiment;

[0054] Fig.12 is a schematic diagram of point cloud data after filtering in one embodiment;

[0055] Fig.13 is a flow chart of a positioning method in one embodiment;

[0056] Fig.14 is a schematic diagram of a point cloud cluster after clustering in one embodiment;

[0057] Fig.15 is a flow chart of a positioning method in one embodiment;

[0058] Fig.16 A schematic diagram of projection of virtual point cloud data on a captured image in one embodiment;

[0059] Fig.17 FIG. 4 is a structural block diagram of a positioning device in an embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] The positioning method provided in the embodiment of the present application can be applied to a robot, wherein the internal structure of the robot is as follows: Figure 1 As shown. The memory in the internal structure of the robot includes a non-volatile storage medium and an internal memory, the non-volatile storage medium stores an operating system, a computer program and a database; the internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database is used to store point cloud data, image data, and position data corresponding to the target. The network interface is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the positioning method provided in the present application is implemented.

[0062] In one embodiment, Figure 2 As shown, a positioning method is provided, which is applied to Figure 1 The robot in the example is used to illustrate the following steps:

[0063] S201, determining calibration information of each object in a captured image based on a captured image acquired by a camera disposed on the robot.

[0064] The calibration information refers to the position of each object in the captured image, including but not limited to the coordinates of any point in the detection frame of each object in the captured image, the pixel width and height of the detection frame of each object, etc. It can be understood that in order to obtain a larger shooting field of view, the camera is set at a higher position of the robot.

[0065] Specifically, after the camera captures the image of the front area, the image is transmitted to the robot. The robot can use the image as an input signal and input it into a commonly used target detection model. The target detection model calibrates each object in the image to obtain a detection frame for each object in the image. The detection frame can be used as the calibration information of the object. Commonly used target detection models include Fast-Regions with CNN Features, Fast-RCNN model, CenterNet model, Yolo model, etc.

[0066] Furthermore, it is understandable that the camera installed on the robot is an RGB camera, which cannot obtain the depth information of each object. Each object in the captured image obtained by the RGB camera can be a pedestrian, an obstacle, an animal, etc., and the detection frame of each object can be a rectangle, a circle, a semicircle, an irregular shape, etc. Figure 3 is a schematic diagram of the detection box of the object in the captured image. Figure 3 In the above example, the object in the captured image is a pedestrian, and the detection frame is a rectangle. Figure 3 The dotted box in the figure represents the detection box of the pedestrian. The calibration information of the pedestrian in the captured image can be expressed as (x, y, w, h), where x and y can be the coordinates of any of the four corner points of the pedestrian detection box in the captured image, and w and h represent the pixel width and height of the pedestrian detection box.

[0067] S202, obtaining virtual point cloud data corresponding to the actual point cloud data according to the actual point cloud data obtained by the laser radar installed on the robot, the position of the camera and the position of the laser radar.

[0068] Among them, single-line laser radar can not only be used for positioning and map construction or concurrent mapping and positioning (SLAM), but also for obstacle avoidance. Therefore, single-line laser radar is generally installed below the robot.

[0069] Optionally, the server can obtain virtual point cloud data at the same height as the camera based on the actual point cloud data, and obtain virtual point cloud data corresponding to the actual point cloud data. Alternatively, the server can determine the highest position of the shooting blind spot of the object closest to the robot based on the calibration information of each object in the captured image, and obtain virtual point cloud data at the highest position of the shooting blind spot between the camera and the laser radar based on the actual point cloud data, and obtain virtual point cloud data corresponding to the actual point cloud data. Alternatively, obtain the height difference between the highest position of the shooting blind spot and the camera installation position, determine the virtual point cloud data based on the height difference and the actual point cloud data, and obtain virtual point cloud data corresponding to the actual point cloud data. This embodiment does not limit the height of the virtual point cloud data.

[0070] Furthermore, it is understandable that Figure 4 This is a schematic diagram of the blind area of ​​image perception. Figure 4 The objects in the figure include pedestrian 1 and pedestrian 2. The camera can capture a full-body image of pedestrian 1, and the lidar can also capture the depth information of pedestrian 1 accordingly. Therefore, the image taken by the camera and the actual point cloud data obtained by the lidar can be fused to obtain the position information of pedestrian 1.

[0071] However, for pedestrian 2, the camera can only capture images of the part above the thigh of pedestrian 2, so the position below the thigh is determined as the shooting blind spot, and the thigh position is the highest position of the shooting blind spot. The lidar can only obtain the actual point cloud data of the knee or below, that is, its actual point cloud data is not located in the captured image taken by the camera, that is, its actual point cloud data cannot be directly projected into the captured image, so that the actual point cloud data cannot be fused with the captured image, and the position information of pedestrian 2 cannot be directly obtained.

[0072] Optionally, the robot may obtain the height difference between the highest position of the shooting blind spot and the camera, and obtain virtual point cloud data according to the height difference and actual point cloud data.

[0073] S203: Determine the position information of each object according to the calibration information of each object and the virtual point cloud data.

[0074] The position information of each object may be the relative position information between each object and the robot, or the position information of each object in the world coordinate system. For example, the position information may be distance information, orientation information, height information, speed information, posture information, etc.

[0075] Specifically, the robot can fuse the calibration information of each object in the captured image with the virtual point cloud data, map the virtual point cloud data into the detection frame of each object in the captured image, and obtain the mapping relationship between the virtual point cloud data and each object. According to the correspondence between the virtual point cloud data and the actual point cloud data, and the mapping relationship between the virtual point cloud data and each object, the correspondence between the actual point cloud data and each object can be obtained. According to the correspondence and the actual point cloud data, the position information of each object in the captured image can be obtained.

[0076] It is understandable that when each object is not in the blind spot of the RGB camera, the position information of each object can be determined based on the calibration information of each object and the actual point cloud data collected by the lidar, as well as the correspondence between the actual point cloud data and each object.

[0077] In the above positioning method, the calibration information of each object in the captured image is determined according to the captured image obtained by the camera set on the robot, the virtual point cloud data corresponding to the actual point cloud data is obtained according to the actual point cloud data obtained by the laser radar set on the robot, the position of the camera and the position of the laser radar, and the position information of each object is determined according to the calibration information of each object and the virtual point cloud data. By determining the calibration information of each object in the captured image, the method can accurately distinguish each object in the captured image, and obtain the virtual point cloud data corresponding to the actual point cloud data according to the actual point cloud data, the position of the camera and the position of the laser radar, thereby avoiding the problem that the actual point cloud data cannot be matched with the captured image due to the height difference between the camera and the laser radar. The method determines the actual point cloud data corresponding to each object through the correspondence between the obtained virtual point cloud data and the calibration information of each object, and can accurately obtain the position information of each object in the area in front of the robot, eliminating the blind spot of the camera shooting and improving the robot's perception ability of close-range objects.

[0078] Figure 5 A flowchart of a positioning method provided in an embodiment of the present application. The embodiment of the present application relates to an optional implementation method for determining the position information of each object based on the calibration information and virtual point cloud data of each object. Figure 2 Based on the embodiment shown, Figure 5 As shown, the above S203 may include the following steps:

[0079] S501, matching the virtual point cloud data with the calibration information of each object respectively, and determining the target object corresponding to the virtual point cloud data.

[0080] Specifically, the robot can project the virtual point cloud data onto the captured image, calculate the proportion of the virtual point cloud data in the detection frame of each object, and determine the object corresponding to the detection frame with the largest proportion as the target object corresponding to the virtual point cloud data. Alternatively, the proportion of the virtual point cloud data in the detection frame of each object can be compared with a preset threshold, and the object corresponding to the detection frame with a proportion greater than the preset threshold can be determined as the target object corresponding to the virtual point cloud data. The target object can correspond to one virtual point cloud or multiple virtual point clouds.

[0081] S502: Determine the position information of the target object corresponding to the virtual point cloud data according to the actual point cloud data corresponding to the virtual point cloud data.

[0082] Specifically, the robot obtains the correspondence between the virtual point cloud and the target object through step S501, and then obtains the correspondence between the actual point cloud data and the target object based on the correspondence between the actual point cloud data and the virtual point cloud data. By analyzing the actual point cloud data, the position information of the target object corresponding to the actual point cloud data is obtained, and thus the position information of the target object corresponding to the virtual point cloud data can be obtained.

[0083] In the above positioning method, the virtual point cloud data is matched with the calibration information of each object respectively to determine the target object corresponding to the virtual point cloud data, and the position information of the target object corresponding to the virtual point cloud data is determined according to the actual point cloud data corresponding to the virtual point cloud data. According to the matching result of the virtual point cloud data and the calibration information of each object, the method can accurately determine the target object corresponding to the virtual point cloud data, thereby determining the target object corresponding to the actual point cloud data, and then accurately obtaining the position information of the target object according to the actual point cloud data.

[0084] Figure 6 A flowchart of a positioning method provided in an embodiment of the present application. The present application embodiment relates to an optional implementation method in which the calibration information includes a detection frame, and the virtual point cloud data is matched with the calibration information of each object to determine the target object corresponding to the virtual point cloud data. Figure 5 Based on the embodiment shown, Figure 6 As shown, the above S501 may include the following steps:

[0085] S601, projecting the virtual point cloud data onto the captured image, and determining the proportion of the virtual point cloud data in the detection frame of each object.

[0086] Specifically, the robot can obtain the calibration parameters of the camera and virtual lidar, and obtain the projection matrix of the virtual point cloud data to the captured image based on the camera's intrinsic parameter matrix, the rotation matrix from the camera to the captured image plane, and the extrinsic parameter matrix from the virtual point cloud to the camera. The projection matrix is ​​used to project the virtual point cloud data onto the captured image, and the proportion of each virtual point cloud data in each object detection frame is calculated.

[0087] S602: Determine the target object corresponding to the virtual point cloud data according to the proportion.

[0088] Optionally, the robot may arrange the proportion of a virtual point cloud data in the detection frame of each object in order from large to small, determine the object corresponding to the first detection frame as the target object corresponding to the virtual point cloud data, and obtain the target object corresponding to each virtual point cloud data in the above manner. Optionally, the robot may compare the proportion of a virtual point cloud data in the detection frame of each object with a preset threshold, determine the object corresponding to the detection frame whose proportion is greater than the preset threshold as the target object corresponding to the virtual point cloud data, or calculate the difference between the proportion of the virtual point cloud data in the detection frame of each object and the preset threshold, compare the difference with the preset difference, and determine the object corresponding to the detection frame whose difference is less than the preset difference as the target object corresponding to the virtual point cloud data. This embodiment does not limit the method of determining the target object corresponding to the virtual point cloud data based on the proportion.

[0089] Optionally, in another embodiment, the embodiment of the present application relates to an optional implementation method of determining the target object corresponding to the virtual point cloud data according to the proportion. The above S602 may include the following steps: determining the target object as the object corresponding to the detection box whose proportion is greater than a preset threshold; or determining the target object as the object corresponding to the detection box with the highest proportion.

[0090] Specifically, the preset threshold is obtained through historical experience. When a virtual point cloud data corresponds to a detection frame, the object corresponding to the detection frame with the highest proportion of virtual point cloud data in each object detection frame is determined as the target object of the virtual point cloud data; when two or more virtual point cloud data correspond to an object detection frame, the object of the detection frame is determined as the target object corresponding to the virtual point cloud data with a proportion greater than the preset threshold.

[0091] Further, it can be understood that if the proportion of virtual point cloud data in the detection frame of each object is greater than a preset threshold, or the proportion of virtual point cloud data in the detection frame of each object is the highest, then it is determined that all or most of the virtual point cloud data falls within the detection frame, that is, the degree of match between the virtual point cloud data and the object corresponding to the detection frame is the highest. Therefore, the object corresponding to the detection frame greater than the preset threshold or the object corresponding to the detection frame with the highest proportion is determined as the target object corresponding to the virtual point cloud data.

[0092] In the above positioning method, the object corresponding to the detection frame whose proportion is greater than the preset threshold is determined as the target object; or the object corresponding to the detection frame with the highest proportion is determined as the target object. This method uses different methods for analysis according to different situations. By comparing the proportion with the preset threshold, the object corresponding to the detection frame greater than the preset threshold is obtained, or the object corresponding to the detection frame with the highest proportion is obtained. The target object corresponding to the virtual point cloud image can be accurately obtained, making the process of obtaining the target object corresponding to the virtual point cloud more flexible, accurate, and more adaptable to the scene.

[0093] Furthermore, in the above positioning method, the virtual point cloud data is projected onto the captured image, the proportion of the virtual point cloud data in the detection frame of each object is determined, and the target object corresponding to the virtual point cloud data is determined according to the proportion. This method can accurately match the virtual point cloud data with the detection frame of each object by the proportion of the virtual point cloud data in the detection frame of each object, so that the target object corresponding to the virtual point cloud data can be obtained more accurately.

[0094] Figure 7 A flowchart of a positioning method provided in an embodiment of the present application. The embodiment of the present application relates to an optional implementation method for determining the position information of a target object corresponding to virtual point cloud data based on actual point cloud data of the virtual point cloud data. Figure 5 Based on the embodiment shown, Figure 7 As shown, the above S502 may include the following steps:

[0095] S701, obtaining the centroid of the actual point cloud data.

[0096] Specifically, the centroid of the actual point cloud data is a point in the point cloud data, and the coordinates of the centroid are obtained by calculating the average value of all the point clouds in the actual point cloud data. The robot can obtain the centroid of the actual point cloud data according to the centroid calculation formula. Among them, the calculation formula of the point cloud centroid coordinates can be expressed as:

[0097]

[0098] Among them, x i ,y i , z i is the coordinate of each point cloud data in the actual point cloud data.

[0099] S702: Determine the position of the centroid as the position information of the target object.

[0100] Specifically, after the robot obtains the centroid position of the actual point cloud data through step S701, the centroid position of the target object is the position of the target object, and the centroid position of the actual point cloud data is directly determined as the position information of the target object.

[0101] In the above positioning method, the center of mass of the target object is obtained according to the actual point cloud data of the virtual point cloud data, and the position of the center of mass is determined as the position information of the target object. This method can accurately obtain the center of mass coordinate position of the actual point cloud data through the actual point cloud data, so that the position information of the target object can be determined more accurately.

[0102] Figure 8 A flowchart of a positioning method provided in an embodiment of the present application. The embodiment of the present application involves that the position of the camera includes the height of the camera, and the position of the laser radar includes the height of the laser radar; an optional implementation method of obtaining virtual point cloud data corresponding to the actual point cloud data based on the actual point cloud data obtained by the laser radar set on the robot, the position of the camera and the position of the laser radar. Figure 2 Based on the embodiment shown, Figure 8 As shown, the above S202 may include the following steps:

[0103] S801, creating a virtual laser radar according to the height of the camera and the height of the laser radar.

[0104] Specifically, the robot can determine the internal and external parameters of the virtual laser radar based on the internal and external parameters of the laser radar. The internal parameters include the resolution and field of view of the laser radar, and the external parameters include the position information and angle information of the laser radar. The height of the virtual laser radar is consistent with the height of the camera, and the other parameters are consistent with the parameters of the laser radar.

[0105] S802, obtaining virtual point cloud data according to the actual point cloud data and the virtual laser radar.

[0106] Specifically, the robot can use the parameters of the virtual laser radar to determine the virtual point cloud plane of the virtual laser radar, project the actual point cloud data acquired by the laser radar onto the virtual point cloud plane, and obtain virtual point cloud data corresponding to the actual point cloud data.

[0107] In the above positioning method, a virtual laser radar is created according to the height of the camera and the height of the laser radar, and virtual point cloud data is obtained according to the actual point cloud data and the virtual laser radar. By creating a virtual laser radar that is consistent with the height of the camera, this method can better integrate the virtual point cloud data corresponding to the virtual laser radar with the image taken by the camera, so that the target object corresponding to the virtual point cloud data can be obtained more accurately.

[0108] Fig. 9 A flowchart of a positioning method provided in an embodiment of the present application. The embodiment of the present application relates to an optional implementation method for filtering actual point cloud data. Figure 2 Based on the embodiment shown, Fig. 9 As shown, the above method may further include the following steps:

[0109] S901, filtering out redundant point clouds in actual point cloud data to obtain filtered point cloud data; the redundant point clouds include other point clouds except the object.

[0110] Specifically, redundant point clouds may include discrete point clouds, point clouds of obstacles such as walls and counters, etc. The robot may filter out discrete point clouds in the actual point cloud data through commonly used filters, and then filter out straight point clouds of obstacles such as walls and counters through a straight line detection algorithm to obtain filtered point cloud data. Commonly used filters include pass-through filters, voxel filters, statistical filters, and radius filters. Straight line detection algorithms include Hough straight line detection and Line Segment Detector (LSD) algorithms.

[0111] Optionally, according to the actual point cloud data acquired by the laser radar set on the robot, the position of the camera and the position of the laser radar, virtual point cloud data corresponding to the actual point cloud data is acquired, including:

[0112] S902, obtaining virtual point cloud data corresponding to the filtered point cloud data according to the filtered point cloud data, the position of the camera and the position of the laser radar.

[0113] Optionally, the server can obtain virtual point cloud data at the same height as the camera based on the filtered point cloud data, and obtain virtual point cloud data corresponding to the filtered point cloud data. Alternatively, the server can determine the highest position of the shooting blind spot of the object closest to the robot based on the calibration information of each object in the captured image, and obtain virtual point cloud data at the highest position of the shooting blind spot between the camera and the laser radar based on the filtered point cloud data, and obtain virtual point cloud data corresponding to the filtered point cloud data. Alternatively, based on the filtered point cloud data, obtain virtual point cloud data at any height between the highest position of the shooting blind spot and the camera installation position, and obtain virtual point cloud data corresponding to the filtered point cloud data.

[0114] In the above positioning method, redundant point clouds in the actual point cloud data are filtered out to obtain the filtered point cloud data, and virtual point cloud data corresponding to the filtered point cloud data is obtained according to the filtered point cloud data, the position of the camera and the position of the laser radar. Among them, the redundant point cloud includes other point clouds except the object. The redundant point cloud data is filtered out to obtain the virtual point cloud data corresponding to the filtered point cloud data, which can avoid the influence of the redundant point cloud data on the virtual point cloud data.

[0115] Fig.10A flowchart of a positioning method provided in an embodiment of the present application. The embodiment of the present application relates to an optional implementation method of filtering out redundant point clouds in actual point cloud data to obtain filtered point cloud data. Fig. 9 Based on the embodiment shown, Fig.10 As shown, the above S901 may include the following steps:

[0116] S1001, obtaining the point cloud shape of the object in the actual point cloud data.

[0117] Specifically, taking the object as a pedestrian, the shape of the foot is approximately an arc, and the robot can perform preset processing on the actual point cloud data to obtain a top view of the actual point cloud data, in which the shape of the pedestrian's footsteps is a top view. For example, when the object is a cylindrical obstacle, in the top view of the actual point cloud data, the point cloud shape of the cylindrical obstacle is a circle.

[0118] S1002, filtering out other point clouds except the point cloud shapes of each object in the actual point cloud data to obtain filtered point cloud data.

[0119] Specifically, after determining the point cloud image of the object in step S1001, the robot can filter out other point clouds except the point cloud shape of each object through a filter and a straight line detection algorithm to obtain filtered point cloud data.

[0120] For example, Fig.11 Represents a schematic diagram of actual point cloud data. Fig.12 The schematic diagram of the point cloud data after filtering is shown in Fig.11 In the figure, since the shape of pedestrians’ feet is approximately circular, and the point clouds of obstacles such as walls and counters are distributed in straight lines, the robot can use the straight line detection algorithm to filter out the straight line point clouds in the actual point cloud data. The point cloud data after filtering is the shape of human feet. Fig.12 In the figure, the point cloud data after filtering is arc-shaped point cloud data.

[0121] In the above positioning method, a top view of the actual point cloud data is obtained, the point cloud shape of the object in the top view is determined, and other point clouds except the point cloud shape of each object in the actual point cloud data are filtered out to obtain the filtered point cloud data. This method can more accurately obtain the point cloud shape of the object based on the top view of the actual point cloud data, so that other point cloud shapes except the point cloud shape can be accurately filtered out, and virtual point cloud data can be more accurately obtained based on the filtered point cloud data.

[0122] Fig.13 A flowchart of a positioning method provided in an embodiment of the present application. The embodiment of the present application relates to an optional implementation method of clustering the filtered point cloud data. Fig. 9 Based on the embodiment shown, Fig.13 As shown, the above method may further include the following steps:

[0123] S1301, clustering the filtered point cloud data to obtain multiple point cloud clusters.

[0124] Specifically, the robot can cluster the filtered point cloud data through a commonly used clustering algorithm, and classify the point cloud data belonging to the same category into a point cloud cluster, and multiple point cloud clusters can be obtained. Among them, commonly used clustering algorithms can include density-based spatial clustering algorithm (Density-Based Spatial Clustering of Applications with Noise, DBSCAN), principal component analysis algorithm (Principal Components Analysis, PCA), K-means clustering algorithm (K-means Clustering Algorithm, K-means), Gaussian Mixed Model (Gaussian Mixed Model, GMM), spectral clustering algorithm, etc. For example, Fig.14 A schematic diagram showing the clustered point cloud clusters. After clustering the filtered point cloud data, two point cloud clusters are obtained.

[0125] Optionally, according to the filtered point cloud data, the position of the camera, and the position of the laser radar, virtual point cloud data corresponding to the filtered point cloud data is obtained, including:

[0126] S1302, obtaining virtual point cloud data corresponding to each point cloud cluster according to the position of each point cloud cluster, the camera and the position of the laser radar.

[0127] Among them, one virtual point cloud data may correspond to one point cloud cluster or multiple point cloud clusters.

[0128] Optionally, the server can obtain virtual point cloud data at the same height as the camera for each point cloud cluster, and obtain virtual point cloud data corresponding to each point cloud cluster. Alternatively, the server can determine the highest position of the shooting blind spot of the object closest to the robot based on the calibration information of each object in the captured image, and obtain virtual point cloud data at the highest position of the shooting blind spot between the camera and the laser radar based on the filtered point cloud data, and obtain virtual point cloud data corresponding to each point cloud cluster. Alternatively, according to each point cloud cluster, obtain virtual point cloud data at any height between the highest position of the shooting blind spot and the camera installation position, and obtain virtual point cloud data corresponding to each point cloud cluster.

[0129] In the above positioning method, the filtered point cloud data is clustered to obtain multiple point cloud clusters, and the virtual point cloud data corresponding to each point cloud cluster is obtained according to the position of each point cloud cluster, the camera and the position of the laser radar. This method clusters the filtered point cloud data and clusters the point cloud data of the same type into the same cluster of point cloud data, so that the virtual point cloud data corresponding to each point cloud cluster can be obtained more accurately.

[0130] In one embodiment, in order to facilitate understanding by those skilled in the art, the positioning method is described in detail below. Fig.15 As shown, the method may include:

[0131] S1501, determining calibration information of each object in the captured image according to the captured image acquired by the camera disposed on the robot;

[0132] S1502, creating a virtual laser radar according to the height of the camera and the height of the laser radar;

[0133] S1503, obtaining a top view of actual point cloud data, and determining a point cloud shape of the object in the top view;

[0134] S1504, filtering out other point clouds except the point cloud shapes of each object in the actual point cloud data to obtain filtered point cloud data;

[0135] S1505, clustering the filtered point cloud data to obtain multiple point cloud clusters;

[0136] S1506, acquiring virtual point cloud data corresponding to each point cloud cluster according to the position of each point cloud cluster, the camera, and the position of the laser radar;

[0137] S1507, projecting the virtual point cloud data onto the captured image, and determining the proportion of the virtual point cloud data in the detection frame of each object;

[0138] S1508, determining the target object from the objects corresponding to the detection boxes whose proportion is greater than a preset threshold; or determining the target object from the objects corresponding to the detection boxes with the highest proportion;

[0139] S1509, obtaining the centroid of the actual point cloud data;

[0140] S1510: Determine the position of the centroid as the position information of the target object.

[0141] It should be noted that for the description in the above S1501-S1510, reference can be made to the relevant description in the above embodiment, and the effects are similar, so this embodiment will not be repeated here.

[0142] Furthermore, it is understandable that Fig.16Figure 3 is a schematic diagram of the projection of virtual point cloud data on the captured image. Pedestrian 1 is far away from the robot and is not in the blind spot of the camera. Both the actual point cloud data and the virtual point cloud data can be projected into the captured image. Pedestrian 2 is close to the robot and the position below the knee of pedestrian 2 is in the blind spot of the camera. The actual point cloud data of pedestrian 2 cannot be projected into the captured image, but the virtual point cloud data of pedestrian 2 and pedestrian 1 can be projected into the captured image.

[0143] In the above positioning method, by determining the calibration information of each object in the captured image, the objects in the captured image can be accurately distinguished, and virtual point cloud data corresponding to the actual point cloud data can be obtained according to the actual point cloud data, the position of the camera and the position of the laser radar. By calculating the proportion of virtual point cloud data in the detection frame of each object, and by using different methods for analysis according to different situations, by comparing the proportion with a preset threshold, the object corresponding to the detection frame greater than the preset threshold is obtained, or the object corresponding to the detection frame with the highest proportion is obtained, the target object corresponding to the virtual point cloud image can be accurately obtained, so that the target object corresponding to the virtual point cloud data can be more accurately obtained, avoiding the problem that the actual point cloud data cannot be matched with the captured image due to the height difference between the camera and the laser radar, and determining the center of mass of the target object through the actual point cloud data. The position information of the target object can be determined more accurately, thereby improving the robot's perception of close objects.

[0144] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0145] Based on the same inventive concept, the embodiment of the present application also provides a positioning device for implementing the positioning method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more positioning device embodiments provided below can refer to the limitations of the positioning method above, and will not be repeated here.

[0146] In one embodiment, Fig.17 As shown, a positioning device is provided, comprising: a first determination module 11, a first acquisition module 12 and a second determination module 13, wherein:

[0147] A first determination module 11 is used to determine the calibration information of each object in the captured image according to the captured image acquired by the camera disposed on the robot;

[0148] A first acquisition module 12 is used to acquire virtual point cloud data corresponding to the actual point cloud data according to the actual point cloud data acquired by the laser radar set on the robot, the position of the camera and the position of the laser radar;

[0149] The second determination module 13 is used to determine the position information of each object according to the calibration information of each object and the virtual point cloud data.

[0150] The positioning device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0151] In one embodiment, the second determination module includes: a first determination unit and a second determination unit, wherein:

[0152] A first determination unit is used to match the virtual point cloud data with the calibration information of each object respectively, and determine the target object corresponding to the virtual point cloud data;

[0153] The second determining unit is used to determine the position information of the target object corresponding to the virtual point cloud data according to the actual point cloud data of the virtual point cloud data.

[0154] The positioning device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0155] Optionally, the first determining unit is specifically used to project the virtual point cloud data onto the captured image to determine the proportion of the virtual point cloud data in the detection frame of each object;

[0156] The target object corresponding to the virtual point cloud data is determined according to the proportion.

[0157] The positioning device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0158] Optionally, the first determination unit is specifically configured to determine the target object by the object corresponding to the detection frame having a proportion greater than a preset threshold; or, determine the target object by the object corresponding to the detection frame having the highest proportion.

[0159] The positioning device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0160] Optionally, the second determination unit is specifically configured to obtain the centroid of the target object based on the actual point cloud data of the virtual point cloud data; and determine the position of the centroid as the position information of the target object.

[0161] The positioning device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0162] In one embodiment, the first acquisition module includes: a creation unit and an acquisition unit, wherein:

[0163] A creation unit is used to create a virtual lidar according to the height of the camera and the height of the lidar;

[0164] The acquisition unit is used to acquire virtual point cloud data according to actual point cloud data and virtual laser radar.

[0165] The positioning device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0166] In one embodiment, the positioning device further includes: a filtering module and a second acquisition module, wherein:

[0167] A filtering module is used to filter out redundant point clouds in actual point cloud data to obtain filtered point cloud data; the redundant point clouds include other point clouds except the object;

[0168] The second acquisition module is used to acquire virtual point cloud data corresponding to the filtered point cloud data according to the filtered point cloud data, the position of the camera and the position of the laser radar.

[0169] The positioning device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0170] In one embodiment, the filtering module includes: a third determining unit and a filtering unit, wherein:

[0171] A third determining unit is used to obtain a top view of the actual point cloud data and determine a point cloud shape of the object in the top view;

[0172] The filtering unit is used to filter out other point clouds except the point cloud shapes of each object in the actual point cloud data to obtain the filtered point cloud data.

[0173] The positioning device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0174] In one embodiment, the positioning device further includes a third acquisition module, wherein:

[0175] The third acquisition module acquires virtual point cloud data corresponding to each point cloud cluster according to the position of each point cloud cluster, the camera and the position of the laser radar.

[0176] The positioning device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0177] Each module in the above positioning device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of the processor in the robot in hardware form, or can be stored in the memory in the robot in software form, so that the processor can call and execute the corresponding operations of each module.

[0178] In one embodiment, a robot is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, any one of the contents in the above method embodiments is implemented.

[0179] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any one of the contents in the above method embodiments is implemented.

[0180] In one embodiment, a computer program product is provided, including a computer program, which implements any one of the above method embodiments when executed by a processor.

[0181] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0182] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0183] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0184] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A positioning method, characterized in that: Applied to a robot, the method comprises: Determining calibration information of each object in a captured image according to a captured image acquired by a camera disposed on the robot; the calibration information includes a detection frame; According to actual point cloud data acquired by the laser radar arranged on the robot, the position of the camera and the position of the laser radar, virtual point cloud data corresponding to the actual point cloud data is acquired; the virtual point cloud data is point cloud data at the same height as the camera; Projecting the virtual point cloud data onto the captured image, and determining a proportion of the virtual point cloud data in a detection frame of each of the objects; Determine the target object corresponding to the virtual point cloud data according to the proportion; According to the actual point cloud data corresponding to the virtual point cloud data, the position information of the target object corresponding to the virtual point cloud data is determined.

2. The method according to claim 1, characterized in that The step of determining the target object corresponding to the virtual point cloud data according to the proportion includes: The object corresponding to the detection frame whose proportion is greater than a preset threshold is determined as the target object; or, The object corresponding to the detection frame with the highest proportion is determined as the target object.

3. The method according to claim 1, characterized in that The step of determining the position information of the target object corresponding to the virtual point cloud data according to the actual point cloud data of the virtual point cloud data comprises: Obtaining the centroid of the actual point cloud data; The position of the centroid is determined as the position information of the target object.

4. The method according to any one of claims 1 to 3, characterized in that: The position of the camera includes the height of the camera, and the position of the laser radar includes the height of the laser radar; and obtaining virtual point cloud data corresponding to the actual point cloud data according to the actual point cloud data obtained by the laser radar disposed on the robot, the position of the camera, and the position of the laser radar, comprises: Creating a virtual laser radar according to the height of the camera and the height of the laser radar; The virtual point cloud data is acquired according to the actual point cloud data and the virtual laser radar.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Filtering out redundant point clouds in the actual point cloud data to obtain filtered point cloud data; the redundant point clouds include other point clouds except the object; The step of obtaining virtual point cloud data corresponding to the actual point cloud data according to the actual point cloud data obtained by the laser radar disposed on the robot, the position of the camera, and the position of the laser radar comprises: According to the filtered point cloud data, the position of the camera and the position of the laser radar, virtual point cloud data corresponding to the filtered point cloud data is obtained.

6. The method according to claim 5, characterized in that The filtering out redundant point clouds in the actual point cloud data to obtain filtered point cloud data includes: Acquire the point cloud shape of the object in the actual point cloud data; The other point clouds except the point cloud shapes of the objects in the actual point cloud data are filtered out to obtain the filtered point cloud data.

7. The method according to claim 5, characterized in that The method further comprises: Clustering the filtered point cloud data to obtain multiple point cloud clusters; The step of obtaining virtual point cloud data corresponding to the filtered point cloud data according to the filtered point cloud data, the position of the camera, and the position of the laser radar includes: According to each of the point cloud clusters, the position of the camera and the position of the laser radar, virtual point cloud data corresponding to each of the point cloud clusters is obtained.

8. A positioning device, characterized in that: Applied to a robot, the device comprises: A first determination module is used to determine the calibration information of each object in the captured image according to the captured image acquired by the camera disposed on the robot; the calibration information includes a detection frame; A first acquisition module is used to acquire virtual point cloud data corresponding to the actual point cloud data according to the actual point cloud data acquired by the laser radar set on the robot, the position of the camera and the position of the laser radar; the virtual point cloud data is point cloud data at the same height as the camera; The second determination module is used to project the virtual point cloud data onto the captured image to determine the proportion of the virtual point cloud data in the detection frame of each object; determine the target object corresponding to the virtual point cloud data based on the proportion; and determine the position information of the target object corresponding to the virtual point cloud data based on the actual point cloud data corresponding to the virtual point cloud data.

9. A robot comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images