A control method and apparatus of a mobile robot

Abstract

Embodiments of the present application provide a control method and device of a mobile robot, and relate to the technical field of robot vision. The method comprises: acquiring a two-dimensional image containing a specified object in an advancing direction collected by a camera, and a three-dimensional point cloud image of a specified plane parallel to a driving plane collected by a radar; performing instance segmentation on the two-dimensional image to obtain a specified image region occupied by the specified object in the two-dimensional image; determining specified three-dimensional points belonging to the specified object in the three-dimensional point cloud image according to a conversion relationship between a radar coordinate system corresponding to the three-dimensional point cloud image and an image coordinate system corresponding to the two-dimensional image; calculating a size of a space below the specified object according to a distance between the specified three-dimensional points and the radar and image coordinates of the specified image region in the image coordinate system; and controlling the mobile robot to pass through the space below the specified object or not in combination with a size of the mobile robot. The intelligence of the mobile robot can be improved.

Description

Technical Field

[0001] This application relates to the field of robot vision technology, and in particular to a control method and apparatus for a mobile robot. Background Technology

[0002] In logistics, manufacturing, and other industries, mobile robots (e.g., Automated Guided Vehicles (AGVs)) can be used to lift and move goods. Mobile robots can be equipped with electromagnetic or optical automatic guidance devices, enabling them to travel along a predetermined guidance path.

[0003] If there are obstacles with supporting components (such as tables, shelves, etc.) in the direction of a mobile robot's movement, the relevant technologies do not provide corresponding methods to determine whether the mobile robot can pass through the space under the obstacle, resulting in low intelligence of the mobile robot. Summary of the Invention

[0004] The purpose of this application is to provide a control method and apparatus for a mobile robot to improve the intelligence of the mobile robot. The specific technical solution is as follows:

[0005] Firstly, in order to achieve the above objectives, embodiments of this application disclose a control method for a mobile robot, the method comprising:

[0006] Acquire a two-dimensional image of a specified object captured by a camera in the forward direction of the mobile robot, and a three-dimensional point cloud map of a specified plane parallel to the driving plane captured by a radar in the mobile robot.

[0007] The two-dimensional image is segmented to obtain the image region occupied by the specified object in the two-dimensional image, which is used as the specified image region;

[0008] Based on the specified image region and the transformation relationship between the radar coordinate system corresponding to the 3D point cloud map and the image coordinate system corresponding to the 2D image, the 3D points in the 3D point cloud map belonging to the specified object are determined as the specified 3D points;

[0009] The size of the space below the specified object is calculated based on the distance between the specified three-dimensional point and the radar, and the image coordinates of the specified image region in the image coordinate system.

[0010] Based on the size of the space below the specified object and the size of the mobile robot, control whether the mobile robot passes through the space below the specified object.

[0011] Optionally, determining the three-dimensional points belonging to the specified object in the three-dimensional point cloud map as specified three-dimensional points based on the specified image region and the transformation relationship between the radar coordinate system corresponding to the three-dimensional point cloud map and the image coordinate system corresponding to the two-dimensional image includes:

[0012] According to the transformation relationship between the radar coordinate system corresponding to the three-dimensional point cloud map and the image coordinate system corresponding to the two-dimensional image, each three-dimensional point in the three-dimensional point cloud map is projected to the image coordinate system to obtain the corresponding projected pixel point.

[0013] The three-dimensional points corresponding to the projected pixels belonging to the specified image region are determined as the specified three-dimensional points belonging to the specified object in the three-dimensional point cloud map.

[0014] Optionally, calculating the size of the space below the specified object based on the distance between the specified three-dimensional point and the radar, and the image coordinates of the specified image region in the image coordinate system, includes:

[0015] Based on the distance between the specified three-dimensional point and the radar, and the image coordinates of the upper part of the specified object in the image coordinate system, calculate the height of the space below the specified object;

[0016] Based on the image coordinates of the left and right sides of the specified object in the image coordinate system, and the transformation relationship between the camera coordinate system and the image coordinate system, the distance between the left and right sides of the specified object is calculated as the width of the space below the specified object.

[0017] Optionally, calculating the height of the space below the specified object based on the distance between the specified three-dimensional point and the radar, and the image coordinates of the upper part of the specified object in the image coordinate system, includes:

[0018] The distance between the specified three-dimensional point and the radar is calculated based on the distance between the specified three-dimensional point and the radar, and the positional relationship between the camera and the radar.

[0019] Based on the distance between the specified 3D point and the camera, and the image coordinates of the upper part of the specified object in the image coordinate system, calculate the height of the upper part of the specified object in the camera coordinate system.

[0020] The height of the space below the specified object is calculated based on the height of the upper part of the specified object in the camera coordinate system and the installation height of the camera.

[0021] Optionally, controlling whether the mobile robot passes through the space below the specified object based on the size of the space below the specified object and the size of the mobile robot includes:

[0022] If the size of the space below the specified object is not less than the size of the mobile robot, then control the mobile robot to pass through the space below the specified object;

[0023] If the size of the space below the specified object is smaller than the size of the mobile robot, then the mobile robot is controlled to move around the left and right sides of the specified object.

[0024] Secondly, in order to achieve the above objectives, embodiments of this application disclose a control device for a mobile robot, the device comprising:

[0025] The image data acquisition module is used to acquire a two-dimensional image containing a specified object in the forward direction captured by the camera in the mobile robot, and a three-dimensional point cloud map of a specified plane parallel to the driving plane captured by the radar in the mobile robot.

[0026] The specified image region determination module is used to perform instance segmentation on the two-dimensional image to obtain the image region occupied by the specified object in the two-dimensional image, which is used as the specified image region.

[0027] The specified 3D point determination module is used to determine, based on the specified image region and the transformation relationship between the radar coordinate system corresponding to the 3D point cloud map and the image coordinate system corresponding to the 2D image, the 3D points belonging to the specified object in the 3D point cloud map, and to designate the 3D points as specified 3D points.

[0028] The spatial calculation module is used to calculate the size of the space below the specified object based on the distance between the specified three-dimensional point and the radar, and the image coordinates of the specified image region in the image coordinate system.

[0029] The control module is used to control whether the mobile robot passes through the space below the specified object based on the size of the space below the specified object and the size of the mobile robot.

[0030] Optionally, the specified three-dimensional point determination module is specifically used to project each three-dimensional point in the three-dimensional point cloud map to the image coordinate system according to the transformation relationship between the radar coordinate system corresponding to the three-dimensional point cloud map and the image coordinate system corresponding to the two-dimensional image, so as to obtain the corresponding projected pixel point.

[0031] The three-dimensional points corresponding to the projected pixels belonging to the specified image region are determined as the specified three-dimensional points belonging to the specified object in the three-dimensional point cloud map.

[0032] Optionally, the spatial computing module includes:

[0033] The height calculation submodule is used to calculate the height of the space below the specified object based on the distance between the specified three-dimensional point and the radar, and the image coordinates of the upper part of the specified object in the image coordinate system;

[0034] The width calculation submodule is used to calculate the distance between the left and right sides of the specified object as the width of the space below the specified object, based on the image coordinates of the left and right sides of the specified object in the image coordinate system and the transformation relationship between the camera coordinate system and the image coordinate system.

[0035] Optionally, the height calculation submodule is specifically used to calculate the distance between the specified three-dimensional point and the camera based on the distance between the specified three-dimensional point and the radar, and the positional relationship between the camera and the radar;

[0036] Based on the distance between the specified 3D point and the camera, and the image coordinates of the upper part of the specified object in the image coordinate system, calculate the height of the upper part of the specified object in the camera coordinate system.

[0037] The height of the space below the specified object is calculated based on the height of the upper part of the specified object in the camera coordinate system and the installation height of the camera.

[0038] Optionally, the control module is specifically used to control the mobile robot to pass through the space below the specified object if the size of the space below the specified object is not less than the size of the mobile robot;

[0039] If the size of the space below the specified object is smaller than the size of the mobile robot, then the mobile robot is controlled to move around the left and right sides of the specified object.

[0040] In another aspect of this application, in order to achieve the above objectives, embodiments of this application also disclose a mobile robot, including a camera, radar, and a processor;

[0041] The camera is used to acquire two-dimensional images in the forward direction;

[0042] The radar is used to acquire a three-dimensional point cloud map of a designated plane parallel to the driving plane;

[0043] The processor is used to execute any of the control methods for the mobile robot described above.

[0044] In another aspect of this application, in order to achieve the above objectives, embodiments of this application also disclose a control device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0045] The memory is used to store computer programs;

[0046] The processor, when executing the program stored in the memory, implements any of the aforementioned control methods for the mobile robot.

[0047] In another aspect of this application, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the control method for the mobile robot as described above.

[0048] This application also provides a computer program product containing instructions that, when run on a computer, causes the computer to execute any of the above-described control methods for a mobile robot.

[0049] Beneficial effects of the embodiments in this application:

[0050] The mobile robot control method provided in this application acquires a two-dimensional image of a specified object captured by a camera in the forward direction of the mobile robot, and a three-dimensional point cloud map of a specified plane parallel to the driving plane captured by a radar in the mobile robot; performs instance segmentation on the two-dimensional image to obtain the image region occupied by the specified object in the two-dimensional image, which is designated as the specified image region; determines the three-dimensional points belonging to the specified object in the three-dimensional point cloud map as designated three-dimensional points based on the designated image region and the transformation relationship between the radar coordinate system corresponding to the three-dimensional point cloud map and the image coordinate system corresponding to the two-dimensional image; calculates the size of the space below the specified object based on the distance between the designated three-dimensional point and the radar, and the image coordinates of the specified image region in the image coordinate system; and controls whether the mobile robot passes through the space below the specified object based on the size of the space below the specified object and the size of the mobile robot.

[0051] Based on the above processing, by combining the two-dimensional image and the three-dimensional point cloud map, the size of the space below the designated object in the direction of the mobile robot's movement is determined. Combined with the size of the mobile robot, it is possible to control whether the mobile robot passes through the space below the designated object, thereby improving the intelligence of the mobile robot. Furthermore, the method provided in this application only requires acquiring a two-dimensional image and a three-dimensional point cloud map of a designated plane parallel to the driving plane. This means that the camera in the mobile robot can be a monocular camera, and the radar can be a single-line radar, thus reducing the production cost of the mobile robot and increasing the applicability of the method.

[0052] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0054] Figure 1 A flowchart illustrating a control method for a mobile robot provided in an embodiment of this application;

[0055] Figure 2 This is a schematic diagram of a mobile robot driving scenario provided in an embodiment of this application;

[0056] Figure 3 A schematic diagram of a coordinate system provided for an embodiment of this application;

[0057] Figure 4 A flowchart illustrating another mobile robot control method provided in this application embodiment;

[0058] Figure 5 A schematic diagram of three-dimensional points in a specified object provided in an embodiment of this application;

[0059] Figure 6 A flowchart illustrating another mobile robot control method provided in this application embodiment;

[0060] Figure 7 A schematic diagram illustrating the calculation of the height of the space below a specified object, provided as an embodiment of this application;

[0061] Figure 8 A control flowchart for a mobile robot provided in an embodiment of this application;

[0062] Figure 9 A structural diagram of a control device for a mobile robot provided in an embodiment of this application;

[0063] Figure 10 A structural diagram of a mobile robot provided in an embodiment of this application;

[0064] Figure 11 This is a structural diagram of a control device provided in an embodiment of this application. Detailed Implementation

[0065] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0066] If there are obstacles with supporting components (such as tables, shelves, etc.) in the direction of a mobile robot's movement, the relevant technologies do not provide corresponding methods to determine whether the mobile robot can pass through the space under the obstacle, resulting in low intelligence of the mobile robot.

[0067] To address the aforementioned issues, this application provides a control method for a mobile robot. This method can be applied to a control device, for example, the control device can be integrated inside the mobile robot to plan the mobile robot's trajectory.

[0068] See Figure 1 , Figure 1 A flowchart of a control method for a mobile robot provided in this application embodiment, the method may include the following steps:

[0069] S101: Acquire a two-dimensional image of a specified object captured by a camera in the forward direction of the mobile robot, and a three-dimensional point cloud map of a specified plane parallel to the driving plane captured by a radar in the mobile robot.

[0070] S102: Perform instance segmentation on the two-dimensional image to obtain the image region occupied by the specified object in the two-dimensional image, which is used as the specified image region.

[0071] S103: Based on the specified image area and the transformation relationship between the radar coordinate system corresponding to the 3D point cloud map and the image coordinate system corresponding to the 2D image, determine the 3D points in the 3D point cloud map that belong to the specified object, and use them as the specified 3D points.

[0072] S104: Calculate the size of the space below the specified object based on the distance between the specified 3D point and the radar, and the image coordinates of the specified image region in the image coordinate system.

[0073] S105: Based on the size of the space below the specified object and the size of the mobile robot, control whether the mobile robot passes through the space below the specified object.

[0074] The mobile robot control method provided in this application combines two-dimensional images and three-dimensional point cloud maps to determine the size of the space below a specified object in the direction of the mobile robot's movement. Based on the size of the mobile robot, it can control whether the mobile robot can pass through the space below the specified object, thereby improving the intelligence of the mobile robot. Furthermore, the method provided in this application only requires acquiring two-dimensional images and a three-dimensional point cloud map of a specified plane parallel to the driving plane. That is, the camera in the mobile robot can be a monocular camera, and the radar can be a single-line radar; that is, no additional three-dimensional sensors are needed to determine whether the mobile robot can pass through the space below an obstacle. This reduces the production cost of the mobile robot and expands the applicability of the method.

[0075] For example, see Figure 2 , Figure 2 This is a schematic diagram of a mobile robot driving a scenario provided in an embodiment of this application.

[0076] Figure 2 In this example, the mobile robot is an AGV (Automated Guided Vehicle). It is equipped with a LiDAR and a monocular camera, and a shelf is located in the robot's forward direction. The control device within the mobile robot can acquire a two-dimensional image of the specified object (i.e., the shelf) in the forward direction captured by the monocular camera, and a three-dimensional point cloud map of a specified plane parallel to the ground captured by the LiDAR. Figure 2 In this embodiment, the 3D point cloud image acquired by the lidar includes 3D points on the shelf legs. Therefore, the control device can control whether the mobile robot passes through the space beneath the shelf, based on the method provided in this application.

[0077] In step S101, the mobile robot can be equipped with a camera and a radar. The designated object can be an object such as a shelf or a table. The camera can be a monocular camera or a multi-view camera, and it can acquire two-dimensional images along the mobile robot's forward direction. The radar can be a single-line radar or a multi-line radar, and it acquires a three-dimensional point cloud map of a designated plane parallel to the travel plane. For example, if the mobile robot is traveling on a horizontal surface, the designated plane can represent the plane where the radar is located, which is parallel to the ground. The radar can be of different types, such as lidar, millimeter-wave radar, or microwave radar, but is not limited to these. The two-dimensional image and the three-dimensional point cloud map acquired in this step correspond to the same time.

[0078] Regarding step S102, the control device can perform instance segmentation on the two-dimensional image based on the network model, thereby determining the type of object to which each pixel in the two-dimensional image belongs, as well as the pixels belonging to the same object, and thus determining the image region occupied by the specified object (i.e., the specified image region in this embodiment).

[0079] The network model mentioned above can be RCNN (Region Convolutional Neural Networks), or Fast RCNN (Fast Region Convolutional Neural Networks), or Mask R-CNN (Mask Fast Region Convolutional Neural Networks), but is not limited to these.

[0080] Regarding step S103, each 3D point in the 3D point cloud image has radar coordinates in the radar coordinate system, and each pixel in the 2D image has image coordinates in the image coordinate system. Additionally, the camera in the mobile robot also has a corresponding coordinate system (i.e., the camera coordinate system in this embodiment). The mobile robot may also have its own coordinate system (which can be called the robot coordinate system).

[0081] See Figure 3 , Figure 3 This is a schematic diagram of a coordinate system provided in an embodiment of this application.

[0082] Figure 3 In the robot coordinate system X r Y r Z r The X-axis is defined as the front of the mobile robot, the Z-axis is vertical, and the Y-axis is horizontal. A monocular camera is horizontally mounted on the top of the mobile robot, with the lens facing directly forward, at a height of h. c Camera coordinate system X c Y c Z c With the lens orientation as the Z-axis, the vertical direction as the Y-axis, and the horizontal direction as the X-axis, the lidar is horizontally mounted behind the monocular camera. The lidar coordinate system is X... l Y l Z l Using the front of the mobile robot as the X-axis, the vertical direction as the Z-axis, and the horizontal direction as the Y-axis, if the LiDAR is a single-line LiDAR, it can only detect one plane. Therefore, the coordinates of all the collected 3D points on the Z-axis are equal to the installation height of the LiDAR.

[0083] The robot coordinate system, camera coordinate system, and radar coordinate system satisfy the rigid body transformation conditions and can be converted into each other through translation and rotation.

[0084] In addition, based on the camera's intrinsic parameters, the conversion between the camera coordinate system and the image coordinate system can be achieved according to formula (1).

[0085]

[0086] Where (u,v) represents the image coordinates of the pixel in the image coordinate system, f x f represents the focal length of the camera along the X-axis of the camera coordinate system. y c represents the focal length of the camera along the Y-axis of the camera coordinate system. x c represents the offset of the camera's optical axis along the X-axis of the camera coordinate system. y This represents the offset of the camera's optical axis along the Y-axis of the camera coordinate system. (x) c ,y c ,z c ) represents the coordinates of the image coordinates (u,v) in the camera coordinate system.

[0087] Furthermore, based on the transformation relationship between the camera coordinate system and the radar coordinate system, as well as the transformation relationship between the camera coordinate system and the image coordinate system, the transformation relationship between the radar coordinate system and the image coordinate system can also be obtained.

[0088] In one embodiment, see Figure 4 ,exist Figure 1 Based on this, step S103 above may include the following steps:

[0089] S1031: According to the transformation relationship between the radar coordinate system corresponding to the 3D point cloud map and the image coordinate system corresponding to the 2D image, project each 3D point in the 3D point cloud map to the image coordinate system to obtain the corresponding projected pixel.

[0090] S1032: Determine the 3D points corresponding to the projected pixels belonging to the specified image region as the specified 3D points belonging to the specified object in the 3D point cloud map.

[0091] In this embodiment, the control device can project each three-dimensional point in the three-dimensional point cloud image onto the image coordinate system to obtain the corresponding projected pixel. That is, according to the transformation relationship between the radar coordinate system and the image coordinate system, the radar coordinates of each three-dimensional point in the three-dimensional point cloud image are transformed to obtain the corresponding image coordinates (i.e., the image coordinates of the projected pixel).

[0092] against Figure 2 In the scenario shown, the 3D point cloud image acquired by the radar contains 3D points on the shelf legs. After projecting these 3D points onto the image coordinate system, the control device can determine that the projected pixels belong to the image area occupied by the shelf in the 2D image, and thus determine that these 3D points are the specified 3D points.

[0093] Based on the above processing, it is possible to identify 3D points belonging to the same object in the 3D point cloud image; that is, by combining the 2D image and the 3D point cloud image, complete positional information of the object can be obtained. For example, see... Figure 5 The control equipment can determine Figure 5 Multiple 3D points in the data belong to the shelf, which means that the location information of the entire shelf can be obtained by combining the 3D data.

[0094] Regarding step S104, after determining the specified three-dimensional point, the distance between the specified three-dimensional point and the radar can be obtained based on the radar coordinates of the specified three-dimensional point in the radar coordinate system. Then, by combining this distance with the position of the specified object in the image coordinate system (i.e., the image coordinates of the specified image area in the image coordinate system), the size of the space below the specified object can be determined.

[0095] In one embodiment, see Figure 6 ,exist Figure 1 Based on this, step S104 above may include the following steps:

[0096] S1041: Calculate the height of the space below the specified object based on the distance between the specified 3D point and the radar, and the image coordinates of the upper part of the specified object in the image coordinate system.

[0097] S1042: Calculate the distance between the left and right sides of the specified object based on the image coordinates of the left and right sides of the specified object in the image coordinate system, and the transformation relationship between the camera coordinate system and the image coordinate system, and use this distance as the width of the space below the specified object.

[0098] In this embodiment of the application, the control device can determine the height and width of the space below the specified object to obtain the size of the space below the specified object.

[0099] In one embodiment, step S1041 above may include the following steps:

[0100] Step 1: Calculate the distance between the specified 3D point and the radar based on the distance between the specified 3D point and the radar, as well as the positional relationship between the camera and the radar.

[0101] Step 2: Calculate the height of the upper part of the specified object in the camera coordinate system based on the distance between the specified 3D point and the camera, and the image coordinates of the upper part of the specified object in the image coordinate system.

[0102] Step 3: Calculate the height of the space below the specified object based on the height of the upper part of the specified object in the camera coordinate system and the installation height of the camera.

[0103] In one implementation, the image coordinates of the lowest point of the upper part of the specified object in the image coordinate system can be obtained to calculate the height of the space below the specified object.

[0104] For example, see Figure 7 , Figure 7 This is a schematic diagram illustrating how to calculate the height of the space below a specified object, as provided in an embodiment of this application. Figure 7 In this context, the specified object is the shelf. The specified 3D point is a 3D point on the shelf leg. Based on the radar coordinates of the specified 3D point, the distance between the specified 3D point and the lidar in the optical axis direction of the monocular camera can be obtained; that is, the distance d between the shelf leg and the lidar can be obtained. l Furthermore, based on the positional relationship between the lidar and the monocular camera, the distance between them (i.e., the distance between them along the optical axis of the monocular camera) d can be determined. cl Therefore, the distance d between the shelf leg and the monocular camera can be obtained. c =d l -d cl .

[0105] In this embodiment of the application, the distance between a specified three-dimensional point and a monocular camera in the optical axis direction can be used as the distance between the lowest point of the upper part of the specified object and the monocular camera in the optical axis direction.

[0106] If the lowest point of the upper part of the specified object (i.e. the lower edge of the shelf tray) has image coordinates (u1, v1) in the image coordinate system, then the height of the lower edge of the shelf tray in the camera coordinate system can be obtained according to formula (1), as shown in formula (2).

[0107]

[0108] Furthermore, by combining the installation height of the monocular camera, the distance from the lower edge of the shelf support to the travel plane of the mobile robot can be obtained, that is, the height of the space below the shelf. For example, the height of the space below the shelf can be obtained by calculating the sum of the installation height of the monocular camera and the height of the lower edge of the shelf support in the camera coordinate system.

[0109] In one embodiment, for step S1042, pixels in the left and right side components of a specified object can be determined respectively, and these two pixels correspond to the same image height. For example, for Figure 5 The two pixels in the left and right legs of the shelf are pixel 1 (u2, v2) and pixel 2 (u3, v2), respectively. Then, combined with formula (1), we can calculate the position of pixel 1 and pixel 2 in X. c The distance along the axis, that is, the distance between the left and right legs of the shelf, is obtained as shown in formula (3).

[0110]

[0111] Where D represents the position of pixel 1 and pixel 2 in X. c Distance along the axial direction.

[0112] In one embodiment, step S105 above may include the following steps:

[0113] If the size of the space below the specified object is not less than the size of the mobile robot, then the mobile robot is controlled to pass through the space below the specified object; otherwise, the mobile robot is controlled to go around the left and right sides of the specified object.

[0114] In one implementation, if the height of the space below the specified object is not less than the height of the mobile robot, and the width of the space below the specified object is not less than the width of the mobile robot, then the control device can control the mobile robot to pass through the space below the specified object. Otherwise, the mobile robot is controlled to go around the left and right sides of the specified object.

[0115] In one embodiment, the control device can also determine whether there are other objects in the space below a specified object based on instance segmentation. If so, it is determined that the mobile robot cannot pass through, and the mobile robot is controlled to go around the left and right sides of the specified object; if not, and the size of the space below the specified object is not less than the size of the mobile robot, the control device can control the mobile robot to pass through the space below the specified object.

[0116] See Figure 8 , Figure 8 This is a control flowchart of a mobile robot provided in an embodiment of this application.

[0117] The control device can simultaneously acquire a 3D point cloud map collected by a lidar and a 2D image collected by a monocular camera. Then, it projects the 3D point cloud map onto the image coordinate system, that is, it determines the pixel point corresponding to each 3D point in the image coordinate system (i.e., the projected pixel point in this embodiment). Instance segmentation is performed on the 2D image to obtain the complete type and outline of the object in the 2D image.

[0118] Then, clustering is performed based on the projection results and instance segmentation results to obtain laser obstacles and complete virtual obstacles. That is, 3D points belonging to the same object are determined. For example, if the obstacle is a shelf, 3D points belonging to the same shelf leg (i.e., laser obstacles) can be determined. Furthermore, all 3D points belonging to that shelf can be determined, and combined with the 2D image, the position information of the entire shelf (i.e., the complete virtual obstacle) can be determined.

[0119] If the virtual obstacle is passable, calculate the size of the space below the virtual obstacle. If the size of the space below the virtual obstacle is not less than the size of the mobile robot, plan a path to pass through; otherwise, control the mobile robot to detour.

[0120] If the virtual obstacle is impassable, control the mobile robot to go around it.

[0121] Based on the same inventive concept, this application also provides a control device for a mobile robot, see [link to relevant documentation]. Figure 9 The device includes:

[0122] The image data acquisition module 901 is used to acquire a two-dimensional image containing a specified object in the forward direction captured by the camera in the mobile robot, and a three-dimensional point cloud map of a specified plane parallel to the driving plane captured by the radar in the mobile robot.

[0123] The specified image region determination module 902 is used to perform instance segmentation on the two-dimensional image to obtain the image region occupied by the specified object in the two-dimensional image, which is used as the specified image region.

[0124] The designated 3D point determination module 903 is used to determine, based on the designated image region and the transformation relationship between the radar coordinate system corresponding to the 3D point cloud map and the image coordinate system corresponding to the 2D image, the 3D points belonging to the designated object in the 3D point cloud map, and to designate the 3D points as designated 3D points.

[0125] The spatial calculation module 904 is used to calculate the size of the space below the specified object based on the distance between the specified three-dimensional point and the radar, and the image coordinates of the specified image region in the image coordinate system.

[0126] The control module 905 is used to control whether the mobile robot passes through the space below the specified object based on the size of the space below the specified object and the size of the mobile robot.

[0127] Optionally, the specified three-dimensional point determination module 903 is specifically used to project each three-dimensional point in the three-dimensional point cloud map to the image coordinate system according to the transformation relationship between the radar coordinate system corresponding to the three-dimensional point cloud map and the image coordinate system corresponding to the two-dimensional image, so as to obtain the corresponding projected pixel point.

[0128] The three-dimensional points corresponding to the projected pixels belonging to the specified image region are determined as the specified three-dimensional points belonging to the specified object in the three-dimensional point cloud map.

[0129] Optionally, the spatial computing module 904 includes:

[0130] The height calculation submodule is used to calculate the height of the space below the specified object based on the distance between the specified three-dimensional point and the radar, and the image coordinates of the upper part of the specified object in the image coordinate system;

[0131] The width calculation submodule is used to calculate the distance between the left and right sides of the specified object as the width of the space below the specified object, based on the image coordinates of the left and right sides of the specified object in the image coordinate system and the transformation relationship between the camera coordinate system and the image coordinate system.

[0132] Optionally, the height calculation submodule is specifically used to calculate the distance between the specified three-dimensional point and the camera based on the distance between the specified three-dimensional point and the radar, and the positional relationship between the camera and the radar;

[0133] Based on the distance between the specified 3D point and the camera, and the image coordinates of the upper part of the specified object in the image coordinate system, calculate the height of the upper part of the specified object in the camera coordinate system.

[0134] The height of the space below the specified object is calculated based on the height of the upper part of the specified object in the camera coordinate system and the installation height of the camera.

[0135] Optionally, the control module 905 is specifically used to control the mobile robot to pass through the space below the specified object if the size of the space below the specified object is not less than the size of the mobile robot;

[0136] If the size of the space below the specified object is smaller than the size of the mobile robot, then the mobile robot is controlled to move around the left and right sides of the specified object.

[0137] This application also provides a mobile robot, such as... Figure 10 As shown, it includes a camera 1001, a radar 1002, and a processor 1003;

[0138] The camera 1001 is used to acquire two-dimensional images in the forward direction;

[0139] The radar 1002 is used to acquire a three-dimensional point cloud map of a specified plane parallel to the driving plane.

[0140] The processor 1003 is used to execute any of the control methods for the mobile robot described above.

[0141] This application also provides a control device, such as... Figure 11As shown, it includes a processor 1101, a communication interface 1102, a memory 1103, and a communication bus 1104. The processor 1101, communication interface 1102, and memory 1103 communicate with each other via the communication bus 1104.

[0142] Memory 1103 is used to store computer programs;

[0143] When processor 1101 executes the program stored in memory 1103, it performs the following steps:

[0144] Acquire a two-dimensional image of a specified object captured by a camera in the forward direction of the mobile robot, and a three-dimensional point cloud map of a specified plane parallel to the driving plane captured by a radar in the mobile robot.

[0145] The two-dimensional image is segmented to obtain the image region occupied by the specified object in the two-dimensional image, which is used as the specified image region;

[0146] Based on the specified image region and the transformation relationship between the radar coordinate system corresponding to the 3D point cloud map and the image coordinate system corresponding to the 2D image, the 3D points in the 3D point cloud map belonging to the specified object are determined as the specified 3D points;

[0147] The size of the space below the specified object is calculated based on the distance between the specified three-dimensional point and the radar, and the image coordinates of the specified image region in the image coordinate system.

[0148] Based on the size of the space below the specified object and the size of the mobile robot, control whether the mobile robot passes through the space below the specified object.

[0149] The communication bus mentioned in the control device above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0150] The communication interface is used for communication between the aforementioned control device and other devices.

[0151] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0152] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0153] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described mobile robot control methods.

[0154] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the control methods for mobile robots described in the above embodiments.

[0155] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0156] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0157] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, mobile robots, control devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0158] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A control method for a mobile robot, characterized in that, The method includes: Acquire a two-dimensional image of a specified object captured by a camera in the forward direction of the mobile robot, and a three-dimensional point cloud map of a specified plane parallel to the driving plane captured by a radar in the mobile robot. The two-dimensional image is segmented to obtain the image region occupied by the specified object in the two-dimensional image, which is used as the specified image region; Based on the specified image region and the transformation relationship between the radar coordinate system corresponding to the 3D point cloud map and the image coordinate system corresponding to the 2D image, the 3D points in the 3D point cloud map belonging to the specified object are determined as the specified 3D points; The size of the space below the specified object is calculated based on the distance between the specified 3D point and the radar, and the image coordinates of the specified image region in the image coordinate system; the size of the space below the specified object includes the height and width of the space below. Based on the size of the space below the specified object and the size of the mobile robot, control whether the mobile robot passes through the space below the specified object.

2. The method according to claim 1, characterized in that, The step of determining the three-dimensional points belonging to the specified object in the three-dimensional point cloud map as specified three-dimensional points, based on the specified image region and the transformation relationship between the radar coordinate system corresponding to the three-dimensional point cloud map and the image coordinate system corresponding to the two-dimensional image, includes: According to the transformation relationship between the radar coordinate system corresponding to the three-dimensional point cloud map and the image coordinate system corresponding to the two-dimensional image, each three-dimensional point in the three-dimensional point cloud map is projected to the image coordinate system to obtain the corresponding projected pixel point. The three-dimensional points corresponding to the projected pixels belonging to the specified image region are determined as the specified three-dimensional points belonging to the specified object in the three-dimensional point cloud map.

3. The method according to claim 1, characterized in that, The step of calculating the size of the space below the specified object based on the distance between the specified 3D point and the radar, and the image coordinates of the specified image region in the image coordinate system, includes: Based on the distance between the specified three-dimensional point and the radar, and the image coordinates of the upper part of the specified object in the image coordinate system, calculate the height of the space below the specified object; Based on the image coordinates of the left and right sides of the specified object in the image coordinate system, and the transformation relationship between the camera coordinate system and the image coordinate system, the distance between the left and right sides of the specified object is calculated as the width of the space below the specified object.

4. The method according to claim 3, characterized in that, The step of calculating the height of the space below the specified object based on the distance between the specified three-dimensional point and the radar, and the image coordinates of the upper part of the specified object in the image coordinate system, includes: The distance between the specified three-dimensional point and the radar is calculated based on the distance between the specified three-dimensional point and the radar, and the positional relationship between the camera and the radar. Based on the distance between the specified 3D point and the camera, and the image coordinates of the upper part of the specified object in the image coordinate system, calculate the height of the upper part of the specified object in the camera coordinate system. The height of the space below the specified object is calculated based on the height of the upper part of the specified object in the camera coordinate system and the installation height of the camera.

5. The method according to claim 1, characterized in that, The step of controlling whether the mobile robot passes through the space below the specified object based on the size of the space below the specified object and the size of the mobile robot includes: If the size of the space below the specified object is not less than the size of the mobile robot, then control the mobile robot to pass through the space below the specified object; If the size of the space below the specified object is smaller than the size of the mobile robot, then the mobile robot is controlled to move around the left and right sides of the specified object.

6. A control device for a mobile robot, characterized in that, The device includes: The image data acquisition module is used to acquire a two-dimensional image containing a specified object in the forward direction captured by the camera in the mobile robot, and a three-dimensional point cloud map of a specified plane parallel to the driving plane captured by the radar in the mobile robot. The specified image region determination module is used to perform instance segmentation on the two-dimensional image to obtain the image region occupied by the specified object in the two-dimensional image, which is used as the specified image region. The specified 3D point determination module is used to determine, based on the specified image region and the transformation relationship between the radar coordinate system corresponding to the 3D point cloud map and the image coordinate system corresponding to the 2D image, the 3D points belonging to the specified object in the 3D point cloud map, and to designate the 3D points as specified 3D points. The spatial calculation module is used to calculate the size of the space below the specified object based on the distance between the specified three-dimensional point and the radar, and the image coordinates of the specified image region in the image coordinate system; the size of the space below the specified object includes the height and width of the space below. The control module is used to control whether the mobile robot passes through the space below the specified object based on the size of the space below the specified object and the size of the mobile robot.

7. The apparatus according to claim 6, characterized in that, The specified 3D point determination module is specifically used to project each 3D point in the 3D point cloud map to the image coordinate system according to the transformation relationship between the radar coordinate system corresponding to the 3D point cloud map and the image coordinate system corresponding to the 2D image, so as to obtain the corresponding projected pixel point. The three-dimensional points corresponding to the projected pixels belonging to the specified image region are determined as the specified three-dimensional points belonging to the specified object in the three-dimensional point cloud map.

8. The apparatus according to claim 6, characterized in that, The spatial computing module includes: The height calculation submodule is used to calculate the height of the space below the specified object based on the distance between the specified three-dimensional point and the radar, and the image coordinates of the upper part of the specified object in the image coordinate system; The width calculation submodule is used to calculate the distance between the left and right sides of the specified object as the width of the space below the specified object, based on the image coordinates of the left and right sides of the specified object in the image coordinate system and the transformation relationship between the camera coordinate system and the image coordinate system.

9. The apparatus according to claim 8, characterized in that, The height calculation submodule is specifically used to calculate the distance between the specified three-dimensional point and the camera based on the distance between the specified three-dimensional point and the radar, and the positional relationship between the camera and the radar; Based on the distance between the specified 3D point and the camera, and the image coordinates of the upper part of the specified object in the image coordinate system, calculate the height of the upper part of the specified object in the camera coordinate system. The height of the space below the specified object is calculated based on the height of the upper part of the specified object in the camera coordinate system and the installation height of the camera.

10. The apparatus according to claim 6, characterized in that, The control module is specifically used to control the mobile robot to pass through the space below the specified object if the size of the space below the specified object is not less than the size of the mobile robot. If the size of the space below the specified object is smaller than the size of the mobile robot, then the mobile robot is controlled to move around the left and right sides of the specified object.

11. A mobile robot, characterized in that, Includes cameras, radar, and processors; The camera is used to acquire two-dimensional images in the forward direction; The radar is used to acquire a three-dimensional point cloud map of a designated plane parallel to the driving plane; The processor is configured to execute the method steps of any one of claims 1-5.

12. A control device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the method steps of any one of claims 1-5.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method steps of any one of claims 1-5.

Images