Obstacle ranging method, mobile robot, device and medium

Abstract

The embodiments of the present application provide an obstacle ranging method, mobile robot, device, and medium, relating to the field of computer technology, including: obtaining an image to be measured captured by a monocular camera deployed on the mobile robot; detecting obstacles contained in the image to be measured and determining target pixel points used to represent the position of the obstacles in the image to be measured; using the pixel coordinates of the target pixel points in the image to be measured and a preset coordinate transformation relationship to obtain the spatial coordinates corresponding to the target pixel points, where the coordinate transformation relationship is: the transformation relationship between the coordinates in the image captured by the monocular camera and the coordinates in a preset spatial coordinate system; based on the obtained spatial coordinates, determining the distance of the obstacle relative to the mobile robot. Applying the solution provided by the embodiments of the present application can improve the efficiency of obtaining the distance between the obstacle and the mobile robot.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an obstacle ranging method, mobile robot, device and medium. Background Art

[0002] During the operation of a mobile robot, in order to prevent the mobile robot from colliding with an obstacle in front, it is usually necessary to detect the distance between the obstacle in front and the mobile robot.

[0003] In related technologies, in order to detect the distance between obstacles and mobile robots, in addition to deploying cameras on the mobile robots, lidars and ultrasonic sensors are also deployed on the mobile robots. The images collected by the cameras, the radar signals emitted by the lidars, and the ultrasonic signals emitted by the ultrasonic sensors are used to detect the distance to obstacles in front of the mobile robot.

[0004] Although the above solution can obtain the distance between the obstacle and the mobile robot, it requires deploying more sensors on the mobile robot, and thus needs to use the data collected by more sensors to calculate the distance to the obstacle. This not only leads to high cost of the mobile robot, but also requires a large amount of calculation, resulting in low efficiency in obtaining the distance between the obstacle and the mobile robot. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide an obstacle ranging method, mobile robot, device, and medium to improve the efficiency of obtaining the distance between an obstacle and the mobile robot. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides an obstacle ranging method, the method comprising:

[0007] Obtaining the image to be tested captured by a monocular camera deployed on a mobile robot;

[0008] Detecting obstacles contained in the image to be measured, and determining target pixel points used to represent the position of the obstacles in the image to be measured;

[0009] Obtaining the spatial coordinates corresponding to the target pixel point using the pixel coordinates of the target pixel point in the image to be measured and a preset coordinate transformation relationship, wherein the coordinate transformation relationship is a transformation relationship between the coordinates in the image captured by the monocular camera and the coordinates in a preset spatial coordinate system, the coordinate transformation relationship is obtained based on pre-calibrated camera parameters of the monocular camera, and the spatial coordinate system is a coordinate system established based on the mobile robot;

[0010] Based on the obtained spatial coordinates, the distance of the obstacle relative to the mobile robot is determined.

[0011] In one embodiment of the present application, the method further includes:

[0012] determining an obstacle area where the obstacle is located;

[0013] Based on the distance and the obstacle area, the mobile robot is controlled to avoid the obstacle area during operation.

[0014] In one embodiment of the present application, determining the obstacle area where the obstacle is located includes:

[0015] Determining the pixel width of the obstacle in the image to be measured;

[0016] The actual width of the obstacle is calculated using the pixel width, the spatial coordinates, and the camera parameters of the monocular camera, as the area width of the obstacle area where the obstacle is located.

[0017] In one embodiment of the present application, determining the obstacle area where the obstacle is located includes:

[0018] Identifying the target category of obstacles contained in the image to be tested;

[0019] According to the preset depth correspondence relationship between the depth and the category, the target depth corresponding to the target category is determined as the regional depth of the obstacle area where the obstacle is located.

[0020] In one embodiment of the present application, determining the obstacle area where the obstacle is located includes:

[0021] When the distance is less than a preset close distance threshold, determining the obstacle area where the obstacle is located; and / or

[0022] In a case where the obstacle is located in a non-edge area of ​​the image to be detected, an obstacle area where the obstacle is located is determined.

[0023] In one embodiment of the present application, determining a target pixel point for representing a position of the obstacle in the image to be measured includes:

[0024] The midpoint of the lower boundary of the image area occupied by the obstacle in the image to be measured is determined as a target pixel point for representing the position of the obstacle in the image to be measured.

[0025] In one embodiment of the present application, obtaining an image to be measured captured by a monocular camera deployed on a mobile robot includes:

[0026] Motion information collected by a motion sensor deployed on a mobile robot is obtained, and when the motion information meets a preset stability condition, an image to be measured collected by a monocular camera deployed on the mobile robot is obtained.

[0027] In one embodiment of the present application, obtaining the spatial coordinates corresponding to the target pixel point by using the pixel coordinates of the target pixel point in the image to be measured and a preset coordinate transformation relationship includes:

[0028] Dedistortion processing is performed on the target pixel point, and the spatial coordinates corresponding to the target pixel point are obtained by using the pixel coordinates of the target pixel point in the image to be measured after the dedistortion processing and a preset coordinate conversion relationship.

[0029] In a second aspect, an embodiment of the present application provides a mobile robot, comprising a monocular camera and a processor, wherein:

[0030] The monocular camera is used to: collect the image to be tested and send the image to be tested to the processor;

[0031] The processor is used to: receive the image to be measured, detect the obstacle contained in the image to be measured, and determine the target pixel point used to represent the position of the obstacle in the image to be measured; use the pixel coordinates of the target pixel point in the image to be measured and a preset coordinate transformation relationship to obtain the spatial coordinates corresponding to the target pixel point, wherein the coordinate transformation relationship is: the transformation relationship between the coordinates in the image captured by the monocular camera and the coordinates in a preset spatial coordinate system, the coordinate transformation relationship is obtained based on pre-calibrated camera parameters of the monocular camera, and the spatial coordinate system is: a coordinate system established based on the mobile robot; based on the obtained spatial coordinates, determine the distance of the obstacle relative to the mobile robot.

[0032] In one embodiment of the present application, the mobile robot further includes a motion sensor, wherein:

[0033] The motion sensor is used to: collect motion information and send the motion information to the processor;

[0034] The processor is used to: receive the motion information, and when the motion information meets a preset stability condition, obtain an image to be measured captured by a monocular camera deployed on the mobile robot.

[0035] In a third aspect, an embodiment of the present application provides an obstacle ranging device, the device comprising:

[0036] An image acquisition module is used to obtain the image to be tested captured by a monocular camera deployed on the mobile robot;

[0037] a pixel point determination module, configured to detect obstacles contained in the image to be measured and determine target pixels used to represent the position of the obstacles in the image to be measured;

[0038] a coordinate conversion module, configured to obtain spatial coordinates corresponding to the target pixel point using the pixel coordinates of the target pixel point in the image to be measured and a preset coordinate conversion relationship, wherein the coordinate conversion relationship is a conversion relationship between coordinates in the image captured by the monocular camera and coordinates in a preset spatial coordinate system, the coordinate conversion relationship being obtained based on pre-calibrated camera parameters of the monocular camera, and the spatial coordinate system being a coordinate system established based on the mobile robot;

[0039] The obstacle ranging module is used to determine the distance of the obstacle relative to the mobile robot based on the obtained spatial coordinates.

[0040] In one embodiment of the present application, the device further comprises:

[0041] An area determination module, configured to determine an obstacle area where the obstacle is located;

[0042] The obstacle avoidance module is used to control the mobile robot to avoid the obstacle area during operation based on the distance and the obstacle area.

[0043] In one embodiment of the present application, the region determination module is specifically configured to:

[0044] Determining the pixel width of the obstacle in the image to be measured;

[0045] The actual width of the obstacle is calculated using the pixel width, the spatial coordinates, and the camera parameters of the monocular camera, as the area width of the obstacle area where the obstacle is located.

[0046] In one embodiment of the present application, the region determination module is specifically configured to:

[0047] Identifying the target category of obstacles contained in the image to be tested;

[0048] According to the preset depth correspondence relationship between the depth and the category, the target depth corresponding to the target category is determined as the regional depth of the obstacle area where the obstacle is located.

[0049] In one embodiment of the present application, the region determination module is specifically configured to:

[0050] When the distance is less than a preset close distance threshold, determining the obstacle area where the obstacle is located; and / or

[0051] In a case where the obstacle is located in a non-edge area of ​​the image to be detected, an obstacle area where the obstacle is located is determined.

[0052] In one embodiment of the present application, the pixel point determination module is specifically configured to:

[0053] The midpoint of the lower boundary of the image area occupied by the obstacle in the image to be measured is determined as a target pixel point for representing the position of the obstacle in the image to be measured.

[0054] In one embodiment of the present application, the image acquisition module is specifically configured to:

[0055] Motion information collected by a motion sensor deployed on a mobile robot is obtained, and when the motion information meets a preset stability condition, an image to be measured collected by a monocular camera deployed on the mobile robot is obtained.

[0056] In one embodiment of the present application, the coordinate conversion module is specifically used to:

[0057] Dedistortion processing is performed on the target pixel point, and the spatial coordinates corresponding to the target pixel point are obtained by using the pixel coordinates of the target pixel point in the image to be measured after the dedistortion processing and a preset coordinate conversion relationship.

[0058] In a fourth aspect, an embodiment of the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0059] Memory for storing computer programs;

[0060] The processor is configured to implement any of the method steps described in the first aspect when executing a program stored in the memory.

[0061] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the method steps described in the first aspect is implemented.

[0062] An embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-described obstacle ranging methods.

[0063] Beneficial effects of the embodiments of the present application:

[0064] In the obstacle ranging solution provided in the embodiments of the present application, an image to be measured, captured by a monocular camera deployed on a mobile robot, is obtained; obstacles contained in the image to be measured are detected, and target pixels representing the position of the obstacles in the image to be measured are determined; the spatial coordinates corresponding to the target pixels are obtained using the pixel coordinates of the target pixels in the image to be measured and a preset coordinate transformation relationship, wherein the coordinate transformation relationship is the transformation relationship between the coordinates in the image captured by the monocular camera and the coordinates in a preset spatial coordinate system, which is obtained based on pre-calibrated camera parameters of the monocular camera, and the spatial coordinate system is a coordinate system established based on the mobile robot; based on the obtained spatial coordinates, the distance of the obstacle relative to the mobile robot is determined. In this manner, only the monocular camera needs to be deployed on the mobile robot, and the pixel coordinates of the obstacle in the image captured by the monocular camera are determined. The coordinate transformation relationship is then used to obtain the coordinates of the obstacle in space, thereby obtaining the distance of the obstacle relative to the mobile robot. This distance can be obtained without using data collected by other sensors, thereby reducing the amount of computation required. It can be seen that the application of the obstacle ranging solution provided in the embodiment of the present application can improve the efficiency of obtaining the distance between the obstacle and the mobile robot, and can also reduce the cost of the mobile robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0066] Figure 1 A flowchart of an obstacle ranging method provided in an embodiment of the present application;

[0067] Figure 2 A schematic diagram of a target pixel provided in an embodiment of the present application;

[0068] Figure 3 A schematic diagram of a calibration process provided in an embodiment of the present application;

[0069] Figure 4 A flowchart of another obstacle ranging method provided in an embodiment of the present application;

[0070] Figure 5 A schematic diagram of an obstacle area image provided in an embodiment of the present application;

[0071] Figure 6 A binary boundary map provided in an embodiment of the present application;

[0072] Figure 7A schematic diagram of an envelope frame provided in an embodiment of the present application;

[0073] Figure 8 A schematic structural diagram of a mobile robot provided in an embodiment of the present application;

[0074] Figure 9 A schematic structural diagram of an obstacle ranging device provided in an embodiment of the present application;

[0075] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0077] In order to improve the efficiency of obtaining the distance between an obstacle and a mobile robot, the embodiments of the present application provide an obstacle ranging method, a mobile robot, a device and a medium, which are described in detail below.

[0078] The present application provides an obstacle ranging method that can be applied to electronic devices such as mobile robots, computers, and servers. The method includes:

[0079] Obtaining the image to be tested captured by a monocular camera deployed on a mobile robot;

[0080] Detecting obstacles contained in the image to be tested and determining target pixel points used to represent the position of the obstacles in the image to be tested;

[0081] The spatial coordinates corresponding to the target pixel point are obtained using the pixel coordinates of the target pixel point in the image to be measured and a preset coordinate transformation relationship, wherein the coordinate transformation relationship is the transformation relationship between the coordinates in the image captured by the monocular camera and the coordinates in a preset spatial coordinate system. The coordinate transformation relationship is obtained based on pre-calibrated camera parameters of the monocular camera, and the spatial coordinate system is a coordinate system established based on the mobile robot.

[0082] Based on the obtained spatial coordinates, the distance of the obstacle relative to the mobile robot is determined.

[0083] In this way, only a monocular camera needs to be deployed on the mobile robot. Using the image captured by the monocular camera, the pixel coordinates of the obstacle in the image are determined. Then, through coordinate transformation, the coordinates of the obstacle in space are obtained, and the distance of the obstacle relative to the mobile robot is obtained. This distance can be obtained without using data collected by other sensors, thereby reducing the amount of calculation. Thus, the obstacle ranging solution provided by the above embodiment can improve the efficiency of obtaining the distance between the obstacle and the mobile robot, and can also reduce the cost of the mobile robot.

[0084] The above obstacle ranging method is introduced in detail below.

[0085] See also Figure 1 , Figure 1 A flowchart of an obstacle ranging method provided in an embodiment of the present application is provided. The method includes the following steps S101-S104:

[0086] S101, obtaining an image to be tested captured by a monocular camera deployed on a mobile robot.

[0087] The mobile robot is a robot that moves on the ground, for example, a sweeping robot, a patrol robot, a transfer robot, an AGV (Automated Guided Vehicle), etc.

[0088] The monocular camera can be deployed on the top, bottom, side, or front of the mobile robot. The monocular camera can be a regular camera, a fisheye camera, a wide-angle camera, etc. The monocular camera can be oriented in the direction of the mobile robot's movement, which facilitates the monocular camera to capture images of obstacles in front of the mobile robot.

[0089] Specifically, the mobile robot can run on the ground. During the running process, the image captured by the monocular camera can be obtained as the image to be tested, so as to facilitate the subsequent detection of obstacles on the ground based on the image to be tested.

[0090] In one embodiment of the present application, motion information collected by a motion sensor deployed on a mobile robot can be obtained, and when the motion information meets a preset stability condition, an image to be measured collected by a monocular camera deployed on the mobile robot can be obtained.

[0091] The motion sensor may be a gyroscope, a GPS (Global Positioning System), or the like.

[0092] The above-mentioned motion information refers to motion data collected by the motion sensor, for example, it can be at least one of acceleration, angular velocity, tilt angle and other information.

[0093] The above stability condition is used to determine whether the current motion state of the motion sensor is stable. For example, it can be at least one of the conditions of acceleration being less than a preset acceleration threshold, angular velocity being less than a preset angular velocity threshold, and the like.

[0094] Specifically, a motion sensor can be deployed on a mobile robot. The motion information collected by the motion sensor can reflect the motion of the mobile robot. If the motion information meets the above-mentioned stability conditions, it indicates that the mobile robot is not shaking violently. In this case, the quality of the image collected by the monocular camera on the mobile robot is high, which facilitates subsequent obstacle ranging based on the image. Therefore, the image collected by the monocular camera can be obtained as the image to be measured.

[0095] If the above motion information does not meet the stability conditions, it means that the mobile robot is currently shaking violently. In this case, the quality of the image captured by the monocular camera on the mobile robot is low, which is not convenient for subsequent obstacle ranging based on the image. Therefore, the image captured by the monocular camera can be omitted.

[0096] S102 : Detect obstacles contained in the image to be measured, and determine target pixel points used to represent the positions of the obstacles in the image to be measured.

[0097] The above obstacles may include: towels, data cables, packaging bags, shoes, socks, etc.

[0098] The target pixel point may be a center point or an edge point of the image area occupied by the obstacle in the image to be measured. The image area may be rectangular, circular, elliptical, or the area enclosed by the outline of the obstacle.

[0099] Specifically, image detection can be performed on the image to be tested to detect obstacles contained in the image to be tested, and then target pixels used to represent the image position of the obstacle in the image to be tested are determined based on the area occupied by the detected obstacle in the image to be tested.

[0100] In one embodiment of the present application, when detecting obstacles contained in an image to be tested, the image to be tested can be input into a pre-trained obstacle detection model, and the model is used to detect the obstacles contained in the image to be tested to obtain an output result of the model, which may include location information of the obstacle in the image to be tested.

[0101] In one embodiment of the present application, when determining the target pixel point, the midpoint of the lower boundary of the image area occupied by the obstacle in the image to be measured may be determined as the target pixel point for representing the position of the obstacle in the image to be measured.

[0102] Specifically, the image area occupied by the obstacle in the image to be tested can be determined first, and then the midpoint of the lower boundary of the image area can be determined, and the midpoint can be used as the target pixel point representing the position of the obstacle in the image to be tested. Figure 2 , Figure 2 A schematic diagram of a target pixel point provided in an embodiment of the present application assumes that the obstacle contained in the image to be measured is a towel, and the image area occupied by the towel is rectangular. Therefore, the midpoint of the lower boundary of the rectangular area can be used as the target pixel point representing the towel's position in the image to be measured. By using the midpoint of the lower boundary of the image area occupied by the obstacle in the image to be measured as the obstacle's position, the position closest to the mobile robot can be determined. When subsequently calculating the distance of the obstacle to the mobile robot based on this position, the closest distance can be calculated, thereby facilitating subsequent obstacle avoidance based on this distance.

[0103] S103 , obtaining the spatial coordinates corresponding to the target pixel point by using the pixel coordinates of the target pixel point in the image to be measured and a preset coordinate conversion relationship.

[0104] The coordinate conversion relationship is the conversion relationship between the coordinates in the image captured by the monocular camera and the coordinates in the preset spatial coordinate system.

[0105] The coordinate transformation relationship is based on the pre-calibrated camera parameters of the monocular camera. These parameters include the camera's intrinsic and extrinsic parameters. The extrinsic parameters include the camera's height relative to the ground, pitch angle θ, and roll angle α, while the intrinsic parameters include the projection matrix K and distortion matrix D.

[0106] The spatial coordinate system is: a three-dimensional coordinate system established based on the mobile robot.

[0107] Specifically, the camera parameters of the monocular camera can be pre-calibrated, and the coordinate transformation relationship between the coordinates in the image captured by the monocular camera and the coordinates in the spatial coordinate system established based on the mobile robot can be determined based on the camera parameters. Then, the pixel coordinates of the above-mentioned target pixel point in the image to be measured can be obtained, and then the pixel coordinates are transformed using the coordinate transformation relationship to obtain the spatial coordinates corresponding to the target pixel point. The spatial coordinates are: the coordinates of the obstacle in the spatial coordinate system established based on the mobile robot.

[0108] In one embodiment of the present application, the target pixel point may be subjected to dedistortion processing, and the spatial coordinates corresponding to the target pixel point may be obtained using the pixel coordinates of the target pixel point in the image to be measured after dedistortion processing and a preset coordinate conversion relationship.

[0109] Specifically, the target pixel point can be dedistorted using the distortion matrix D in the intrinsic parameters of the monocular camera to obtain the processed target pixel point. Subsequently, the above-mentioned coordinate transformation relationship can be used to transform the pixel coordinates of the target pixel point after dedistortion in the image to be measured, thereby obtaining the spatial coordinates corresponding to the target pixel point after dedistortion.

[0110] Such dedistortion processing of the target pixel points can eliminate the problem of pixel coordinate offset of the target pixel points caused by distortion, thereby improving the accuracy of the pixel coordinates of the target pixel points obtained, and further improving the accuracy of the distance obtained based on the subsequent measurement of the target pixel points. In addition, there is no need to perform dedistortion processing on all pixel points in the image to be tested, which can reduce the amount of calculation, improve calculation efficiency, and avoid the problem of reduced image field of view due to dedistortion processing, thereby ensuring that the image to be tested has a wider field of view and increasing the obstacle detection range.

[0111] In one embodiment of the present application, the process of establishing the above coordinate transformation relationship is as follows:

[0112] After pre-calibrating the external parameters of the monocular camera, the rotation matrix R between the image coordinate system of the image captured by the camera and the spatial coordinate system established based on the mobile robot can be established based on the pitch angle θ and roll angle α in the external parameters. wc :

[0113]

[0114] Using the rotation matrix R wc , we can get the coordinate transformation relationship between the coordinates (u, v) in the image coordinate system and the coordinates (x, y, -h) in the space coordinate system:

[0115]

[0116] Where h represents the height of the monocular camera, x and y represent the horizontal coordinates and vertical coordinates of the obstacle, respectively, K represents the projection matrix in the calibrated intrinsic parameters, and s represents the scale factor. The size of s can be calculated based on the parameters in the third row of the matrix contained in the above coordinate transformation relationship.

[0117] In one embodiment of the present application, a preset calibration algorithm can be used to obtain the intrinsic and extrinsic parameters of the monocular camera. Figure 3 , Figure 3A schematic diagram of a calibration process provided in an embodiment of the present application is provided. A checkerboard calibration plate perpendicular to the ground can be set up, and a two-dimensional coordinate system can be established with the calibration plate as a reference, as the checkerboard system. A three-dimensional coordinate system can then be established with the ground as a reference, as the ground system. A first transformation relationship between the checkerboard system and the ground system can be obtained by measurement. A three-dimensional coordinate system is established with the camera as a reference, as the camera system. An image captured by the camera of the checkerboard calibration plate is obtained. Based on the correspondence between the pixel coordinates of each checkerboard in the image and the coordinates of each checkerboard in the checkerboard system, a second transformation relationship between the camera system and the checkerboard system is obtained. Based on the second transformation relationship, the intrinsic and extrinsic parameters of the monocular camera relative to the checkerboard calibration plate can be determined. The intrinsic and extrinsic parameters are then transformed using the above-mentioned first transformation relationship to obtain the intrinsic and extrinsic parameters of the monocular camera relative to the ground.

[0118] S104: Determine the distance of the obstacle relative to the mobile robot based on the obtained spatial coordinates.

[0119] Specifically, the above spatial coordinates are the coordinates of the obstacle in the spatial coordinate system established with the mobile robot as the reference. Using these coordinates, the distance of the obstacle relative to the mobile robot can be calculated. For example, assuming that the coordinates of the obstacle in the above spatial coordinate system are (x, y, z), the distance of the obstacle relative to the mobile robot can be calculated as (x 2 +y 2 +z 2 ) 1 / 2 Or, since the obstacle is on the ground, the height information of the obstacle can be ignored. In this case, the distance between the obstacle and the mobile robot can be (x 2 +y 2 ) 1 / 2 .

[0120] In the obstacle ranging solution provided in the above embodiment, an image to be measured, captured by a monocular camera deployed on a mobile robot, is obtained; obstacles contained in the image to be measured are detected, and target pixels representing the position of the obstacles in the image to be measured are determined; the spatial coordinates corresponding to the target pixels are obtained using the pixel coordinates of the target pixels in the image to be measured and a preset coordinate transformation relationship, wherein the coordinate transformation relationship is the transformation relationship between the coordinates in the image captured by the monocular camera and the coordinates in a preset spatial coordinate system, which is obtained based on pre-calibrated camera parameters of the monocular camera, and the spatial coordinate system is a coordinate system established based on the mobile robot; and based on the obtained spatial coordinates, the distance of the obstacle relative to the mobile robot is determined. In this manner, only the monocular camera needs to be deployed on the mobile robot, and the pixel coordinates of the obstacle in the image captured by the monocular camera are determined. The coordinate transformation relationship is then used to obtain the coordinates of the obstacle in space, thereby obtaining the distance of the obstacle relative to the mobile robot. This distance can be obtained without using data collected by other sensors, thereby reducing the amount of computation required. It can be seen that the application of the obstacle ranging solution provided by the above embodiment can improve the efficiency of obtaining the distance between the obstacle and the mobile robot, and can also reduce the cost of the mobile robot.

[0121] See also Figure 4 , Figure 4 A flowchart of another obstacle ranging method provided in an embodiment of the present application is provided. The method further includes the following steps S105-S106:

[0122] S105: Determine the obstacle area where the obstacle is located.

[0123] Specifically, after the image area where the obstacle is located is identified in the image to be tested, the obstacle area occupied by the obstacle in the actual scene can be further determined based on the image area, so that the mobile robot can be controlled to avoid the obstacle area in the subsequent process.

[0124] In one embodiment of the present application, the pixel width of the obstacle in the image to be measured can be determined; the actual width of the obstacle is calculated using the pixel width, spatial coordinates, and camera parameters of the monocular camera, which is used as the area width of the obstacle area where the obstacle is located.

[0125] Specifically, the area width w can be calculated according to the following formula:

[0126] w=(+u*x) / K 1,1

[0127] Among them, Δu represents the pixel width of the obstacle in the image to be measured, x represents the x coordinate of the spatial coordinate corresponding to the target pixel point, K 1,1 Represents the parameters of the first row and first column in the projection matrix of the monocular camera.

[0128] In one embodiment of the present application, the target category of the obstacle contained in the image to be tested can also be identified; according to the depth correspondence between the preset depth and category, the target depth corresponding to the target category is determined as the regional depth of the obstacle area where the obstacle is located.

[0129] Specifically, the correspondence between different obstacle categories and depths can be pre-set as the depth correspondence. For example, the depth of the towel can be set to 10 cm, the depth of the shoes can be set to 8 cm, and the depth of the data cable can be set to 5 cm. In actual applications, the category of the obstacle in the image to be tested can be identified as the target category, and then the depth corresponding to the target category can be found from the above depth correspondence as the target depth, and then the target depth can be used as the regional depth of the obstacle area where the obstacle is located.

[0130] S106 , based on the distance and the obstacle area, controlling the mobile robot to avoid the obstacle area during operation.

[0131] Specifically, after obtaining the distance between the obstacle and the mobile robot and the obstacle area occupied by the obstacle, the mobile robot can be controlled to bypass the obstacle area during movement.

[0132] In one embodiment of the present application, the obstacle area where the obstacle is located can be determined when the distance is less than a preset close distance threshold.

[0133] Among them, the above-mentioned close distance threshold can be 80 cm, 100 cm, 150 cm, etc.

[0134] Specifically, when the measured distance of the obstacle relative to the mobile robot is less than the close distance threshold, it means that the mobile robot is about to reach the obstacle. Therefore, the obstacle area where the obstacle is located can be determined to facilitate subsequent obstacle avoidance.

[0135] When the measured distance of the obstacle relative to the mobile robot is greater than or equal to the above-mentioned close distance threshold, it means that the mobile robot is far away from the obstacle. At this time, the error of the distance obtained may be large, and the error of the determined obstacle area may be large. Therefore, there is no need to determine the obstacle area where the obstacle is located.

[0136] In one embodiment of the present application, when the obstacle is in a non-edge area of ​​the image to be measured, the obstacle area where the obstacle is located can be determined.

[0137] Among them, the above-mentioned non-edge area refers to: the area other than the edge area in the image to be tested, and the above-mentioned edge area can be: the area in the image to be tested that is less than the preset pixel distance from the boundary, and the above-mentioned preset pixel distance can be 10 pixels, 20 pixels, 30 pixels, etc.

[0138] Specifically, since the monocular camera is usually oriented in the forward direction of the mobile robot, if the obstacle is in the non-edge area of ​​the image to be measured, it means that the obstacle is in front of the mobile robot and the mobile robot may collide with the obstacle. Therefore, it is necessary to determine the obstacle area where the obstacle is located;

[0139] When the obstacle is in the non-edge area of ​​the image to be measured, it means that the obstacle is on the side of the mobile robot and it is difficult for the mobile robot to collide with the obstacle. Therefore, there is no need to determine the obstacle area where the obstacle is located.

[0140] In one embodiment of the present application, when identifying the category of an obstacle, the image to be tested can be input into a pre-trained category recognition model to obtain the category of the obstacle output by the model. In addition, a preset algorithm can be used to extract the features of the obstacle, and the obstacles can be classified based on the features to obtain the category of the obstacle. The above-mentioned algorithm can be Canny operator, Sobel algorithm, Laplacian algorithm, etc.

[0141] The obstacle detection process is described below with reference to a specific embodiment.

[0142] First, use Yolov5 to detect obstacles in the image to be tested, and then crop the image area where the obstacles are located to obtain the obstacle area image, see Figure 5 , Figure 5 This is a schematic diagram of an obstacle area image provided in an embodiment of the present application. Assuming that the obstacle contained in the image to be measured is a data line, the image area where the data line is located is cropped, and the following can be obtained: Figure 5 The image of the obstacle area shown;

[0143] Using the Sobel operator, the gradient of the obstacle area image is calculated in the horizontal and vertical directions respectively. Then, the gradients in the horizontal and vertical directions are weighted summed to obtain the total gradient of the obstacle area image.

[0144] Then, based on the above total gradient, the obstacle area image is threshold filtered to obtain the filtered binary boundary map, see Figure 6 , Figure 6 A binary boundary map is provided in the embodiment of the present application, using the total gradient to Figure 5 After threshold filtering of the obstacle area image shown in Figure 6 The binary boundary image shown can reflect the boundary area of ​​the data line;

[0145] The binary boundary map is filtered again, and then the connected domain algorithm is used to determine the connected domains in the binary boundary map. Finally, the intersecting connected domains are merged to determine the minimum envelope of the connected domains. The envelope is used as the image area where the obstacle is located. Figure 7 , Figure 7 This is a schematic diagram of an envelope frame provided in an embodiment of the present application. Figure 7 The envelope shown is the image area where the data line is located;

[0146] This image area can then be used to detect the category of the obstacle.

[0147] In the obstacle ranging solution provided in the above embodiment, an image to be measured, captured by a monocular camera deployed on a mobile robot, is obtained; obstacles contained in the image to be measured are detected, and target pixels representing the position of the obstacles in the image to be measured are determined; the spatial coordinates corresponding to the target pixels are obtained using the pixel coordinates of the target pixels in the image to be measured and a preset coordinate transformation relationship, wherein the coordinate transformation relationship is the transformation relationship between the coordinates in the image captured by the monocular camera and the coordinates in a preset spatial coordinate system, which is obtained based on pre-calibrated camera parameters of the monocular camera, and the spatial coordinate system is a coordinate system established based on the mobile robot; and based on the obtained spatial coordinates, the distance of the obstacle relative to the mobile robot is determined. In this manner, only the monocular camera needs to be deployed on the mobile robot, and the pixel coordinates of the obstacle in the image captured by the monocular camera are determined. The coordinate transformation relationship is then used to obtain the coordinates of the obstacle in space, thereby obtaining the distance of the obstacle relative to the mobile robot. This distance can be obtained without using data collected by other sensors, thereby reducing the amount of computation required. It can be seen that the application of the obstacle ranging solution provided by the above embodiment can improve the efficiency of obtaining the distance between the obstacle and the mobile robot, and can also reduce the cost of the mobile robot.

[0148] Corresponding to the above-mentioned obstacle ranging method, an embodiment of the present application also provides a mobile robot, which is introduced in detail below.

[0149] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of a mobile robot provided in an embodiment of the present application. The mobile robot includes a monocular camera 801 and a processor 802, wherein:

[0150] The monocular camera 801 is used to: collect the image to be tested and send the image to be tested to the processor 802;

[0151] The processor 802 is used to: receive an image to be measured, detect obstacles contained in the image to be measured, and determine a target pixel point used to represent the position of the obstacle in the image to be measured; use the pixel coordinates of the target pixel point in the image to be measured and a preset coordinate transformation relationship to obtain the spatial coordinates corresponding to the target pixel point, wherein the coordinate transformation relationship is: the transformation relationship between the coordinates in the image captured by the monocular camera 801 and the coordinates in a preset spatial coordinate system, the coordinate transformation relationship is obtained based on pre-calibrated camera parameters of the monocular camera 801, and the spatial coordinate system is: a coordinate system established based on the mobile robot; based on the obtained spatial coordinates, determine the distance of the obstacle relative to the mobile robot.

[0152] Specifically, a mobile robot can be equipped with a monocular camera and a processor. The monocular camera can capture an image of the area in front of the mobile robot as a test image, and send the test image to the processor. The processor uses the test image to determine the distance of the obstacle in front of the mobile robot relative to the mobile robot. In this way, it is only necessary to deploy a monocular camera on the mobile robot, use the image captured by the monocular camera to determine the pixel coordinates of the obstacle in the image, and then obtain the coordinates of the obstacle in space through a coordinate conversion relationship, and then obtain the distance of the obstacle relative to the mobile robot. This distance can be obtained without using data collected by other sensors, thereby reducing the amount of calculation. Therefore, the obstacle ranging solution provided by the above embodiment can improve the efficiency of obtaining the distance between the obstacle and the mobile robot, and can also reduce the cost of the mobile robot.

[0153] In one embodiment of the present application, the mobile robot further includes a motion sensor, wherein:

[0154] The motion sensor is used to: collect motion information and send the motion information to the processor 802;

[0155] The processor 802 is used to receive motion information and, when the motion information satisfies a preset stability condition, obtain an image to be measured captured by the monocular camera 801 deployed on the mobile robot.

[0156] Corresponding to the above-mentioned obstacle ranging method, the embodiment of the present application further provides an obstacle ranging device, which is described in detail below.

[0157] See also Figure 9 , Figure 9 This is a schematic structural diagram of an obstacle ranging device provided in an embodiment of the present application, the device comprising:

[0158] The image acquisition module 901 is used to obtain the image to be measured captured by the monocular camera deployed on the mobile robot;

[0159] A pixel point determination module 902 is configured to detect obstacles contained in the image to be measured and determine target pixels used to represent the position of the obstacles in the image to be measured;

[0160] A coordinate conversion module 903 is configured to obtain spatial coordinates corresponding to the target pixel point using the pixel coordinates of the target pixel point in the image to be measured and a preset coordinate conversion relationship, wherein the coordinate conversion relationship is a conversion relationship between coordinates in the image captured by the monocular camera and coordinates in a preset spatial coordinate system, the coordinate conversion relationship being obtained based on pre-calibrated camera parameters of the monocular camera, and the spatial coordinate system being a coordinate system established based on the mobile robot;

[0161] The obstacle ranging module 904 is configured to determine the distance of the obstacle relative to the mobile robot based on the obtained spatial coordinates.

[0162] In one embodiment of the present application, the device further comprises:

[0163] An area determination module, configured to determine an obstacle area where the obstacle is located;

[0164] The obstacle avoidance module is used to control the mobile robot to avoid the obstacle area during operation based on the distance and the obstacle area.

[0165] In one embodiment of the present application, the region determination module is specifically configured to:

[0166] Determining the pixel width of the obstacle in the image to be measured;

[0167] The actual width of the obstacle is calculated using the pixel width, the spatial coordinates, and the camera parameters of the monocular camera, as the area width of the obstacle area where the obstacle is located.

[0168] In one embodiment of the present application, the region determination module is specifically configured to:

[0169] Identifying the target category of obstacles contained in the image to be tested;

[0170] According to the preset depth correspondence relationship between the depth and the category, the target depth corresponding to the target category is determined as the regional depth of the obstacle area where the obstacle is located.

[0171] In one embodiment of the present application, the region determination module is specifically configured to:

[0172] When the distance is less than a preset close distance threshold, determining the obstacle area where the obstacle is located; and / or

[0173] In a case where the obstacle is located in a non-edge area of ​​the image to be detected, an obstacle area where the obstacle is located is determined.

[0174] In one embodiment of the present application, the pixel point determination module 902 is specifically configured to:

[0175] The midpoint of the lower boundary of the image area occupied by the obstacle in the image to be measured is determined as a target pixel point for representing the position of the obstacle in the image to be measured.

[0176] In one embodiment of the present application, the image acquisition module 901 is specifically configured to:

[0177] Motion information collected by a motion sensor deployed on a mobile robot is obtained, and when the motion information meets a preset stability condition, an image to be measured collected by a monocular camera deployed on the mobile robot is obtained.

[0178] In one embodiment of the present application, the coordinate conversion module 903 is specifically configured to:

[0179] Dedistortion processing is performed on the target pixel point, and the spatial coordinates corresponding to the target pixel point are obtained by using the pixel coordinates of the target pixel point in the image to be measured after the dedistortion processing and a preset coordinate conversion relationship.

[0180] In the obstacle ranging solution provided in the above embodiment, an image to be measured, captured by a monocular camera deployed on a mobile robot, is obtained; obstacles contained in the image to be measured are detected, and target pixels representing the position of the obstacles in the image to be measured are determined; the spatial coordinates corresponding to the target pixels are obtained using the pixel coordinates of the target pixels in the image to be measured and a preset coordinate transformation relationship, wherein the coordinate transformation relationship is the transformation relationship between the coordinates in the image captured by the monocular camera and the coordinates in a preset spatial coordinate system, which is obtained based on pre-calibrated camera parameters of the monocular camera, and the spatial coordinate system is a coordinate system established based on the mobile robot; and based on the obtained spatial coordinates, the distance of the obstacle relative to the mobile robot is determined. In this manner, only the monocular camera needs to be deployed on the mobile robot, and the pixel coordinates of the obstacle in the image captured by the monocular camera are determined. The coordinate transformation relationship is then used to obtain the coordinates of the obstacle in space, thereby obtaining the distance of the obstacle relative to the mobile robot. This distance can be obtained without using data collected by other sensors, thereby reducing the amount of computation required. It can be seen that the application of the obstacle ranging solution provided by the above embodiment can improve the efficiency of obtaining the distance between the obstacle and the mobile robot, and can also reduce the cost of the mobile robot.

[0181] The present application also provides an electronic device, such as Figure 10As shown, it includes a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004.

[0182] Memory 1003, used for storing computer programs;

[0183] The processor 1001 is configured to implement the steps of the obstacle ranging method when executing the program stored in the memory 1003 .

[0184] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0185] The communication interface is used for communication between the above electronic device and other devices.

[0186] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0187] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0188] In another embodiment provided by the present application, a computer-readable storage medium is further provided, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the steps of any of the above obstacle ranging methods are implemented.

[0189] In another embodiment provided by the present application, a computer program product including instructions is further provided, which, when executed on a computer, enables the computer to execute any obstacle ranging method in the above embodiments.

[0190] In the obstacle ranging solution provided in the above embodiment, an image to be measured, captured by a monocular camera deployed on a mobile robot, is obtained; obstacles contained in the image to be measured are detected, and target pixels representing the position of the obstacles in the image to be measured are determined; the spatial coordinates corresponding to the target pixels are obtained using the pixel coordinates of the target pixels in the image to be measured and a preset coordinate transformation relationship, wherein the coordinate transformation relationship is the transformation relationship between the coordinates in the image captured by the monocular camera and the coordinates in a preset spatial coordinate system, which is obtained based on pre-calibrated camera parameters of the monocular camera, and the spatial coordinate system is a coordinate system established based on the mobile robot; and based on the obtained spatial coordinates, the distance of the obstacle relative to the mobile robot is determined. In this manner, only the monocular camera needs to be deployed on the mobile robot, and the pixel coordinates of the obstacle in the image captured by the monocular camera are determined. The coordinate transformation relationship is then used to obtain the coordinates of the obstacle in space, thereby obtaining the distance of the obstacle relative to the mobile robot. This distance can be obtained without using data collected by other sensors, thereby reducing the amount of computation required. It can be seen that the application of the obstacle ranging solution provided by the above embodiment can improve the efficiency of obtaining the distance between the obstacle and the mobile robot, and can also reduce the cost of the mobile robot.

[0191] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0192] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0193] Each embodiment in this specification is described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the mobile robot embodiment, device embodiment, electronic device embodiment, computer-readable storage medium embodiment, and computer program product embodiment are generally similar to the method embodiment, so their description is relatively simple. For related portions, reference can be made to the description of the method embodiment.

[0194] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. An obstacle ranging method, characterized in that: The method comprises: Obtaining motion information collected by a motion sensor deployed on the mobile robot, and obtaining a test image collected by a monocular camera deployed on the mobile robot when the motion information meets a preset stability condition; the stability condition is used to determine whether the current motion state of the mobile sensor is stable; Detecting an obstacle contained in the image to be tested, and determining a target pixel point used to represent the position of the obstacle in the image to be tested; wherein determining the target pixel point used to represent the position of the obstacle in the image to be tested includes: determining the midpoint of the lower boundary of the image area occupied by the obstacle in the image to be tested as the target pixel point used to represent the position of the obstacle in the image to be tested; Obtaining the spatial coordinates corresponding to the target pixel point using the pixel coordinates of the target pixel point in the image to be measured and a preset coordinate transformation relationship, wherein the coordinate transformation relationship is a transformation relationship between the coordinates in the image captured by the monocular camera and the coordinates in a preset spatial coordinate system, the coordinate transformation relationship is obtained based on pre-calibrated camera parameters of the monocular camera, and the spatial coordinate system is a coordinate system established based on the mobile robot; Based on the obtained spatial coordinates, the distance of the obstacle relative to the mobile robot is determined.

2. The method according to claim 1, characterized in that The method further comprises: determining an obstacle area where the obstacle is located; Based on the distance and the obstacle area, the mobile robot is controlled to avoid the obstacle area during operation.

3. The method according to claim 2, characterized in that The determining the obstacle area where the obstacle is located includes: Determining the pixel width of the obstacle in the image to be measured; The actual width of the obstacle is calculated using the pixel width, the spatial coordinates, and the camera parameters of the monocular camera, as the area width of the obstacle area where the obstacle is located.

4. The method according to claim 2, characterized in that The determining the obstacle area where the obstacle is located includes: Identifying the target category of obstacles contained in the image to be tested; According to the preset depth correspondence relationship between the depth and the category, the target depth corresponding to the target category is determined as the regional depth of the obstacle area where the obstacle is located.

5. The method according to claim 2, characterized in that The determining the obstacle area where the obstacle is located includes: When the distance is less than a preset close distance threshold, determining the obstacle area where the obstacle is located; and / or In a case where the obstacle is located in a non-edge area of ​​the image to be detected, an obstacle area where the obstacle is located is determined.

6. The method according to any one of claims 1 to 5, characterized in that The step of obtaining the spatial coordinates corresponding to the target pixel point by using the pixel coordinates of the target pixel point in the image to be measured and a preset coordinate conversion relationship includes: Dedistortion processing is performed on the target pixel point, and the spatial coordinates corresponding to the target pixel point are obtained by using the pixel coordinates of the target pixel point in the image to be measured after the dedistortion processing and a preset coordinate conversion relationship.

7. A mobile robot, characterized in that: The mobile robot includes a monocular camera, a processor, and a motion sensor, wherein: The monocular camera is used to: collect the image to be tested and send the image to be tested to the processor; The motion sensor is used to: collect motion information and send the motion information to the processor; The processor is configured to: receive the motion information, and if the motion information satisfies a preset stability condition, receive the image to be measured, detect an obstacle contained in the image to be measured, and determine a target pixel point used to represent the position of the obstacle in the image to be measured; obtain spatial coordinates corresponding to the target pixel point using pixel coordinates of the target pixel point in the image to be measured and a preset coordinate transformation relationship, wherein the coordinate transformation relationship is a transformation relationship between coordinates in the image captured by the monocular camera and coordinates in a preset spatial coordinate system, the coordinate transformation relationship being obtained based on pre-calibrated camera parameters of the monocular camera, and the spatial coordinate system being a coordinate system established based on the mobile robot; determine the distance of the obstacle relative to the mobile robot based on the obtained spatial coordinates; determining the target pixel point used to represent the position of the obstacle in the image to be measured includes determining a midpoint of a lower boundary of an image area occupied by the obstacle in the image to be measured as the target pixel point used to represent the position of the obstacle in the image to be measured; and the stability condition is used to determine whether the current motion state of the mobile sensor is stable.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 6 when executing a program stored in a memory.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps of any one of claims 1 to 6 are implemented.

Images