An image processing method and device, a mobile robot, and a storage medium
By combining binocular and depth cameras, the depth of obstacles can be identified and verified, solving the problem of poor perception caused by weak and repetitive textures, and improving the smoothness of mobile robot operation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU HIKROBOT TECH CO LTD
- Filing Date
- 2022-09-19
- Publication Date
- 2026-06-05
AI Technical Summary
The presence of weak and repetitive textures in physical space leads to poor depth camera perception capabilities, affecting the smoothness of mobile robot operation.
By acquiring left and right images using a binocular camera, the first and second regions of obstacles are determined. The depth of the obstacles is obtained by combining the depth image with a depth camera. The second depth in the depth image is verified using the first depth image to identify falsely detected obstacles and perform depth false detection processing.
It improved the camera's perception capabilities, reduced the impact of obstacle depth errors, and enhanced the smoothness of the mobile robot's operation.
Smart Images
Figure CN115375744B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of machine vision technology, and in particular to an image processing method, apparatus, mobile robot, and storage medium. Background Technology
[0002] With the development of machine vision technology, depth cameras are widely used in mobile robots to collect the depth of obstacles, enabling mobile robots to avoid obstacles during movement.
[0003] However, there are interference factors in physical space such as weak textures and repetitive textures. These factors lead to poor camera perception capabilities, which will cause the depth camera to capture the depth of obstacles incorrectly, thus affecting the smoothness of the mobile robot's operation. Summary of the Invention
[0004] The purpose of this application is to provide an image processing method, apparatus, mobile robot, and storage medium to identify falsely detected obstacles in images, improve the camera's perception capabilities, and enhance the smoothness of the mobile robot's operation. The specific technical solution is as follows:
[0005] In a first aspect, embodiments of this application provide an image processing method, the method comprising:
[0006] Determine the first and second regions of obstacles in the left and right images captured by the binocular camera;
[0007] Using the first region and the second region, determine the first depth of the obstacle to the depth camera;
[0008] Obtain the second depth of the third region corresponding to the obstacle in the depth map captured by the depth camera;
[0009] Based on the first depth and the second depth, depth false detection processing is performed on the obstacles in the depth map.
[0010] In some embodiments, the step of determining the first region and the second region of the obstacle in the left and right images captured by the binocular camera includes:
[0011] The left and right images captured by the binocular camera are segmented into instances to obtain the first and second masks of the obstacles.
[0012] In the left figure, the area corresponding to the first mask is the first region, and in the right figure, the area corresponding to the second mask is the second region.
[0013] In some embodiments, the number of the first region and the second region are both multiple;
[0014] The step of determining the first depth of the obstacle to the depth camera using the first region and the second region includes:
[0015] The multiple first regions in the left image are matched with the multiple second regions in the right image to obtain multiple sets of matching regions, each matching region including a first region and a second region;
[0016] Calculate the first depth from the obstacle to the depth camera for each matching region.
[0017] In some embodiments, before determining the first and second regions of obstacles in the left and right images captured by the binocular camera, the method further includes:
[0018] A fourth region of the ground in a specified image is determined, wherein the specified image is the left or right image captured by the binocular camera, or the specified image is an RGB image captured by an RGB camera, and the fourth region corresponds to the fifth region in the depth map;
[0019] Plane fitting is performed on multiple depth points within the fifth region to obtain the target plane where the ground is located;
[0020] In the sixth region outside the fifth region in the depth map, delete depth points whose distance from the target plane is less than a preset distance, and delete depth points in the fifth region.
[0021] In some embodiments, the step of determining a fourth region of the ground in a specified image includes:
[0022] Instance segmentation is performed on the specified image to obtain a third mask of the ground, which corresponds to a fourth region of the ground in the specified image.
[0023] In some embodiments, the step of performing depth false detection processing on the obstacle in the depth map based on the first depth and the second depth includes:
[0024] Calculate the difference between the second depth and the first depth. If the calculated difference is greater than a preset difference threshold, then perform depth false detection processing on the obstacle in the depth map; or,
[0025] Calculate the difference between the second depth and the first depth. If the ratio of the calculated difference to the target depth is greater than a preset ratio threshold, then perform depth false detection processing on the obstacle in the depth map. The target depth is either the first depth or the second depth.
[0026] In some embodiments, the step of performing depth false detection processing on the obstacles in the depth map includes:
[0027] Delete the depth points in the third region of the depth map; or
[0028] Update the depth of the depth points in the third region to the first depth.
[0029] In some embodiments, after performing depth false detection processing on the obstacles in the depth map, the method further includes:
[0030] Using the processed depth map, plan the movement route of the mobile robot;
[0031] Control the mobile robot to move according to the stated action route.
[0032] Secondly, embodiments of this application provide an image processing apparatus, the apparatus comprising:
[0033] The first determining module is used to determine the first and second regions of obstacles in the left and right images captured by the binocular camera;
[0034] The second determining module is used to determine the first depth of the obstacle to the depth camera using the first region and the second region;
[0035] The acquisition module is used to acquire the second depth of the third region corresponding to the obstacle in the depth map captured by the depth camera;
[0036] The processing module is configured to perform depth false detection processing on the obstacles in the depth map based on the first depth and the second depth.
[0037] In some embodiments, the first determining module is specifically used for:
[0038] The left and right images captured by the binocular camera are segmented into instances to obtain the first and second masks of the obstacles.
[0039] In the left figure, the area corresponding to the first mask is the first region, and in the right figure, the area corresponding to the second mask is the second region.
[0040] In some embodiments, the number of the first region and the second region are both multiple;
[0041] The second determining module is specifically used for:
[0042] The multiple first regions in the left image are matched with the multiple second regions in the right image to obtain multiple sets of matching regions, each matching region including a first region and a second region;
[0043] Calculate the first depth from the obstacle to the depth camera for each matching region.
[0044] In some embodiments, the apparatus further includes:
[0045] The third determining module is used to determine a fourth region of the ground in a specified image before determining the first and second regions of obstacles in the left and right images captured by the binocular camera. The specified image is the left or right image captured by the binocular camera, or the specified image is an RGB image captured by an RGB camera. The fourth region corresponds to the fifth region in the depth map.
[0046] The fitting module is used to perform planar fitting on multiple depth points in the fifth region to obtain the target plane where the ground is located.
[0047] The deletion module is used to delete depth points in the sixth region outside the fifth region in the depth map that are less than a preset distance from the target plane, and to delete depth points in the fifth region.
[0048] In some embodiments, the third determining module is specifically used for:
[0049] Instance segmentation is performed on the specified image to obtain a third mask of the ground, which corresponds to a fourth region of the ground in the specified image.
[0050] In some embodiments, the processing module is specifically used for:
[0051] Calculate the difference between the second depth and the first depth. If the calculated difference is greater than a preset difference threshold, then perform depth false detection processing on the obstacle in the depth map; or,
[0052] Calculate the difference between the second depth and the first depth. If the ratio of the calculated difference to the target depth is greater than a preset ratio threshold, then perform depth false detection processing on the obstacle in the depth map. The target depth is either the first depth or the second depth.
[0053] In some embodiments, the processing module is specifically used for:
[0054] Delete the depth points in the third region of the depth map; or
[0055] Update the depth of the depth points in the third region to the first depth.
[0056] In some embodiments, the apparatus further includes:
[0057] The planning module is used to plan the movement route of the mobile robot using the processed depth map after performing depth false detection processing on the obstacles in the depth map.
[0058] The control module is used to control the mobile robot to move according to the action route.
[0059] Thirdly, embodiments of this application provide a mobile robot, including a binocular camera, a depth camera, a processor, and a machine-readable storage medium;
[0060] The binocular camera captures the left and right images, and the depth camera captures the depth map;
[0061] The machine-readable storage medium stores machine-executable instructions that can be executed by the processor, which is prompted by the machine-executable instructions to implement any of the above-described image processing method steps based on the left image, the right image, and the depth image.
[0062] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the image processing method steps described above.
[0063] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the image processing method steps described above.
[0064] Beneficial effects of the embodiments in this application:
[0065] In the technical solution provided in this application embodiment, a binocular camera captures a left image and a right image, while a depth camera captures a depth map. Using the first and second regions of the same obstacle in the left and right images, the first depth from the obstacle to the depth camera is determined. The first depth is compared with the second depth of the third region corresponding to the same obstacle in the depth map to determine whether the obstacle has been falsely detected. In other words, the first depth is used to verify the second depth in the depth map to determine if the obstacle depth is incorrect, thus identifying falsely detected obstacles in the image. Based on the depth map that accurately identifies falsely detected obstacles, the mobile robot is controlled, reducing the impact of obstacle depth errors, improving the camera's perception capability, and enhancing the smoothness of the mobile robot's operation.
[0066] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description
[0067] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.
[0068] Figure 1 This is a schematic diagram of a first flowchart of an image processing method provided in an embodiment of this application;
[0069] Figure 2 This is a second flowchart illustrating the image processing method provided in the embodiments of this application;
[0070] Figure 3 This is a schematic diagram of a third process for the image processing method provided in the embodiments of this application;
[0071] Figure 4 This is a schematic diagram of the fourth process of the image processing method provided in the embodiments of this application;
[0072] Figure 5 A fifth flowchart illustrating the image processing method provided in this application embodiment;
[0073] Figure 6 A sixth flowchart illustrating the image processing method provided in this application embodiment;
[0074] Figure 7 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;
[0075] Figure 8 This is a schematic diagram of a mobile robot provided in an embodiment of this application. Detailed Implementation
[0076] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.
[0077] For ease of understanding, the terms appearing in the embodiments of this application are explained below.
[0078] Depth camera: also known as a 3D camera, it can detect the depth distance of the shooting space.
[0079] Perception: The process by which consciousness perceives, senses, pays attention to, and understands information from the internal and external world.
[0080] False detection of obstacles: obstacles assigned the wrong depth or obstacles that are mistakenly identified as non-obstacles.
[0081] In physical space, there exist objects with weak textures and objects with repetitive textures. Due to the existence of objects with weak textures and objects with repetitive textures, such as distant obstacles being objects with weak textures, or distant obstacles having textures that are repetitive with those of nearby obstacles, depth cameras will assign a closer depth to distant obstacles. This will cause mobile robots to repeatedly perform start and stop operations in open scenes, affecting the smoothness of the mobile robot's operation.
[0082] To identify falsely detected obstacles in images and improve the smoothness of mobile robot operation, this application provides an image processing method. This method can be applied to a depth camera, a server or terminal connected to the depth camera, or a mobile robot equipped with a depth camera. For ease of understanding, the following description uses a mobile robot as the execution subject, but this is not intended to limit the scope. The mobile robot can be an AGV (Automated Guided Vehicle) or other types of mobile robots, and is not limited thereto.
[0083] In the image processing method provided in this application embodiment, a binocular camera captures a left image and a right image, and a depth camera captures a depth map. The mobile robot uses a first region and a second region of the same obstacle in the left and right images to determine the first depth of the obstacle from the depth camera. By comparing the first depth with the second depth of the third region corresponding to the same obstacle in the depth map, it determines whether the obstacle has been falsely detected. That is, the first depth is used to verify the second depth in the depth map to determine if the obstacle depth is incorrect, thus identifying falsely detected obstacles in the image. Based on the depth map that accurately identifies falsely detected obstacles, the mobile robot is controlled, reducing the impact of obstacle depth errors, improving the camera's perception capability, and enhancing the smoothness of the mobile robot's operation.
[0084] The image processing method provided in this application will be described in detail below through specific embodiments.
[0085] See Figure 1 , Figure 1 This is a first flowchart illustrating an image processing method provided in an embodiment of this application. The method includes the following steps:
[0086] Step S11: Determine the first and second regions of obstacles in the left and right images captured by the binocular camera.
[0087] A binocular camera consists of a left camera and a right camera. The image captured by the left camera is the left image, and the image captured by the right camera is the right image.
[0088] When processing images and detecting false obstacles, the mobile robot performs target detection on both the left and right images, identifying the first region of the obstacle in the left image and the second region of the same obstacle in the right image. The target detection algorithms include, but are not limited to, CNN (Convolutional Neural Network), DMP (Deformable Parts Model), and SSD (Single Shot MultiBox Detector).
[0089] In this embodiment, the left and right images may include one or more obstacles. When the left and right images include multiple obstacles, the mobile robot detects the left and right images respectively, and can obtain multiple first regions and multiple second regions respectively. For ease of understanding, the following description uses one first region and one second region as an example, and is not intended to be limiting.
[0090] Step S12: Using the first region and the second region, determine the first depth of the obstacle to the depth camera.
[0091] In this embodiment, the depth camera can be a separate camera from the stereo camera. This depth camera can employ technologies such as laser or radar to collect depth information and generate a depth map. This embodiment does not specifically limit the implementation method of the depth camera. In this case, the mobile robot can determine the depth from the obstacle to the stereo camera using the first and second regions. Combining this with the positional relationship between the depth camera and the stereo camera, the robot can determine the depth from the obstacle to the depth camera, i.e., the first depth. For example, the mobile robot determines the depth from the obstacle to the stereo camera as z1, and the positional relationship between the depth camera and the stereo camera is that the distance between them is d, and the depth camera is farther from the obstacle relative to the stereo camera. Therefore, the mobile robot can determine the depth z2 from the obstacle to the depth camera as z1+d.
[0092] In this embodiment, the depth camera can be implemented using the aforementioned binocular camera. That is, the binocular camera uses the left and right images to perform depth calculations and generate a depth map, thus functioning as a depth camera. In this case, after determining the first and second regions, the mobile robot can use the parallax and focal length between the first and second regions, as well as the distance between the optical centers of the left and right cameras, to determine the depth of the obstacle from the depth camera, i.e., the first depth.
[0093] For example, a mobile robot can use the following formula (1) to determine the first depth z of the obstacle to the depth camera:
[0094]
[0095] In formula (1), z represents the first depth from the obstacle to the depth camera, f represents the focal length, b represents the distance between the optical centers of the left and right cameras, and d represents the parallax between the first region and the second region of the obstacle.
[0096] Step S13: Obtain the second depth of the third region corresponding to the obstacle in the depth map captured by the depth camera.
[0097] The mapping relationship between the left image, the right image, and the depth map means that the pixels in the left image can be mapped to the depth points in the depth map according to the mapping relationship between the left image and the depth map, and the pixels in the right image can be mapped to the depth points in the depth map according to the mapping relationship between the right image and the depth map.
[0098] In this embodiment, after determining the first and second regions, the mobile robot can determine the third region corresponding to the first and second regions in the depth map, i.e., the third region corresponding to the same obstacle, according to the above mapping relationship. Each depth point in the depth map has depth information. After determining the third region in the depth map, the mobile robot can obtain the second depth based on the depth of the depth points in the third region of the depth map.
[0099] The mobile robot obtains the second depth based on the depth of the depth points in the third region of the depth map. This can be achieved by obtaining the maximum depth from the depth points in the third region of the depth map and using it as the second depth.
[0100] The mobile robot obtains the second depth based on the depth of the depth points in the third region of the depth map. Alternatively, it can obtain the minimum depth from the depth points in the third region of the depth map and use it as the second depth.
[0101] The mobile robot obtains a second depth based on the depth of the depth points in the third region of the depth map. Alternatively, it can calculate the average depth of the depth points in the third region of the depth map as the second depth.
[0102] In this embodiment of the application, the mobile robot may also use other methods to obtain the second depth, and there is no limitation on this.
[0103] Step S14: Perform depth false detection processing on obstacles in the depth map based on the first depth and the second depth.
[0104] In this embodiment of the application, the mobile robot compares the first depth and the second depth to determine whether the obstacle is a false detection obstacle. That is, the first depth is used to verify the second depth in the depth map to determine whether the depth of the obstacle is incorrect, and then the depth false detection processing is performed on the obstacle in the depth map.
[0105] In some embodiments, the mobile robot can preset a difference threshold, i.e., a pre-defined difference threshold. This pre-defined difference threshold can be set according to actual needs; for example, it can be 0.5 meters, 1 meter, or 2 meters, etc., without limitation. Having obtained the first and second depths, the mobile robot can calculate the difference between the second and first depths and determine whether the calculated difference is greater than the pre-defined difference threshold, thus verifying the depth information of obstacles in the depth map. If the calculated difference is less than or equal to the pre-defined difference threshold, it indicates that the obstacle corresponding to the third region is not a falsely detected obstacle, meaning the depth of the verified obstacle is correct, and the mobile robot does not need to perform any further processing. If the calculated difference is greater than the pre-defined difference threshold, it indicates that the obstacle corresponding to the third region is a falsely detected obstacle, meaning the depth of the verified obstacle is incorrect, and the mobile robot performs depth false detection processing on the obstacles in the depth map.
[0106] In other embodiments, the mobile robot can pre-set preset difference thresholds corresponding to different depths. Having obtained a first depth and a second depth, the mobile robot can calculate the difference between the second depth and the first depth, and obtain the preset difference threshold corresponding to the first depth; it then determines whether the calculated difference is greater than the obtained preset difference threshold, thus verifying the depth information of obstacles in the depth map. If the calculated difference is less than or equal to the obtained preset difference threshold, it indicates that the obstacle corresponding to the third region is not a falsely detected obstacle, meaning the depth of the verified obstacle is correct, and the mobile robot does not need to perform any further processing. If the calculated difference is greater than the obtained preset difference threshold, it indicates that the obstacle corresponding to the third region is a falsely detected obstacle, meaning the depth of the verified obstacle is incorrect, and the mobile robot performs depth false detection processing on the obstacles in the depth map.
[0107] In some embodiments, the mobile robot can preset a ratio threshold, i.e., a preset ratio threshold. This preset ratio threshold can be set according to actual needs; for example, it can be 0.001, 0.002, or 0.005, etc., without limitation. Having obtained the first and second depths, the mobile robot can calculate the difference between the second and first depths, and calculate the ratio of this difference to the target depth. It then determines whether the calculated ratio is greater than the preset ratio threshold, thus verifying the depth information of obstacles in the depth map. If the calculated ratio is less than or equal to the preset ratio threshold, it indicates that the obstacle corresponding to the third region is not a falsely detected obstacle, meaning the depth of the verified obstacle is correct, and the mobile robot does not need to perform any further processing. If the calculated ratio is greater than the preset ratio threshold, it indicates that the obstacle corresponding to the third region is a falsely detected obstacle, meaning the depth of the verified obstacle is incorrect, and the mobile robot performs depth false detection processing on the obstacles in the depth map. The target depth can be either the first depth or the second depth.
[0108] In this embodiment, the depth misdetection processing of obstacles in the depth map in step S14 can be performed by deleting depth points in the third region of the depth map. For example, when it is determined that a distant obstacle is assigned a closer depth, the mobile robot can delete the depth points in the third region, avoiding the problem of the mobile robot repeatedly performing start-stop operations in an open scene due to the misdetected obstacle being too close to the mobile robot, thus improving the smoothness of the mobile robot's operation.
[0109] The depth misdetection processing performed on obstacles in the depth map in step S14 above can also be achieved by updating the depth of the depth points in the third region to the first depth. For example, when a nearby obstacle is determined to be assigned a greater depth, the mobile robot can update the depth of the depth points in the third region to the first depth. This avoids the problem of the mobile robot colliding with an obstacle due to a misdetected obstacle being too far away, thus forcing the mobile robot to stop running and improving the smoothness of the mobile robot's operation.
[0110] In this embodiment of the application, the depth false detection processing of obstacles in the depth map in step S14 can also be implemented in other ways, such as outputting alarm information indicating false obstacle detection, and there is no limitation on this.
[0111] In the image processing method provided in this application embodiment, a binocular camera captures a left image and a right image, and a depth camera captures a depth map. The mobile robot uses a first region and a second region of the same obstacle in the left and right images to determine the first depth of the obstacle from the depth camera. By comparing the first depth with the second depth of the third region corresponding to the same obstacle in the depth map, it determines whether the obstacle has been falsely detected. That is, the first depth is used to verify the second depth in the depth map to determine if the obstacle depth is incorrect, thus identifying falsely detected obstacles in the image. Based on the depth map that accurately identifies falsely detected obstacles, the mobile robot is controlled, reducing the impact of obstacle depth errors, improving the camera's perception capability, and enhancing the smoothness of the mobile robot's operation.
[0112] In some embodiments, such as Figure 2 As shown, an image processing method is also provided, which may include steps S21-S24.
[0113] Step S21: Perform instance segmentation on the left and right images captured by the binocular camera to obtain the first mask and the second mask of the obstacle. The area corresponding to the first mask in the left image is the first region, and the area corresponding to the second mask in the right image is the second region.
[0114] In this embodiment, the mobile robot performs instance segmentation on the left image to obtain a first mask of obstacles. Since the first mask is obtained by instance segmentation of the left image, the mobile robot can obtain the region corresponding to the first mask, i.e., the first region, in the left image.
[0115] In addition, the mobile robot performs instance segmentation on the right image to obtain a second mask for the obstacles. Since the second mask is obtained from instance segmentation of the right image, the mobile robot can obtain the region corresponding to the second mask, i.e., the second region, in the right image.
[0116] In some embodiments, if the left and right images include multiple obstacles, the mobile robot can perform instance segmentation on the left and right images respectively to obtain multiple first masks and multiple second masks. One first mask corresponds to one first region, and one second mask corresponds to one second region. For the first and second regions of the same obstacle, the mobile robot can perform steps S22-S24.
[0117] Step S22: Using the first region and the second region, determine the first depth of the obstacle from the depth camera. See the relevant description in step S12 above for details.
[0118] Step S23: Obtain the second depth of the third region corresponding to the obstacle in the depth map captured by the depth camera. See the relevant description in step S13 above for details.
[0119] Step S24: Based on the first depth and the second depth, perform depth false detection processing on obstacles in the depth map. See the relevant description in step S14 above for details.
[0120] In the technical solution provided in this application, the mobile robot determines the mask of obstacles in the left and right images through instance segmentation. Based on the obstacle mask, the interference of background color on the determination of foreground targets can be reduced, that is, the obstacle region in the left and right images can be accurately determined. Subsequently, the mobile robot uses the obstacle region determined by instance segmentation to verify the depth information of the obstacles, which can effectively identify falsely detected obstacles, further reduce the impact of depth errors of obstacles, improve the camera's perception capability, and further improve the smoothness of the mobile robot's operation. This greatly increases the competitiveness of the mobile robot and the depth camera.
[0121] In some embodiments, the number of first regions and second regions are both multiple, that is, the mobile robot determines multiple first regions and multiple second regions. In this case, embodiments of this application also provide an image processing method, such as... Figure 3 As shown, it may include the following steps S31-S35.
[0122] Step S31: Determine the first and second regions of the obstacles in the left and right images captured by the binocular camera. See steps S11 and S21 for details.
[0123] Step S32: Match multiple first regions in the left image with multiple second regions in the right image to obtain multiple sets of matching regions. Each matching region includes a first region and a second region.
[0124] In this embodiment of the application, the mobile robot can perform instance segmentation on the left and right images respectively to obtain multiple first masks and multiple second masks. Based on this, multiple first regions and multiple second regions corresponding to the multiple first masks and multiple second masks can be obtained.
[0125] In some embodiments, for each first region, the mobile robot can match the first region with a plurality of second regions respectively to obtain a second region that matches the first region, and the first region and the matched second region are considered as a set of matching regions.
[0126] In other embodiments, for each second region, the mobile robot can match the second region with a plurality of first regions respectively to obtain a first region that matches the second region, and the second region and the matched first region are a set of matching regions.
[0127] In this embodiment, the mobile robot may also use other methods to determine multiple sets of matching regions, and there is no limitation on this. The mobile robot may use at least one of the following methods to determine whether two regions match:
[0128] Method 1: If the similarity between region 1 and region 2 is greater than the preset similarity threshold, and the similarity between region 1 and region 2 is the highest among the similarities between region 1 and multiple regions, then region 1 and region 2 are determined to be a match.
[0129] Method 2: Map region 1 and region 2 to a specified coordinate system. If the distance between region 1 and region 2 in the specified coordinate system is less than a preset threshold, then region 1 and region 2 are considered to be a match.
[0130] In this embodiment of the application, the mobile robot may also use other methods to determine whether two regions match, and there is no limitation on this.
[0131] Step S33: Calculate the first depth from the obstacle to the depth camera for each matching area.
[0132] After determining multiple sets of matching regions, for each set of matching regions, the mobile robot can use the parallax and focal length between the first and second regions in that set of matching regions, as well as the distance between the optical centers of the left and right cameras, to determine the depth of the corresponding obstacle to the depth camera, i.e., the first depth.
[0133] Step S34: Obtain the second depth of the third region corresponding to the obstacle in the depth map acquired by the depth camera. See the relevant description in step S13 above for details.
[0134] Step S35: Based on the first depth and the second depth, perform depth false detection processing on obstacles in the depth map. See the relevant description in step S14 above for details.
[0135] For each set of matching areas, the mobile robot can execute steps S34-S35 to determine whether the obstacle corresponding to the set of matching areas is a false detection obstacle, and if the obstacle corresponding to the set of matching areas is a false detection obstacle, perform the corresponding deep false detection processing.
[0136] In the technical solution provided in this application, the mobile robot can simultaneously verify the depth information of multiple obstacles, improving verification efficiency, further reducing the impact of depth errors in obstacles, enhancing the camera's perception capabilities, and further improving the smoothness of the mobile robot's operation. This significantly increases the competitiveness of the mobile robot and the depth camera.
[0137] In some embodiments, such as Figure 4 As shown, an image processing method is also provided, which may include steps S41-S43.
[0138] Step S41: Determine the fourth region of the ground in the specified image. The specified image is either the left or right image captured by a stereo camera, or an RGB image captured by an RGB camera. The fourth region corresponds to the fifth region in the depth map.
[0139] In this embodiment, the binocular camera can be integrated with the RGB camera on the same device or on different devices; there is no limitation on this.
[0140] Before performing step S11 to determine the first and second regions of obstacles in the left and right images captured by the binocular cameras, the mobile robot can determine the ground region in the specified image, i.e., the fourth region. After determining the fourth region, the mobile robot can determine the fifth region corresponding to the fourth region in the depth map, i.e., the fifth region corresponding to the ground, according to the mapping relationship between the specified image and the depth map. The mapping relationship between the specified image and the depth map can be obtained by calibrating the depth camera and the RGB camera.
[0141] Step S42: Perform plane fitting on multiple depth points in the fifth region to obtain the target plane where the ground is located.
[0142] In this embodiment of the application, the mobile robot can use plane fitting algorithms such as least squares algorithm and ICP (Iterative Closest Point) to perform plane fitting on multiple depth points in the fifth region to obtain the plane equation, that is, the target plane where the ground is located.
[0143] Step S43: In the sixth region outside the fifth region of the depth map, delete depth points whose distance from the target plane is less than a preset distance, and delete depth points in the fifth region.
[0144] After obtaining the target plane where the ground is located, the mobile robot can traverse the depth points in the sixth region outside the fifth region of the depth map and determine whether the distance between the traversed depth point and the target plane is less than a preset distance. If the distance between the traversed depth point and the target plane is less than the preset distance, the mobile robot can determine that the depth point is a ground point and delete the depth point from the depth map. If the distance between the traversed depth point and the target plane is greater than or equal to the preset distance, the mobile robot can determine that the depth point is not a ground point and does not perform any processing on the depth point in the depth map.
[0145] Additionally, the fifth region corresponds to the ground plane, meaning all depth points within the fifth region are ground points. The mobile robot removed the depth points from the fifth region of the depth map.
[0146] In the technical solution provided in this application embodiment, the mobile robot removes ground points from the depth map, reducing the probability of misdetecting low-height obstacles as ground, i.e., reducing the probability of false ground detection. Furthermore, since the mobile robot removes ground points from the depth map, there is no need for strict obstacle size limitations when detecting ground obstacles, improving the detection capability of low-lying obstacles, enhancing the performance of ground obstacle detection, and strengthening the depth camera's perception capability.
[0147] In some embodiments, such as Figure 5 As shown, an image processing method is also provided, which may include steps S51-S53.
[0148] Step S51: Perform instance segmentation on the specified image to obtain a third mask for the ground. The third mask corresponds to the fourth region of the ground in the specified image.
[0149] In this embodiment, the mobile robot performs instance segmentation on a specified image to obtain a third mask for the ground. Since the third mask is obtained by instance segmentation of the specified image, the mobile robot can obtain the region corresponding to the third mask in the specified image, i.e., the fourth region.
[0150] Step S52 involves performing plane fitting on multiple depth points within the fifth region to obtain the target plane where the ground is located. See step S42 for a detailed description.
[0151] Step S53: In the sixth region outside the fifth region of the depth map, delete depth points whose distance from the target plane is less than a preset distance, and delete depth points within the fifth region. See the relevant description in step S43 for details.
[0152] In the technical solution provided in this application embodiment, the mobile robot determines a ground mask in a specified image through instance segmentation. Based on the ground mask, the interference of background color on the determination of foreground targets can be reduced, thus accurately determining the ground region in the specified image. Subsequently, by utilizing the ground region determined through instance segmentation, the mobile robot can accurately remove ground points in the depth map, further reducing the probability of false ground detection, further improving the detection capability of low-lying obstacles, enhancing the performance of detecting obstacles close to the ground, and strengthening the perception capability of the depth camera.
[0153] In some embodiments, after performing depth false detection processing on obstacles in the depth map, the mobile robot uses the processed depth map to plan its movement route and controls the mobile robot to move according to the movement route.
[0154] In the technical solution provided in this application embodiment, the mobile robot eliminates falsely detected obstacles. That is, the depth map retains obstacles with correct depth information. By using obstacles with correct depth information to plan the movement route of the mobile robot, the movement route can correctly avoid obstacles, reduce the probability of repeatedly performing start and stop operations in open scenes, improve the smoothness of the mobile robot's operation, and reduce the probability of the mobile robot colliding with obstacles.
[0155] The following is combined Figure 6 The image processing method shown herein provides a detailed description of the image processing method provided in the embodiments of this application. The depth camera is implemented using a stereo camera. The stereo camera acquires a left image and a right image, and determines a depth map based on the left and right images.
[0156] In step S61, the mobile robot performs instance segmentation on the left and right images respectively to obtain the third mask of the ground, the first mask of multiple obstacles, and the second mask of multiple obstacles.
[0157] The third mask corresponds to the fourth region of the ground in the left or right image, and the fourth region corresponds to the fifth region in the depth image; the region corresponding to the first mask in the left image is the first region, the region corresponding to the second mask in the right image is the second region, and the region of the obstacle in the depth image is the third region.
[0158] In step S62, the mobile robot performs plane fitting on multiple depth points in the fifth region to obtain the target plane where the ground is located.
[0159] Step S63: In the sixth region outside the fifth region of the depth map, the mobile robot deletes depth points whose distance from the target plane is less than a preset distance, and deletes depth points in the fifth region.
[0160] In step S64, the mobile robot matches multiple first regions in the left image with multiple second regions in the right image to obtain multiple sets of matching regions.
[0161] In step S65, the mobile robot determines the first depth of the obstacle corresponding to each matching area to the depth camera.
[0162] Step S66: The mobile robot obtains the second depth of the third region corresponding to each set of matching regions.
[0163] In step S67, the mobile robot determines whether the obstacle corresponding to each set of matching regions is a falsely detected obstacle based on the second depth and the first depth. If yes, proceed to step S68; otherwise, end the image processing operation for that set of matching regions.
[0164] Step S68: The mobile robot deletes the depth points in the third region corresponding to the matching region in the depth map.
[0165] The descriptions of steps S61-S68 above are relatively simple; please refer to the above for details. Figures 1-5 The relevant descriptions will not be repeated here.
[0166] In the technical solution provided in this application embodiment, the mobile robot effectively eliminates ground points in the depth map, reduces false ground detections, and enhances the performance of detecting obstacles close to the ground. Furthermore, the mobile robot eliminates false detections caused by typical obstacles in the scene, improving the smoothness of using the depth camera and increasing the product's competitiveness.
[0167] Corresponding to the image processing method described above, embodiments of this application also provide an image processing apparatus, such as... Figure 7 As shown, it includes:
[0168] The first determining module 71 is used to determine the first region and the second region of obstacles in the left and right images captured by the binocular camera;
[0169] The second determining module 72 is used to determine the first depth of the obstacle to the depth camera using the first region and the second region;
[0170] The acquisition module 73 is used to acquire the second depth of the third region corresponding to the obstacle in the depth map acquired by the depth camera;
[0171] The processing module 74 is used to perform depth false detection processing on obstacles in the depth map based on the first depth and the second depth.
[0172] In some embodiments, the first determining module 71 may be specifically used for:
[0173] The left and right images captured by the binocular camera are segmented into instances to obtain the first and second masks of the obstacles.
[0174] In the left image, the area corresponding to the first mask is the first region, and in the right image, the area corresponding to the second mask is the second region.
[0175] In some embodiments, the number of first regions and second regions are both multiple;
[0176] The second determining module 72 can be specifically used for:
[0177] Match multiple first regions in the left image with multiple second regions in the right image to obtain multiple sets of matching regions. Each matching region includes one first region and one second region.
[0178] Calculate the first depth from the obstacle to the depth camera for each matching region.
[0179] In some embodiments, the image processing apparatus may further include:
[0180] The third determining module is used to determine the fourth region of the ground in a specified image before determining the first and second regions of obstacles in the left and right images captured by the binocular camera. The specified image is either the left or right image captured by the binocular camera, or the specified image is an RGB image captured by an RGB camera. The fourth region corresponds to the fifth region in the depth map.
[0181] The fitting module is used to perform plane fitting on multiple depth points in the fifth region to obtain the target plane where the ground is located.
[0182] The deletion module is used to delete depth points in the sixth region outside the fifth region of the depth map that are less than a preset distance from the target plane, and to delete depth points in the fifth region.
[0183] In some embodiments, the third determining module may specifically be used for:
[0184] Perform instance segmentation on the specified image to obtain a third mask for the ground, which corresponds to a fourth region of the ground in the specified image.
[0185] In some embodiments, the processing module 74 may be specifically used for:
[0186] Calculate the difference between the second depth and the first depth. If the calculated difference is greater than a preset difference threshold, perform depth false detection processing on obstacles in the depth map; or,
[0187] Calculate the difference between the second depth and the first depth. If the ratio of the calculated difference to the target depth is greater than a preset ratio threshold, then perform depth false detection processing on the obstacles in the depth map, and the target depth is either the first depth or the second depth.
[0188] In some embodiments, the processing module 74 may be specifically used for:
[0189] Delete the depth point in the third region of the depth map; or
[0190] Update the depth of the depth points in the third region to the first depth.
[0191] In some embodiments, the image processing apparatus may further include:
[0192] The planning module is used to plan the movement route of the mobile robot after performing depth false detection processing on the obstacles in the depth map and using the processed depth map.
[0193] The control module is used to control the mobile robot to move according to the action route.
[0194] In the image processing apparatus provided in this application embodiment, a binocular camera captures a left image and a right image, and a depth camera captures a depth map. A mobile robot uses a first region and a second region of the same obstacle in the left and right images to determine the first depth of the obstacle from the depth camera. By comparing the first depth with the second depth of the third region corresponding to the same obstacle in the depth map, it determines whether the obstacle has been falsely detected. That is, the first depth is used to verify the second depth in the depth map to determine if the obstacle depth is incorrect, thus identifying falsely detected obstacles in the image. Based on the depth map that accurately identifies falsely detected obstacles, the mobile robot is controlled, reducing the impact of obstacle depth errors, improving the camera's perception capability, and enhancing the smoothness of the mobile robot's operation.
[0195] Corresponding to the image processing method described above, this application also provides a mobile robot, such as... Figure 8 As shown, it includes a binocular camera 81, a depth camera 82, a processor 83, and a machine-readable storage medium 84;
[0196] The binocular camera 81 captures the left and right images, and the depth camera 82 captures the depth map.
[0197] Machine-readable storage medium 84 stores machine-executable instructions that can be executed by processor 83, which in turn cause processor 83 to: implement the above based on the left image, right image, and depth image. Figures 1-6Any of the steps in the image processing method described above.
[0198] Machine-readable storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the machine-readable storage medium may also be at least one storage device located remotely from the aforementioned processor.
[0199] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0200] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the above-described... Figures 1-6 Any of the steps in the image processing method described above.
[0201] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the above-described... Figures 1-6 Any of the steps in the image processing method described above.
[0202] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).
[0203] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0204] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of devices, mobile robots, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0205] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.
Claims
1. An image processing method, characterized in that, Applied to a mobile robot, the mobile robot including a binocular camera and a depth camera, the method includes: The left and right images captured by the binocular camera are segmented to obtain a third mask for the ground, a first mask for the obstacle, and a second mask for the obstacle. The third mask corresponds to a fourth region of the ground in the left or right image. The fourth region corresponds to a fifth region in the depth image captured by the depth camera. The first mask in the left image corresponds to a first region, and the second mask in the right image corresponds to a second region. Plane fitting is performed on multiple depth points in the fifth region to obtain the target plane where the ground is located; in the sixth region outside the fifth region in the depth map, depth points whose distance from the target plane is less than a preset distance are deleted, and depth points in the fifth region are also deleted. Using the first region corresponding to the first mask and the second region corresponding to the second mask, the first depth of the obstacle to the depth camera is determined; the second depth of the third region corresponding to the obstacle in the depth map acquired by the depth camera is obtained; the second depth in the depth map is verified using the first depth; if the depth of the obstacle is found to be incorrect, the obstacle in the depth map is subjected to depth false detection processing. The mobile robot is controlled to move based on the processed depth map.
2. The method according to claim 1, characterized in that, The number of the first mask and the second mask are both multiple; The step of determining the first depth of the obstacle to the depth camera using the first region corresponding to the first mask and the second region corresponding to the second mask includes: The multiple first regions corresponding to the multiple first masks in the left figure are matched with the multiple second regions corresponding to the multiple second masks in the right figure to obtain multiple sets of matching regions. Each matching region includes a first region and a second region. Using the first region corresponding to the first mask and the second region corresponding to the second mask in each matching region, the depth from the obstacle to the stereo camera in each matching region is determined. Then, combined with the positional relationship between the depth camera and the stereo camera, the first depth from the obstacle to the depth camera in each matching region is calculated.
3. The method according to any one of claims 1-2, characterized in that, The step of verifying the second depth in the depth map using the first depth, and performing depth misdetection processing on the obstacle in the depth map if the verification finds that the depth of the obstacle is incorrect, includes: Calculate the difference between the second depth and the first depth. If the calculated difference is greater than a preset difference threshold, then perform depth false detection processing on the obstacle in the depth map; or, Calculate the difference between the second depth and the first depth. If the ratio of the calculated difference to the target depth is greater than a preset ratio threshold, then perform depth false detection processing on the obstacle in the depth map. The target depth is either the first depth or the second depth.
4. The method according to claim 3, characterized in that, The step of performing depth false detection processing on the obstacles in the depth map includes: Delete the depth points in the third region of the depth map; or Update the depth of the depth points in the third region to the first depth.
5. The method according to any one of claims 1-2, characterized in that, The step of controlling the movement of the mobile robot based on the processed depth map includes: Using the processed depth map, plan the movement route of the mobile robot; Control the mobile robot to move according to the stated action route.
6. An image processing apparatus, characterized in that, Applied to a mobile robot, the mobile robot including a binocular camera and a depth camera, the device includes: The first determining module is used to perform instance segmentation on the left and right images acquired by the binocular camera to obtain a third mask for the ground, a first mask for obstacles, and a second mask for obstacles. The third mask corresponds to a fourth region of the ground in the left or right image, and the fourth region corresponds to a fifth region in the depth image acquired by the depth camera. The first mask in the left image corresponds to a first region, and the second mask in the right image corresponds to a second region. The fitting module is used to perform planar fitting on multiple depth points in the fifth region to obtain the target plane where the ground is located. The deletion module is used to delete depth points in the sixth region outside the fifth region in the depth map that are less than a preset distance from the target plane, and to delete depth points in the fifth region. The second determining module is used to determine the first depth of the obstacle to the depth camera using the first region corresponding to the first mask and the second region corresponding to the second mask; The acquisition module is used to acquire the second depth of the third region corresponding to the obstacle in the depth map captured by the depth camera; The processing module is used to verify the second depth in the depth map using the first depth. If the verification finds that the depth of the obstacle is incorrect, then the obstacle in the depth map is subjected to depth misdetection processing. The control module is used to control the movement of the mobile robot based on the processed depth map.
7. The apparatus according to claim 6, characterized in that, The number of first masks and second masks are both multiple; the second determining module is specifically used to match multiple first regions corresponding to the multiple first masks in the left image with multiple second regions corresponding to the multiple second masks in the right image to obtain multiple sets of matching regions, each matching region including a first region and a second region; using the first region corresponding to the first mask and the second region corresponding to the second mask in each set of matching regions, the depth from the obstacle to the stereo camera in each set of matching regions is determined, and then, combined with the positional relationship between the depth camera and the stereo camera, the first depth from the obstacle to the depth camera in each set of matching regions is calculated; or... The processing module is specifically used to calculate the difference between the second depth and the first depth. If the calculated difference is greater than a preset difference threshold, then the obstacle in the depth map is subjected to depth false detection processing. Alternatively, the difference between the second depth and the first depth is calculated. If the ratio of the calculated difference to the target depth is greater than a preset ratio threshold, then depth false detection processing is performed on the obstacle in the depth map, where the target depth is either the first depth or the second depth; or... The processing module is specifically used to delete depth points in the third region of the depth map; Alternatively, the depth of the depth points in the third region can be updated to the first depth; or... The device further includes: The control module is used to plan the movement route of the mobile robot using the processed depth map after performing depth false detection processing on the obstacles in the depth map. Control the mobile robot to move according to the stated action route.
8. A mobile robot, characterized in that, Includes a binocular camera, a depth camera, a processor, and a machine-readable storage medium; The binocular camera captures the left and right images, and the depth camera captures the depth map; The machine-readable storage medium stores machine-executable instructions that can be executed by the processor, which is prompted by the machine-executable instructions to implement the steps of the method according to any one of claims 1-5 based on the left image, the right image, and the depth image.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-5.