Image-based depth determination method, apparatus, binocular camera and mobile robot

Abstract

The application provides an image-based depth determination method and device, a binocular camera and a mobile robot. In the perception process of an object, a visible light image and a depth image corresponding to the visible light image are acquired. Instance detection is performed on the visible light image to obtain at least one instance segmentation region. For each instance segmentation region, a corresponding obstacle mask region and a plurality of obstacle mask sub-regions are determined in the depth image. For each instance segmentation region, a depth value of the instance segmentation region is determined according to at least one region of the corresponding obstacle mask region and the plurality of obstacle mask sub-regions. The depth value of each instance segmentation region in the image, i.e. the distance between the object and the camera, can be determined, thereby realizing image-based distance measurement.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual perception, and in particular to an image-based depth determination method and device, a binocular camera, and a mobile robot. BACKGROUND

[0002] An AGV (Automated Guided Vehicle) is also known as an unmanned carrier or a mobile robot. It is a kind of unmanned vehicle equipped with an automatic guiding system, which can ensure that the mobile robot can automatically travel along a predetermined route without the need for manual navigation, and automatically transport goods or materials from the starting point to the destination. In the process of automatic travel of the mobile robot, obstacles will inevitably be encountered. In order to ensure the safe travel of the mobile robot, the obstacle avoidance function of the mobile robot needs to be enhanced. The key to the obstacle avoidance function lies in distance measurement, i.e. measuring the distance between the mobile robot and the obstacle. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide an image-based depth determination method and device, a binocular camera, and a mobile robot, to realize image-based distance measurement. The specific technical solutions are as follows:

[0004] In a first aspect, the embodiments of the present application provide an image-based depth determination method, which comprises:

[0005] obtaining a visible light image and a depth image corresponding to the visible light image;

[0006] performing instance detection on the visible light image to obtain at least one instance segmentation region;

[0007] For each instance segmentation region, determining an obstacle mask region corresponding to the instance segmentation region and a plurality of obstacle mask sub-regions in the depth image;

[0008] For each instance segmentation region, determining a depth value of the instance segmentation region according to at least one region of the obstacle mask region corresponding to the instance segmentation region and the plurality of obstacle mask sub-regions.

[0009] In a possible implementation, for each instance segmentation region, determining an obstacle mask region corresponding to the instance segmentation region and a plurality of obstacle mask sub-regions in the depth image comprises:

[0010] For each instance segmentation region, mapping the instance segmentation region to the depth image to obtain an obstacle mask region corresponding to the instance segmentation region;

[0011] The instance segmentation region corresponding obstacle mask region is divided into a plurality of sub-regions, and a plurality of obstacle mask sub-regions corresponding to the instance segmentation region are obtained.

[0012] In a possible implementation, the instance segmentation region corresponding obstacle mask region is divided into a plurality of sub-regions, and a plurality of obstacle mask sub-regions corresponding to the instance segmentation region are obtained, including:

[0013] The instance segmentation region corresponding obstacle mask region is divided into a plurality of sub-regions with different sizes, taking the center of the instance segmentation region corresponding obstacle mask region as the center of each sub-region, and a plurality of obstacle mask sub-regions corresponding to the instance segmentation region are obtained.

[0014] In a possible implementation, for each instance segmentation region, the instance segmentation region corresponding obstacle mask region and a plurality of obstacle mask sub-regions are determined in the depth image, including:

[0015] For each instance segmentation region, the instance segmentation region is mapped into the depth image to obtain the instance segmentation region corresponding obstacle mask region.

[0016] The instance segmentation region is divided into a plurality of sub-regions, and each sub-region of the instance segmentation region is mapped into the depth image to obtain a plurality of obstacle mask sub-regions corresponding to the instance segmentation region.

[0017] In a possible implementation, the instance segmentation region is divided into a plurality of sub-regions, including:

[0018] The instance segmentation region is divided into a plurality of sub-regions with different sizes, taking the center of the instance segmentation region as the center of each sub-region.

[0019] In a possible implementation, for each instance segmentation region, the depth value of the instance segmentation region is determined according to at least one region of the instance segmentation region corresponding obstacle mask region and a plurality of obstacle mask sub-regions, including:

[0020] For each instance segmentation region, a current to-be-analyzed region is selected in the instance segmentation region corresponding obstacle mask region and a plurality of obstacle mask sub-regions in order of region area from small to large.

[0021] The current to-be-analyzed region is traversed point by point for the depth value, and in a case where the minimum depth value of the current to-be-analyzed region is less than a preset depth threshold, the minimum depth value of the current to-be-analyzed region is determined as the depth value of the instance segmentation region.

[0022] In a case where the minimum depth value of the current to-be-analyzed region is not less than the preset depth threshold, a next to-be-analyzed region is selected to continue analysis until the depth value of the instance segmentation region is determined, or the obstacle mask region corresponding to the instance segmentation region and the plurality of obstacle mask sub-regions are all analyzed.

[0023] In a possible implementation, the mobile robot is controlled to avoid obstacles based on the depth values of the instance segmentation regions.

[0024] In a second aspect, an embodiment of the present application provides a depth determination apparatus based on an image, and the apparatus comprises:

[0025] An image acquisition module is configured to acquire a visible light image and a depth image corresponding to the visible light image.

[0026] An instance detection module is configured to perform instance detection on the visible light image to obtain at least one instance segmentation region.

[0027] An obstacle mask region determination module is configured to, for each instance segmentation region, determine, in the depth image, an obstacle mask region corresponding to the instance segmentation region and a plurality of obstacle mask sub-regions.

[0028] A depth value determination module is configured to, for each instance segmentation region, determine a depth value of the instance segmentation region according to at least one region of the obstacle mask region corresponding to the instance segmentation region and the plurality of obstacle mask sub-regions.

[0029] In a possible implementation, the obstacle mask region determination module comprises:

[0030] An instance segmentation sub-module is configured to, for each instance segmentation region, map the instance segmentation region to the depth image to obtain an obstacle mask region corresponding to the instance segmentation region.

[0031] A sub-region segmentation sub-module is configured to divide the obstacle mask region corresponding to the instance segmentation region into a plurality of sub-regions to obtain a plurality of obstacle mask sub-regions corresponding to the instance segmentation region.

[0032] In a possible implementation, the sub-region segmentation sub-module is specifically configured to divide the obstacle mask region corresponding to the instance segmentation region into a plurality of sub-regions of different sizes with the center of the obstacle mask region corresponding to the instance segmentation region as the center of each sub-region to obtain the plurality of obstacle mask sub-regions corresponding to the instance segmentation region.

[0033] In a possible implementation, the obstacle mask region determination module comprises:

[0034] The mask region segmentation sub-module is configured to segment a region of each instance, map the region of the instance to the depth image, and obtain an obstacle mask region corresponding to the region of the instance.

[0035] The sub-region mapping sub-module is configured to divide the region of the instance into a plurality of sub-regions, and map each sub-region of the region of the instance to the depth image to obtain a plurality of obstacle mask sub-regions corresponding to the region of the instance.

[0036] In a possible implementation, the sub-region mapping sub-module is specifically configured to divide the region of the instance into a plurality of sub-regions with different sizes, with the center of the region of the instance as the center of each sub-region.

[0037] In a possible implementation, the depth value determination module is specifically configured to, for each region of the instance, select a current region to be analyzed in the obstacle mask region and the plurality of obstacle mask sub-regions corresponding to the region of the instance in an order from small to large according to the area of the region; and perform a depth value traversal on the current region to be analyzed point by point, and in a case where the minimum depth value of the current region to be analyzed is less than a preset depth threshold, determine the minimum depth value of the current region to be analyzed as the depth value of the region of the instance; in a case where the minimum depth value of the current region to be analyzed is not less than the preset depth threshold, select a next region to be analyzed for continuous analysis until the depth value of the region of the instance is determined, or the obstacle mask region and the plurality of obstacle mask sub-regions corresponding to the region of the instance are all analyzed.

[0038] In a possible implementation, the device further includes an obstacle avoidance module configured to control the mobile robot to perform obstacle avoidance based on the depth value of each region of the instance.

[0039] In a third aspect, an embodiment of the present application provides a binocular camera, which includes:

[0040] A first visible light image acquisition module is configured to acquire a first visible light image.

[0041] A second visible light image acquisition module is configured to acquire a second visible light image.

[0042] A processing module is configured to implement the image-based depth determination method in any of the embodiments of the present application at runtime.

[0043] In a fourth aspect, an embodiment of the present application provides a mobile robot, which includes:

[0044] An image acquisition device and a processor.

[0045] The image acquisition device is configured to acquire a visible light image.

[0046] The processor is configured to implement the image-based depth determination method described in any of the embodiments of the present application.

[0047] In a fifth aspect, the embodiments of the present application provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the image-based depth determination method described in any of the embodiments of the present application.

[0048] The image-based depth determination method, device, binocular camera and mobile robot provided by the embodiments of the present application can determine the depth value of each instance segmentation region in an image, i.e., the distance between the object and the camera, thereby achieving image-based ranging.

[0049] Of course, implementing any product or method of the present application does not necessarily require all the advantages described above to be achieved at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other embodiments can also be obtained by those skilled in the art based on these drawings.

[0051] Figure 1 A flowchart of the image-based depth determination method of the embodiments of the present application;

[0052] Figure 2 A flowchart of a possible implementation of step S101 in the embodiments of the present application;

[0053] Figure 3 A flowchart of the method for determining a plurality of obstacle mask sub-regions in a depth image in the embodiments of the present application;

[0054] Figure 4 Another flowchart of the method for determining a plurality of obstacle mask sub-regions in a depth image in the embodiments of the present application;

[0055] Figure 5 FIG. 1 is a flowchart illustrating one possible implementation of step S104 in the embodiments of the present application;

[0056] Figure 6 FIG. 2 is another flowchart illustrating the image-based depth determination method in the embodiments of the present application;

[0057] Figure 7 FIG. 3 is a schematic diagram of the image-based depth determination apparatus in the embodiments of the present application;

[0058] Figure 8 FIG. 4 is a schematic diagram of the binocular camera in the embodiments of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art based on the present application are within the scope of protection of the present application.

[0060] First, the terms involved in the present application are described.

[0061] Perception: perception is a series of processes of awareness, feeling, attention, and perception of internal and external information.

[0062] Depth camera: depth camera is also called 3D (three-dimensional) camera, which detects the depth distance of the shooting space through the camera.

[0063] Depth completion: using the sparse depth map captured by the depth sensor to predict the dense pixel-level depth.

[0064] In the related art, according to the visible light image collected by the binocular camera and the pre-marked internal parameters of the binocular camera, the depth image of the scene can be calculated, so that the AGV can use the depth image for obstacle avoidance. However, using the above method, for small, reflective, and pure color objects in the visible light image, the depth information is often incomplete and the edge depth information is unreliable, resulting in unreliable depth information and ultimately leading to failure of the mobile robot to avoid obstacles.

[0065] In order to solve at least one of the above problems, the embodiments of the present application provide an image-based depth determination method, as shown in Figure 1 The method comprises the following steps.

[0066] S101, obtaining a visible light image and a depth image corresponding to the visible light image.

[0067] An optical image of a scene to be detected is acquired, and a depth image of the scene to be detected is acquired. In one example, the depth image can be acquired by a depth camera.

[0068] S102, instance detection is performed on the optical image to obtain at least one instance segmentation region.

[0069] The instance detection is performed on the optical image by using computer vision technology to obtain each instance segmentation region in the optical image. The instance refers to an obstacle in the optical image, and the specific type of the instance can be determined according to the actual type of the scene to be detected. For example, for a residential indoor scene, the instance can be furniture, an electrical appliance, or the like; for example, for a warehouse scene, the instance can be goods, a picking table, or the like.

[0070] S103, for each instance segmentation region, an obstacle mask region corresponding to the instance segmentation region in the depth image is determined, and a plurality of obstacle mask sub-regions are determined.

[0071] According to a predetermined mapping relationship (for example, a coordinate conversion relationship) between the optical image and the depth image, each instance segmentation region in the optical image is determined to have a respective corresponding obstacle mask region in the depth image, and a plurality of obstacle mask sub-regions of each obstacle mask region are determined. Each obstacle mask region includes a plurality of obstacle mask sub-regions.

[0072] S104, for each instance segmentation region, a depth value of the instance segmentation region is determined according to at least one region of the corresponding obstacle mask region and the plurality of obstacle mask sub-regions of the instance segmentation region.

[0073] In one example, for each instance segmentation region, the smallest depth value can be selected as the depth value of the instance segmentation region from at least one region of the corresponding obstacle mask region and the plurality of obstacle mask sub-regions of the instance segmentation region. In one example, for each instance segmentation region, the smallest depth value can be selected as the depth value of the instance segmentation region from the corresponding obstacle mask region and the plurality of obstacle mask sub-regions of the instance segmentation region. This operation is performed for each instance segmentation region, so that the depth value of each instance segmentation region can be obtained.

[0074] In the embodiments of the present application, the minimum depth value in at least one of the obstacle mask region corresponding to the instance segmentation region and the plurality of obstacle mask sub-regions is taken as the depth value of the instance segmentation region, the depth completion of the instance segmentation region is completed, the integrity of the depth value of the instance segmentation region is improved, the problem of incomplete depth information and unreliable edges of small, reflective, and pure color objects can be effectively improved, the depth perception ability for small, reflective, and pure color objects is improved, the distance between the camera (mobile robot) and the obstacle is measured, and the obstacle avoidance function of the mobile robot is enhanced.

[0075] The depth image can be obtained by a depth camera, and the depth camera can directly output a visible light image and a depth image. The depth camera can include a left eye lens, a right eye lens, and a visible light lens; the left eye lens and the right eye lens are respectively used to collect a left eye image and a right eye image of an actual scene, and an internal parameter is labeled in advance, a depth image is calculated according to the left eye image and the right eye image, and a visible light image is directly collected by using the visible light lens; the left eye lens and the right eye lens can use visible light as a light source, or can use near-infrared light as a light source, both of which are within the protection scope of the present application. The specific process of calculating the depth image by the depth camera can refer to the working process of the depth camera in the related art, and is not specifically limited in the present application.

[0076] The depth image can also be obtained by a binocular camera. In a possible implementation manner, as shown in FIG. 1, a visible light image and a depth image corresponding to the visible light image are obtained, including: Figure 2

[0077] S1011, a first visible light image and a second visible light image collected by a binocular camera at the same time are obtained, and an internal parameter of the binocular camera is obtained.

[0078] S1012, a depth image is calculated based on the internal parameter, the first visible light image, and the second visible light image, wherein the visible light image is the first visible light image, the second visible light image, or an image synthesized by the first visible light image and the second visible light image.

[0079] ​Specifically, the two lenses of the binocular camera simultaneously capture a first visible light image and a second visible light image at the same time, and the internal parameters of the binocular camera are obtained by pre-calibration. According to the binocular ranging principle, the internal parameters of the binocular camera, the first visible light image and the second visible light image are used to calculate a depth image. The specific calculation process can refer to the binocular camera ranging method in the related art, and will not be described herein. The first visible light image can be directly used as the visible light image, or the second visible light image can be directly used as the visible light image. In one example, the first visible light image and the second visible light image can be synthesized, and the synthesized image can be used as the visible light image. The mapping relationship between the visible light image and the depth image can be obtained according to the internal parameters of the binocular camera.

[0080] In one example, the instance segmentation region can be first mapped to the depth image to obtain an obstacle mask region, and the obstacle mask region is segmented to obtain an obstacle mask sub-region. In one possible implementation, for each instance segmentation region, the obstacle mask region corresponding to the instance segmentation region in the depth image is determined, and a plurality of obstacle mask sub-regions are obtained, including:

[0081] Step one: For each instance segmentation region, the instance segmentation region is mapped to the depth image to obtain the obstacle mask region corresponding to the instance segmentation region.

[0082] Step two: The obstacle mask region corresponding to the instance segmentation region is divided into a plurality of sub-regions to obtain a plurality of obstacle mask sub-regions corresponding to the instance segmentation region.

[0083] The division method of the obstacle mask region can be set according to actual conditions. In one example, the obstacle mask region can be divided into a plurality of obstacle mask sub-regions by using a grid with a preset size, or the obstacle mask region can be divided into a plurality of obstacle mask sub-regions by using a division method with the same center and different areas. In one possible implementation, the obstacle mask region corresponding to the instance segmentation region is divided into a plurality of sub-regions to obtain a plurality of obstacle mask sub-regions corresponding to the instance segmentation region, including: taking the center of the obstacle mask region corresponding to the instance segmentation region as the center of each sub-region, dividing the obstacle mask region corresponding to the instance segmentation region into a plurality of sub-regions with different sizes, and obtaining a plurality of obstacle mask sub-regions corresponding to the instance segmentation region.

[0084] In one example, taking the division of each obstacle mask region into three obstacle mask sub-regions as an example, the example is illustrated. As shown in FIG. 6, the obstacle mask region corresponding to the instance segmentation region is divided into three obstacle mask sub-regions, and the obstacle mask sub-region corresponding to the instance segmentation region is obtained. Figure 3As shown, each instance segmentation region of the perceived object is mapped to the depth image to obtain an obstacle mask region corresponding to the instance segmentation region, and the obstacle mask region of the perceived object is segmented through the following steps:

[0085] Step A: A first obstacle mask sub-region with an area of 1 / 8 of the obstacle mask region is divided with the obstacle mask region as the center.

[0086] Step B: A first obstacle mask sub-region with an area of 1 / 4 of the obstacle mask region is divided with the obstacle mask region as the center.

[0087] Step C: A first obstacle mask sub-region with an area of 1 / 2 of the obstacle mask region is divided with the obstacle mask region as the center.

[0088] In the embodiments of the present application, the instance segmentation region is first mapped to the depth image to obtain the obstacle mask region, and the obstacle mask region is segmented to obtain the obstacle mask sub-region, so that the number of mappings can be reduced, thereby saving the calculation resources occupied by the mapping.

[0089] In one example, the instance segmentation region can be first divided into a plurality of sub-regions, and then each sub-region and the instance segmentation region are mapped to the depth image to obtain the obstacle mask region and the obstacle mask sub-region. In a possible implementation, for each instance segmentation region, the obstacle mask region corresponding to the instance segmentation region and a plurality of obstacle mask sub-regions are determined in the depth image, including:

[0090] Step one, for each instance segmentation region, the instance segmentation region is mapped to the depth image to obtain the obstacle mask region corresponding to the instance segmentation region;

[0091] Step two, the instance segmentation region is divided into a plurality of sub-regions, and each sub-region of the instance segmentation region is mapped to the depth image to obtain a plurality of obstacle mask sub-regions corresponding to the instance segmentation region.

[0092] The division mode of the instance segmentation region can be set according to actual conditions, in one example, the instance segmentation region can be divided into a plurality of sub-regions by using a grid with a preset size, or the instance segmentation region can be divided into a plurality of sub-regions by using a division mode with the same center and different areas. In a possible implementation, the instance segmentation region is divided into a plurality of sub-regions, including: taking the center of the instance segmentation region as the center of each sub-region, and dividing the instance segmentation region into a plurality of sub-regions with different sizes.

[0093] In one example, taking the division of each instance segmentation region into three sub-regions as an example, an example is given. As shown in FIG. 4, the instance segmentation region is divided into three sub-regions 401, 402 and 403.Figure 4 As shown, the instance segmentation region of the perceived object is segmented by the following steps, and the segmented instance segmentation sub-region is respectively mapped into the depth image to obtain the corresponding obstacle mask sub-region:

[0094] Step A: Divide the first instance segmentation sub-region with an area of 1 / 8 of the instance segmentation region.

[0095] Step B: Divide the first instance segmentation sub-region with an area of 1 / 4 of the instance segmentation region.

[0096] Step C: Divide the first instance segmentation sub-region with an area of 1 / 2 of the instance segmentation region.

[0097] In a possible implementation, referring to Figure 5 , the method further includes:

[0098] S1041 For each instance segmentation region, in the obstacle mask region and the plurality of obstacle mask sub-regions corresponding to the instance segmentation region, a current to-be-analyzed region is selected in order of area size from small to large.

[0099] For each instance segmentation region, in the obstacle mask region and the plurality of obstacle mask sub-regions corresponding to the instance segmentation region, the region with the smallest area is first selected as the current to-be-analyzed region; after the analysis of the current to-be-analyzed region is completed, the region with the second smallest area is selected as the current to-be-analyzed region for analysis, and the like, until the analysis of the obstacle mask region corresponding to the instance segmentation region is completed.

[0100] S1042 The current to-be-analyzed region is traversed point by point to determine the minimum depth value of the current to-be-analyzed region as the depth value of the instance segmentation region in a case where the minimum depth value of the current to-be-analyzed region is less than a preset depth threshold.

[0101] The preset depth threshold can be set according to actual conditions. When the depth value of the object is greater than or equal to the preset depth threshold, the object can be considered as a distant object, which does not affect obstacle avoidance. When the depth value of the object is less than the preset depth threshold, the object is considered as a close object, which affects obstacle avoidance. Therefore, when the minimum depth value is selected, the depth value greater than or equal to the preset depth threshold does not need to be considered.

[0102] S1043 In a case where the minimum depth value of the current to-be-analyzed region is not less than the preset depth threshold, a next to-be-analyzed region is selected for continuous analysis until the depth value of the instance segmentation region is determined, or the obstacle mask region corresponding to the instance segmentation region and the plurality of obstacle mask sub-regions are all analyzed.

[0103] The initial value of the depth value of the instance segmentation region is not less than the preset depth threshold. For example, the initial value of the depth value of the instance segmentation region can be set to infinity, or the initial value of the depth value of the instance segmentation region can be set to the preset depth threshold. The current depth value of the instance segmentation region is compared with the size of the currently selected minimum depth value. If the currently selected minimum depth value is less than the depth value of the instance segmentation region, the depth value of the instance segmentation region is updated to the currently selected minimum depth value. If the currently selected minimum depth value is not less than the depth value of the instance segmentation region, a next to-be-analyzed region is selected for continuous analysis until the depth value of the instance segmentation region is determined, or the obstacle mask region corresponding to the instance segmentation region and the plurality of obstacle mask sub-regions are all analyzed.

[0104] In the embodiments of the present application, for each instance segmentation region, the depth value of the instance segmentation region is determined according to at least one region of the obstacle mask region corresponding to the instance segmentation region and the plurality of obstacle mask sub-regions. The depth value obtained is used to complete the depth completion of the instance segmentation region, which improves the integrity of the depth value of the instance segmentation region, effectively improves the problem of incomplete depth information and unreliable edges of small, reflective, and pure color objects, improves the depth perception ability for small, reflective, and pure color objects, measures the distance between the mobile robot and the obstacle, and enhances the obstacle avoidance function of the mobile robot.

[0105] The embodiments of the present application also provide a depth determination method based on an image, as shown in Figure 6 The method comprises the following steps.

[0106] Step one, input the obstacle mask sub-region and the depth image.

[0107] The first obstacle mask sub-region, the second obstacle mask sub-region, and the third obstacle mask sub-region can be sequentially selected as the current to-be-analyzed region.

[0108] Step two: take the first obstacle mask sub-region as the current analysis region, and traverse the depth value of each pixel point in the current analysis region one by one. If there is a depth value less than the preset depth threshold in the current analysis region, select the minimum depth value in the range to fill the obstacle mask region of the perceived object, so as to realize the depth completion of the perceived object. If there is no depth value less than the preset depth threshold in the current analysis region, skip the current analysis region, and take the second obstacle mask sub-region or the second instance segmentation sub-region corresponding to the obstacle mask region as the current analysis region.

[0109] Step three: take the second obstacle mask sub-region as the current analysis region, and traverse the depth value of each pixel point in the current analysis region one by one. If there is a depth value less than the preset depth threshold in the current analysis region, select the minimum depth value in the range to fill the obstacle mask region of the perceived object, so as to realize the depth completion of the perceived object. If there is no depth value less than the preset depth threshold in the current analysis region, skip the current analysis region, and take the third obstacle mask sub-region or the third instance segmentation sub-region corresponding to the obstacle mask region as the current analysis region.

[0110] Step four: take the third obstacle mask sub-region as the current analysis region, and traverse the depth value of each pixel point in the current analysis region one by one. If there is a depth value less than the preset depth threshold in the current analysis region, select the minimum depth value in the range to fill the obstacle mask region of the perceived object, so as to realize the depth completion of the perceived object. If there is no depth value less than the preset depth threshold in the current analysis region, skip the current analysis region, and take the entire obstacle mask region as the current analysis region.

[0111] Step five: take the entire obstacle mask region as the current analysis region, and traverse the depth value of each pixel point in the current analysis region one by one. If there is a depth value less than the preset depth threshold in the current analysis region, select the minimum depth value in the range to fill the obstacle mask region of the perceived object, so as to realize the depth completion of the perceived object. If there is no depth value less than the preset depth threshold in the current analysis region, end.

[0112] In a possible implementation, the method further includes: based on the depth value of each instance segmentation region, controlling the mobile robot to avoid obstacles.

[0113] For example, if there is a depth value less than the preset depth threshold in the obstacle mask area of the perceived object, the depth of the obstacle mask area is completed, thereby improving the perception ability of the perceived object, and the mobile robot can better control the perception of the object and obstacle avoidance; if there is no depth value less than the preset depth threshold in the obstacle mask area of the perceived object, it is indicated that the perceived object belongs to a distant obstacle, and the mobile robot does not need to be controlled to avoid obstacles.

[0114] As shown in Figure 7 The embodiment of the application also provides a depth determination device based on an image, which comprises:

[0115] An image acquisition module 301 is configured to acquire a visible light image and a depth image corresponding to the visible light image.

[0116] An instance detection module 302 is configured to perform instance detection on the visible light image to obtain at least one instance segmentation region.

[0117] An obstacle mask area determination module 303 is configured to determine, for each instance segmentation region, an obstacle mask area corresponding to the instance segmentation region and a plurality of obstacle mask sub-regions in the depth image.

[0118] A depth value determination module 304 is configured to determine, for each instance segmentation region, a depth value of the instance segmentation region according to at least one region of the obstacle mask area corresponding to the instance segmentation region and the plurality of obstacle mask sub-regions.

[0119] In a possible implementation, the obstacle mask area determination module comprises:

[0120] An instance segmentation sub-module is configured to map each instance segmentation region to the depth image to obtain an obstacle mask area corresponding to the instance segmentation region.

[0121] A sub-region segmentation sub-module is configured to divide the obstacle mask area corresponding to the instance segmentation region into a plurality of sub-regions to obtain a plurality of obstacle mask sub-regions corresponding to the instance segmentation region.

[0122] In a possible implementation, the sub-region segmentation sub-module is specifically configured to divide the obstacle mask area corresponding to the instance segmentation region into a plurality of sub-regions of different sizes with the center of the obstacle mask area corresponding to the instance segmentation region as the center of each sub-region, to obtain a plurality of obstacle mask sub-regions corresponding to the instance segmentation region.

[0123] In a possible implementation, the obstacle mask area determination module comprises:

[0124] The mask region segmentation sub-module is configured to map each instance segmentation region to the depth image to obtain an obstacle mask region corresponding to the instance segmentation region.

[0125] The sub-region mapping sub-module is configured to divide the instance segmentation region into a plurality of sub-regions, and map each sub-region of the instance segmentation region to the depth image to obtain a plurality of obstacle mask sub-regions corresponding to the instance segmentation region.

[0126] In a possible implementation, the sub-region mapping sub-module is specifically configured to divide the instance segmentation region into a plurality of sub-regions with different sizes, with the center of the instance segmentation region as the center of each sub-region.

[0127] In a possible implementation, the depth value determination module is specifically configured to: for each instance segmentation region, select a current to-be-analyzed region in the obstacle mask region and the plurality of obstacle mask sub-regions corresponding to the instance segmentation region in a sequence from small to large according to the area of the region; and perform depth value traversal on the current to-be-analyzed region point by point, and in a case where the minimum depth value of the current to-be-analyzed region is less than a preset depth threshold, determine the minimum depth value of the current to-be-analyzed region as the depth value of the instance segmentation region; in a case where the minimum depth value of the current to-be-analyzed region is not less than the preset depth threshold, select a next to-be-analyzed region for continuous analysis until the depth value of the instance segmentation region is determined, or the obstacle mask region and the plurality of obstacle mask sub-regions corresponding to the instance segmentation region are all analyzed.

[0128] In a possible implementation, the apparatus further includes an obstacle avoidance module configured to control the mobile robot to perform obstacle avoidance based on the depth value of each instance segmentation region.

[0129] In another embodiment of the present application, as shown in Figure 8 A binocular camera is also provided, and the binocular camera includes:

[0130] The first visible light image acquisition module 401 is configured to acquire a first visible light image.

[0131] The second visible light image acquisition module 402 is configured to acquire a second visible light image.

[0132] The processing module 403 is configured to implement the image-based depth determination method described in any of the embodiments of the present application in runtime.

[0133] In another embodiment of the present application, a mobile robot is also provided, and the mobile robot includes:

[0134] An image acquisition device and a processor.

[0135] The image acquisition device is configured to acquire a visible light image.

[0136] The processor is configured to implement any of the image-based depth determination methods described herein.

[0137] In another embodiment provided in the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of any of the image-based depth determination methods described in the present application.

[0138] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (for example, floppy disk, hard disk, magnetic tape), optical media (for example, DVD), or semiconductor media (for example, solid state disk (SSD)) and the like.

[0139] It should be noted that in this paper, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitation, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0140] Each of the embodiments in the specification is described in a relevant manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0141] The above only describes the preferred embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining depth based on images, characterized in that, The method comprises: acquiring a visible light image and a depth image corresponding to the visible light image; performing instance detection on the visible light image to obtain at least one instance segmentation region, each instance segmentation region containing a perceived object; for each instance segmentation region, determining a corresponding obstacle mask region and a plurality of obstacle mask sub-regions in the depth image; the center of each obstacle mask sub-region is the center of the instance segmentation region or the center of the corresponding obstacle mask region of the instance segmentation region; each obstacle mask sub-region has a different size; for each instance segmentation region, selecting a current analysis region in the corresponding obstacle mask region and the plurality of obstacle mask sub-regions of the instance segmentation region in order of area from small to large; performing depth value traversal on the current analysis region point by point, and determining the depth value of the instance segmentation region in a case where the minimum depth value of the current analysis region is less than a preset depth threshold; in a case where the minimum depth value of the current analysis region is not less than the preset depth threshold, selecting a next analysis region for continuous analysis until the depth value of the instance segmentation region is determined or the corresponding obstacle mask region and the plurality of obstacle mask sub-regions of the instance segmentation region are all analyzed; completing depth completion of the perceived object in the instance segmentation region through the depth value of the instance segmentation region.

2. The method of claim 1, wherein, The method comprises: for each instance segmentation region, mapping the instance segmentation region to the depth image to obtain the corresponding obstacle mask region of the instance segmentation region; dividing the corresponding obstacle mask region of the instance segmentation region into a plurality of sub-regions to obtain a plurality of obstacle mask sub-regions corresponding to the instance segmentation region.

3. The method of claim 2, wherein, The method comprises: taking the center of the corresponding obstacle mask region of the instance segmentation region as the center of each sub-region, dividing the corresponding obstacle mask region of the instance segmentation region into a plurality of sub-regions with different sizes, and obtaining a plurality of obstacle mask sub-regions corresponding to the instance segmentation region.

4. The method of claim 1, wherein, The method comprises: for each instance segmentation region, mapping the instance segmentation region to the depth image to obtain the corresponding obstacle mask region of the instance segmentation region; dividing the instance segmentation region into a plurality of sub-regions and mapping each sub-region of the instance segmentation region to the depth image to obtain a plurality of obstacle mask sub-regions corresponding to the instance segmentation region.

5. The method of claim 4, wherein, The method comprises: The center of the instance segmentation region is taken as the center of each sub-region, and the instance segmentation region is divided into a plurality of sub-regions with different sizes.

6. The method of claim 1, wherein, The method further comprises: Based on the depth value of each instance segmentation region, the mobile robot is controlled to avoid obstacles.

7. An image-based depth determination apparatus, characterized by Comprise: An image acquisition module is configured to acquire a visible light image and a depth image corresponding to the visible light image; An instance detection module is configured to perform instance detection on the visible light image to obtain at least one instance segmentation region, each instance segmentation region containing a perceived object; An obstacle mask region determination module is configured to, for each instance segmentation region, determine, in the depth image, an obstacle mask region corresponding to the instance segmentation region and a plurality of obstacle mask sub-regions, the center of each obstacle mask sub-region being the center of the instance segmentation region or the center of the obstacle mask region corresponding to the instance segmentation region, and each obstacle mask sub-region having a different size; A depth value determination module is configured to, for each instance segmentation region, select, in the obstacle mask region corresponding to the instance segmentation region and the plurality of obstacle mask sub-regions, a current analysis region in order of area from small to large, and perform point-by-point traversal of the depth value of the current analysis region, and in a case where the minimum depth value of the current analysis region is less than a preset depth threshold, determine the minimum depth value of the current analysis region as the depth value of the instance segmentation region, and in a case where the minimum depth value of the current analysis region is not less than the preset depth threshold, select a next analysis region for continued analysis until the depth value of the instance segmentation region is determined or the obstacle mask region corresponding to the instance segmentation region and the plurality of obstacle mask sub-regions are all analyzed, and complete depth completion of the perceived object in the instance segmentation region through the depth value of the instance segmentation region.

8. The apparatus of claim 7, wherein, The obstacle mask region determination module comprises: An instance segmentation sub-module is configured to, for each instance segmentation region, map the instance segmentation region to the depth image to obtain an obstacle mask region corresponding to the instance segmentation region; A sub-region segmentation sub-module is configured to divide the obstacle mask region corresponding to the instance segmentation region into a plurality of sub-regions to obtain a plurality of obstacle mask sub-regions corresponding to the instance segmentation region; The sub-region segmentation sub-module is specifically configured to take the center of the obstacle mask region corresponding to the instance segmentation region as the center of each sub-region, divide the obstacle mask region corresponding to the instance segmentation region into a plurality of sub-regions with different sizes, and obtain a plurality of obstacle mask sub-regions corresponding to the instance segmentation region; The obstacle mask region determination module comprises: A mask region segmentation sub-module is configured to, for each instance segmentation region, map the instance segmentation region to the depth image to obtain an obstacle mask region corresponding to the instance segmentation region; A sub-region mapping sub-module is configured to divide the instance segmentation region into a plurality of sub-regions and map each sub-region of the instance segmentation region to the depth image to obtain a plurality of obstacle mask sub-regions corresponding to the instance segmentation region; The sub-region mapping submodule is specifically configured to divide the instance segmentation region into a plurality of sub-regions of different sizes with centers of the instance segmentation region as centers of the sub-regions. The device further comprises an obstacle avoidance module configured to control the mobile robot to avoid obstacles based on the depth values of the instance segmentation regions.

9. A binocular camera, characterized by The device comprises: a first image acquisition module configured to acquire a first image; a second image acquisition module configured to acquire a second image; a processing module configured to implement the image-based depth determination method of any one of claims 1-6 at runtime.

10. A mobile robot, characterized by The device comprises: an image acquisition device and a processor; the image acquisition device is configured to acquire a visible light image; the processor is configured to implement the image-based depth determination method of any one of claims 1-6 at runtime.

11. A computer readable storage medium, characterized in that, The computer program is stored in the computer-readable storage medium and is executed by the processor to implement the image-based depth determination method of any one of claims 1-6.

Images