Monocular depth estimation method and system

Abstract

The invention relates to a monocular depth estimation method and system. The method comprises the following steps: receiving a single-view image from a camera; projecting the coordinate of each pixel of a to-be-projected image to a pixel plane of the single-view-angle image according to the parameters of the camera so as to obtain a corresponding relation between the pixels in the projected image and the pixels of the single-view-angle image; filling corresponding pixels of the projection image with the pixel information of the pixels from the single-view image according to the corresponding relation; and sending the filled projection image to a monocular depth estimation model for depth estimation to obtain a depth map.

Description

Technical Field

[0001] The present disclosure relates to the technical field of depth estimation, and in particular to a monocular depth estimation method and system. Background Art

[0002] In recent years, depth estimation tasks have become increasingly popular in the field of computer vision. Effective depth estimation is of great significance for various downstream tasks (such as safe and reliable autonomous driving functions). Currently in the field of autonomous driving, a vehicle model is usually equipped with multiple different types of cameras (for example, pinhole cameras for periscopic vision, fisheye cameras for surround vision, etc.), and the parameters used by cameras of the same type are also different. This situation has a great negative impact on monocular depth estimation.

[0003] The present disclosure has been improved with respect to but not limited to the above-mentioned factors. Summary of the invention

[0004] To this end, the present disclosure proposes a monocular depth estimation method and system. The method and system of the present disclosure can realize depth estimation for images output by any type of camera and are therefore universal, thereby solving the problems caused by different camera models and different camera parameters to the depth estimation system. In the monocular depth estimation method and system of the present disclosure, a universal projection module is proposed, which can convert the image output by any camera into a projection image of a universal representation (such as a Cassini projection image), so that a single depth estimation model can be used to estimate the depth across cameras. As a result, the monocular depth estimation method and system of the present disclosure reduce the burden of the depth estimation model on different camera types and different camera parameters, and can achieve depth estimation for multiple cameras through less data training.

[0005] According to a first aspect of the present disclosure, a monocular depth estimation method is provided, wherein the method comprises: receiving a monocular image from a camera; projecting the coordinates of each pixel of a desired projection image onto a pixel plane of the monocular image according to parameters of the camera, thereby obtaining a correspondence between pixels in the projection image and pixels of the monocular image; filling pixel information of pixels from the monocular image into corresponding pixels of the projection image according to the correspondence; and sending the filled projection image to a monocular depth estimation model for depth estimation to obtain a depth map.

[0006] According to one embodiment, the desired projection image comprises a transverse cylindrical projection image.

[0007] According to another embodiment, the desired projection image comprises a Cassini projection image.

[0008] According to yet another embodiment, the coordinates of each pixel of the desired projection image are coordinates in a Cartesian coordinate system generated according to the size and the field of view angle range of the desired projection image.

[0009] According to another embodiment, the size of the desired projection image includes an image width W and an image height H, and the field of view angle range of the desired projection image includes a field of view angle range [α0, α W-1 ] and the field of view range at the image height [β0,β H-1 ].

[0010] According to yet another embodiment, the size and the viewing angle range of the desired projection image are selected to be smaller than or equal to the size of the single-view image.

[0011] According to yet another embodiment, the coordinates of each pixel in the desired projection image in a Cartesian coordinate system are generated using the following formula:

[0012]

[0013] Where (x ij ,y ij ,z ij ) is the coordinate of the ijth pixel in the desired projection image in the Cartesian coordinate system; i = 0, 1, 2, ..., W-1; j = 0, 1, 2, ..., H-1; α i =α0+i*(α W-1 -α0) / W; β j =β0+j*(β H-1 -β0) / H, ρ is any suitable value.

[0014] According to yet another embodiment, the value of p is taken as 1.

[0015] According to yet another embodiment, the monocular depth estimation model is trained using projected images.

[0016] According to yet another embodiment, the method further comprises sending the depth map to a downstream task for use.

[0017] According to a second aspect of the present disclosure, a monocular depth estimation system is provided, comprising: a receiving module, the receiving module being configured to receive a monocular image from a camera; a projection module, the projection module being configured to: project the coordinates of each pixel of a desired projection image onto a pixel plane of the monocular image according to parameters of the camera, thereby obtaining a correspondence between pixels in the projection image and pixels of the monocular image; and fill pixel information of pixels from the monocular image into corresponding pixels of the projection image according to the correspondence; and a monocular depth estimation module, the monocular depth estimation module being configured to perform depth estimation using the filled projection image to obtain a depth map.

[0018] According to a third aspect of the present disclosure, a monocular depth estimation system is provided, comprising: a camera arranged on a vehicle; and an on-board computer, wherein the camera is configured to capture images and transmit them to the on-board computer, and wherein the on-board computer is configured to execute the method according to the first aspect of the present disclosure.

[0019] According to a fourth aspect of the present disclosure, a vehicle comprising the system according to the third aspect of the present disclosure is provided.

[0020] Aspects generally include methods, apparatus, systems, computer program products, and processing systems substantially as described herein with reference to and as illustrated by the accompanying figures.

[0021] The foregoing has broadly outlined the features and technical advantages of examples according to the present disclosure so that the following detailed description can be better understood. Additional features and advantages will be described hereinafter. The disclosed concepts and specific examples can be easily used as the basis for modifying or designing other structures for implementing the same purpose as the present disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The characteristics of the concepts disclosed herein, both in terms of their organization and method of operation, and the associated advantages will be better understood by considering the following description in conjunction with the accompanying drawings. Each of the accompanying drawings is provided for illustration and description purposes and does not define limitations on the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to understand in detail the manner in which the above-stated features of the present disclosure are used, the above briefly summarized content may be described in more detail with reference to various aspects, some of which are illustrated in the accompanying drawings. However, it should be noted that the accompanying drawings illustrate only certain typical aspects of the present disclosure and should not be considered to limit its scope, as the description may allow for other equally effective aspects. The same reference numerals in different drawings may identify the same or similar elements.

[0023] Figure 1A flow chart of a monocular depth estimation method according to an example embodiment of the present disclosure is shown;

[0024] Figure 2 A schematic block diagram of a monocular depth estimation system according to an example embodiment of the present disclosure is shown;

[0025] Figure 3 A schematic diagram showing another monocular depth estimation system according to an example embodiment of the present disclosure; and

[0026] Figure 4 A schematic diagram of a vehicle according to an example embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] The inventors recognize that monocular depth estimation is a dense depth perception task, and accurate dense depth estimation is crucial for understanding scene structure. With the development of deep learning technology and the computing power of deployment platforms, monocular depth estimation has been applied in many fields, such as 3D scene reconstruction, robots and autonomous driving. In the field of autonomous driving, a vehicle model is usually equipped with multiple different types of cameras (for example, pinhole cameras for periscopic vision, fisheye cameras for surround vision, etc.), and the parameters used by cameras of the same type are also different. This situation has a great negative impact on monocular depth estimation. For example, it is necessary to perform depth estimation on the periscopic image and the surround image separately, and the camera models of the two are different, and one algorithm model cannot meet the depth estimation of the two camera models. Or use two depth estimation models to support two types of cameras respectively, but this undoubtedly adds a great burden. In addition, the parameter differences of the same type of cameras will also have a negative impact on depth estimation.

[0028] To this end, the present disclosure proposes a monocular depth estimation method and system, which can realize depth estimation for images output by any type of camera and is therefore universal, thereby solving the problems caused by different camera models and different camera parameters to the depth estimation system. In the monocular depth estimation method and system of the present disclosure, a universal projection module is proposed, which can convert the image output by any camera into a projection image of a universal representation (such as a Cassini projection image), so that a single depth estimation model can be used to estimate the depth across cameras. As a result, the monocular depth estimation method and system of the present disclosure reduces the burden of the depth estimation model on different camera types and different camera parameters, and can achieve depth estimation for multiple cameras through less data training.

[0029] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. This detailed description includes specific details to provide a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details.

[0030] refer to Figure 1 , which shows a flowchart of an example monocular depth estimation method 100 according to an embodiment of the present disclosure.

[0031] As shown, the method 100 may include receiving a single-view image from a camera at block 110. In one embodiment, the camera may be any suitable imaging device, such as a pinhole camera, a fisheye camera, etc. According to the embodiment, the camera may be a vehicle-mounted camera mounted on a vehicle (such as a vehicle), and may include a variety of different types of cameras.

[0032] Next, at block 120, method 100 may include projecting the coordinates of each pixel of the desired projection image onto the pixel plane of the monoscopic image according to the parameters of the camera, thereby obtaining a correspondence between the pixels in the projection image and the pixels in the monoscopic image. It will be appreciated that this step may be regarded as "shooting" the 3D space represented by the desired projection image through the camera, thereby corresponding the shot "virtual image" to the shot real monoscopic image.

[0033] In an example of the present disclosure, the parameters of the camera may be known and stored in advance, so that the method 100 can read these parameters from the corresponding memory. In this example, the parameters of the camera may be associated with an identifier of the corresponding camera, so that the method 100 may also receive the identifier of the camera when receiving the monoscopic image from the camera, and then use the received camera identifier to find the parameters of the camera. For example, the camera parameters and the camera identifier may be stored in a lookup table.

[0034] In another embodiment of the present disclosure, the parameters of the camera may be transmitted together with the monoscopic image, so that the method 100 may receive the camera parameters and the monoscopic image together.

[0035] At block 130 , the method 100 may include filling pixel information of pixels from the single-view image into corresponding pixels of the projection image according to the above correspondence relationship, thereby obtaining a “projection image” of the single-view image.

[0036] Continue to refer Figure 1 At block 140 , method 100 may include sending the padded projection image to a monocular depth estimation model for depth estimation to obtain a depth map.

[0037] In one embodiment of the present disclosure, method 100 may further include sending the obtained depth map to downstream tasks for use, such as 3D scene reconstruction, autonomous driving, etc.

[0038] In another embodiment of the present disclosure, the projected image may be an image obtained using any suitable projection method, such as a transverse cylindrical projection image. Further according to this embodiment, the projected image may include a Cassini projection image. The embodiments of the present disclosure are specifically explained below using Cassini projection as an example. It will be understood that Cassini projection is only explained as an example and is not a limitation to the embodiments of the present disclosure.

[0039] In a preferred embodiment of the present disclosure, the coordinates of each pixel of the desired projection image may be coordinates in a Cartesian coordinate system generated according to the size and field of view range of the desired projection image. According to this embodiment, in the case of using Cassini projection, the size of the desired Cassini projection image may include an image width W and an image height H, and the field of view range of the Cassini projection image may include a field of view range [α0,α W-1 ] and the field of view range at the image height [β0,β H-1 ]. It will be clear that α0 and α W-1 are the field of view angles at the two end points of the image width, and β0 and β H-1 are the field of view angles at the two end points of the image height.

[0040] Therefore, the coordinates of each pixel in the projected image in the Cartesian coordinate system can be generated using the following formula (1):

[0041]

[0042] Where (x ij ,y ij ,z ij ) is the coordinate of the ijth pixel in the projected image in the Cartesian coordinate system;

[0043] i=0,1,2,……,W-1;

[0044] j=0,1,2,……,H-1;

[0045] α i =α0+i*(α W-1 -α0) / W;

[0046] β j =β0+j*(β H-1 -β0) / H; and

[0047] ρ is any suitable value.

[0048] In a preferred embodiment of the present disclosure, the value of ρ is taken as 1 for ease of calculation.

[0049] In another embodiment of the present disclosure, the size and field of view range of the desired projection image may be selected to be less than or equal to the size of the monoscopic image from the camera. For example, the image width and image height of the desired projection image may be selected to be less than or equal to the image width and image height of the monoscopic image from the camera, respectively, and the field of view range of the desired projection image on the image width and the field of view range on the image height may be selected to be less than or equal to the field of view range on the image width and the field of view range on the image height of the monoscopic image from the camera, respectively. In the presence of multiple types of cameras, the size and field of view range of the desired projection image may be selected to be less than or equal to the size of the smallest monoscopic image in the monoscopic images from these cameras. It will also be understood that, depending on the accuracy requirements of the monocular depth estimation, the desired projection image is generally not selected to be too small. As an example, the desired projection image may be selected to be 400x600 in size, which is suitable for most cameras and corresponding depth estimation requirements.

[0050] It will also be understood that the monocular depth estimation model used in method 100 is trained using projected images rather than directly using images from the camera. Thus, the monocular depth estimation model of the present disclosure is universally applicable to various types of cameras.

[0051] In yet another embodiment of the present disclosure, the obtained depth map can be converted into a point cloud according to the above formula (1) to better adapt to downstream tasks, such as 3D free space generation, 3D obstacle detection, and the like.

[0052] refer to Figure 2 , which shows a schematic block diagram of a monocular depth estimation system 200 according to an example embodiment of the present disclosure.

[0053] like Figure 2 As shown, the monocular depth estimation system 200 may include a receiving module 201, a projection module 203, and a monocular depth estimation module 205. It will be understood that although Figure 2 Only the above three modules are shown, but the monocular depth estimation system 200 may include more or fewer modules as long as they can achieve the corresponding functions.

[0054] In one embodiment of the present disclosure, the receiving module 201 may be configured to receive a single-view image from a camera. In one embodiment, the camera may be any suitable imaging device, such as a pinhole camera, a fisheye camera, etc. According to this embodiment, the camera may be a vehicle-mounted camera mounted on a vehicle (such as a vehicle), and may include a variety of different types of cameras.

[0055] In one embodiment of the present disclosure, the projection module 203 can be configured to project the coordinates of each pixel of the desired projection image onto the pixel plane of the monoscopic image according to the parameters of the camera, thereby obtaining a correspondence between the pixels in the projection image and the pixels of the monoscopic image; and fill the pixel information of the pixels from the monoscopic image into the corresponding pixels of the projection image according to the correspondence.

[0056] In one example of the present disclosure, the parameters of the camera may be known and stored in advance, so that the monocular depth estimation system 200 can read these parameters from the corresponding memory. In this example, the parameters of the camera may be associated with the identifier of the corresponding camera, so that the monocular depth estimation system 200 can also receive the identifier of the camera when receiving the monoscopic image from the camera through the receiving module 201, and then use the received camera identifier to find the parameters of the camera. For example, the camera parameters and the camera identifier can be stored in a lookup table. In another embodiment of the present disclosure, the parameters of the camera can be transmitted together with the monoscopic image, so that the receiving module 201 can receive the camera parameters and the monoscopic image together.

[0057] In an embodiment of the present disclosure, the monocular depth estimation module 205 may be configured to use the padded projection image to perform depth estimation to obtain a depth map.

[0058] In another embodiment of the present disclosure, the monocular depth estimation system 200 may further include an output module ( Figure 2 (not shown) to output the obtained depth map to downstream tasks for use, such as 3D scene reconstruction, autonomous driving, etc.

[0059] In another embodiment of the present disclosure, the monocular depth estimation module 205 may be trained using projected images rather than directly using images from a camera. Thus, the monocular depth estimation system 200 may be applicable to various types of cameras.

[0060] refer to Figure 3 , which shows a schematic diagram of another monocular depth estimation system 300 according to an exemplary embodiment of the present disclosure. Figure 3 As shown, the monocular depth estimation system 300 may include a camera 301 and an onboard computer 303 disposed on a vehicle. In one embodiment of the present disclosure, the camera 301 may be configured to capture images and transmit them to the onboard computer 303 for depth estimation. Furthermore, the onboard computer 303 may be configured to perform the above-mentioned combination of Figure 1 Method 100 is described.

[0061] In one example of the present disclosure, the camera 301 may be a monocular camera. Figure 3Only one camera 301 is shown in FIG. 3 , but the monocular depth estimation system 300 may include any number of cameras (especially monocular cameras), such as Figure 3 Indicated by the ellipsis in .

[0062] refer to Figure 4 , which shows a schematic diagram of a vehicle 400 according to an exemplary embodiment of the present disclosure. In this embodiment, the vehicle 400 may include a reference Figure 3 An example monocular depth estimation system 300 is shown and described.

[0063] As described above, the present disclosure provides a monocular depth estimation system and method. The method and system of the present disclosure are highly versatile and can replace the monocular depth estimation model according to actual usage. The monocular depth estimation model only needs to satisfy the input RGB image to output a depth map. Therefore, the monocular depth estimation system and method of the present disclosure can meet cross-camera requirements and can switch the depth estimation model arbitrarily, with high flexibility.

[0064] The above detailed description includes references to the accompanying drawings, which form part of the detailed description. The accompanying drawings illustrate specific embodiments that can be practiced by way of illustration. These embodiments are also referred to herein as "examples". Such examples may include elements other than those shown or described. However, examples including the elements shown or described are also contemplated. In addition, examples using any combination or arrangement of those elements shown or described are also contemplated, or with reference to the specific examples (or one or more aspects thereof) shown or described herein, or with reference to other examples (or one or more aspects thereof) shown or described herein.

[0065] In the appended claims, the terms "including" and "comprising" are open ended, that is, systems, devices, articles, or processes having elements other than those elements listed after such terms in a claim are still deemed to fall within the scope of that claim. Furthermore, in the appended claims, the terms "first," "second," and "third," etc. are used merely as labels and are not intended to indicate a numerical order to their objects.

[0066] In addition, the order of each operation explained in this specification is exemplary. In alternative embodiments, each operation can be performed in a different order than that shown in the drawings, and each operation can be combined into a single operation or split into more operations.

[0067] The above description is intended to be illustrative, not restrictive. For example, the examples described above (or one or more aspects thereof) may be used in conjunction with other embodiments. Other embodiments may be used, such as by a person of ordinary skill in the art, after reviewing the above description. The abstract allows the reader to quickly determine the nature of the present technology disclosure. The abstract is submitted, and it is understood that the abstract will not be used to interpret or limit the scope or meaning of the claims. In addition, in the above specific embodiments, various features may be grouped together to make the disclosure fluent. However, the claims may not state every feature disclosed herein, because the embodiments may characterize a subset of the features. In addition, the embodiments may include fewer features than those disclosed in a particular example. Therefore, the attached claims are thus incorporated into the specific embodiments, and a claim exists independently as a separate embodiment. The scope of the embodiments disclosed herein should be determined with reference to the attached claims and the full scope of equivalents to which such claims are entitled.

Claims

1. A monocular depth estimation method, It is characterized in that The method comprises: Receiving a single-view image from a camera; Projecting the coordinates of each pixel of the desired projection image onto the pixel plane of the monoscopic image according to the parameters of the camera, thereby obtaining a corresponding relationship between the pixels in the projection image and the pixels of the monoscopic image; Filling pixel information of pixels from the single-view image into corresponding pixels of the projection image according to the corresponding relationship; and The filled projection image is sent to a monocular depth estimation model for depth estimation to obtain a depth map.

2. The method according to claim 1, It is characterized in that The desired projection images include transverse cylindrical projection images.

3. The method according to claim 2, It is characterized in that The desired projection images include Cassini projection images.

4. The method according to claim 3, It is characterized in that The coordinates of each pixel of the desired projection image are coordinates in a Cartesian coordinate system generated according to the size and field angle range of the desired projection image.

5. The method according to claim 4, It is characterized in that The size of the desired projected image includes an image width W and an image height H, and the field of view angle range of the desired projected image includes a field of view angle range [α 0 ,α W-1 ] and the angular range of the image at the image height [β 0 ,β H-1 ].

6. The method according to claim 5, It is characterized in that The size and viewing angle range of the desired projection image are selected to be smaller than or equal to the size of the single-view image.

7. The method according to claim 5, It is characterized in that The Cartesian coordinates of each pixel in the desired projection image are generated using the following formula: Where (x ij ,y ij ,z ij ) is the coordinate of the ijth pixel in the desired projection image in the Cartesian coordinate system; i = 0, 1, 2, ..., W-1; j = 0, 1, 2, ..., H-1; α i =α 0 +i*(α W-1 -α 0 ) / W; β j =β 0 +j*(β H-1 -β 0 ) / H, ρ is any suitable value.

8. The method according to claim 7, It is characterized in that The value of ρ is taken as 1.

9. The method according to claim 1, It is characterized in that The monocular depth estimation model is trained using projected images.

10. The method according to claim 1, It is characterized in that The method also includes sending the depth map to downstream tasks for use.

11. A monocular depth estimation system, It is characterized in that include: A receiving module, wherein the receiving module is configured to receive a single-view image from a camera; A projection module, wherein the projection module is configured to: Projecting the coordinates of each pixel of the desired projection image onto the pixel plane of the monoscopic image according to the parameters of the camera, thereby obtaining a corresponding relationship between the pixels in the projection image and the pixels of the monoscopic image; as well as Filling pixel information of pixels from the single-view image into corresponding pixels of the projection image according to the corresponding relationship; as well as A monocular depth estimation module is configured to perform depth estimation using the padded projection image to obtain a depth map.

12. A monocular depth estimation system, It is characterized in that include: A camera mounted on a vehicle; On-board computer, wherein the camera is configured to capture images and transmit them to the onboard computer, And wherein the onboard computer is configured to perform the method according to any one of claims 1-10.

13. A vehicle comprising a system according to claim 12.

Images