System and method for determining depth information

Abstract

An imaging system for determining depth information of a scene, comprises a first fisheye camera configured to capture a first image of the scene; a second fisheye camera configured to capture a second image at least partially overlapping with the first image, wherein an optical centre of the second fisheye camera is spaced apart from an optical centre of the first fisheye camera along a substantially vertical axis; and a processor configured to receive the first image and second image; and generate depth information for pixels in an overlapping region of the first and second images.

Description

[0001] SYSTEM AND METHOD FOR DETERMINING DEPTH INFORMATION

[0002] FIELD OF THE DISCLOSURE

[0003] The present disclosure relates to an imaging system for determining depth information, particularly to an imaging system including fisheye cameras.

[0004] BACKGROUND OF THE DISCLOSURE

[0005] Stereo depth estimation is a fundamental and crucial task in computer vision, with numerous applications such as autonomous driving, 3D scene understanding, obstacle detection and so on. Conventional stereo cameras are commonly used for depth estimation but are limited by their narrow field of view. To overcome this limitation and achieve a wider and more comprehensive perception field, multiple conventional stereo cameras are typically employed, which introduces complexities and demands significant computational power. Alternatively, fisheye cameras can be used to expand the field of view. However, due to the substantial distortion inherent in the fisheye cameras, accurately estimating depth from fisheye images poses challenges.

[0006] At the core of depth estimation algorithms lies stereo matching, which plays a vital role in determining the depth of objects. Currently, the widely adopted approach for stereo matching is the Semi-Global Matching (SGM) algorithm, which involves a complex algorithmic process. In general, the SGM algorithmic comprises three main steps: cost computation, cost aggregation, and left-right consistency check.

[0007] When estimating the distance between an object and a camera, the common approach involves capturing the object from different camera positions and calculating the depth based on the camera's shooting locations and the pixel coordinates of the object in the images. The pivotal step in this process is matching pixels between the images. For example, consider two images, Image1and Image2, captured by the camera, and you want to find the corresponding point p2in Image2for a given point p1in Image1 / the pixel features and p2should exhibit significant similarity since p±and p2represent the same object in the camera's view. The simplest method would be to measure the similarity between point p1and all the pixels in Image2and select the most similar point as p2■ However, this approach is highly inefficient.

[0008] Fig. 1A an d Fig. IB show a prior art approach to generate depth information.

[0009] There exists an epipolar constraint between images, as illustrated in Fig. 1A. When a camera captures two images, lmage1and lmage2, from locations O and O2, respectively, any point p in Image1will have a corresponding point p2in Image2that lies on a straight line I, referred to as an epipolar line. This constraint allows to search for p2by reducing the comparison of point p with all the pixels in Image2to only the pixels along the epipolar line. The computational workload is significantly reduced.

[0010] To further reduce the computational workload, rectification of the epipolar lines may be performed. A pair of left and right images after the rectification are illustrated in Fig. IB. In stereo matching of this example, for each pixel prin the left image, the similarity with pixels in the same row (epipolar line) of the right image is calculated to find the most similar pixel p2. The difference in x-coordinates between p1and p2is then calculated, which is referred to as disparity. Disparity represents the outcome of stereo matching. Smaller disparities correspond to greater distances between objects and the camera.

[0011] The most crucial challenge lies in addressing the distortion present in fisheye images. The conventional approach involves undistorting the images to obtain regular pinhole camera images. However, this method is limited by the pinhole projection model and fails to accurately represent the ultra-wide field of view captured by fisheye images.

[0012] An alternative approach involves converting a single fisheye image into multiple pinhole camera images oriented in different directions. Depth estimation is then performed separately on each of these images. However, this approach often yields lower quality depth estimation results, particularly at the boundaries between different pinhole camera regions.

[0013] It is an object of the present disclosure to address or at least partially ameliorate some of the above problems of the current approaches. SUMMARY OF THE DISCLOSURE

[0014] Features and advantages of the disclosure will be set forth in the description which follows, and in part will be obvious from the description, or can be learned by practice of the herein disclosed principles. The features and advantages of the disclosure can be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims.

[0015] In accordance with a first aspect of the present disclosure, there is provided an imaging system for determining depth information of a scene. The imaging system comprises a first fisheye camera configured to capture a first image of the scene; a second fisheye camera configured to capture a second image at least partially overlapping with the first image, wherein an optical centre of the second fisheye camera is spaced apart from an optical centre of the first fisheye camera along a substantially vertical axis. The imaging system further comprises a processor configured to receive the first image and second image and generate depth information for pixels in an overlapping region of the first and second images.

[0016] The processor may be configured to generate the depth information by: matching each first image pixel in the overlapping region with a corresponding second image pixel; determining a disparity of the matched pixels; calculating a depth value for the first image pixel. The depth value may be inversely proportional to the disparity.

[0017] The processor may be further configured to transform the first and second images into first and second projected images respectively. A mapping for each image pixel onto its projected pixel coordinate may be determined by calculating an incident light ray vector for each image pixel in each of the first and second images using calibrated intrinsic parameters of the respective cameras; and projecting each incident light ray vector using a preferred projection model to obtain a projected pixel coordinate for each image pixel.

[0018] The step of calculating each incident light ray vector may comprise: determining an initial light ray vector using the camera's calibrated intrinsic parameters; virtually rotating the camera to a new coordinate system where the y-axis of the camera is parallel to a baseline connecting the optical centres of the first and second cameras; and calculating the incident light ray vector based on the transformed camera.

[0019] The first and second cameras' calibrated intrinsic parameters may be based on a double sphere model of a fisheye camera.

[0020] The step of projecting each incident light ray vector using a preferred projection model may comprise: determining a point of intersection between the incident light ray vector and a virtual unit sphere centred on the optical centre of the respective camera. The point of intersection may be defined in a latitude-longitude coordinate system having a polar axis aligned with the vertical axis and a reference meridian aligned with the optical axis of the camera. The x-axis value of the projected pixel coordinate may be proportional to the longitude of the intersection point, and the y-axis value of the projected pixel coordinate may be proportional to the latitude of the intersection point.

[0021] The step of projecting each incident light ray vector using a preferred projection model may comprise determining a point of intersection between the incident light ray vector and a virtual unit cylinder centred on the optical centre of the respective camera and aligned with the vertical axis. The point of intersection may be defined in a cylindrical coordinate system having a longitudinal axis aligned with the vertical axis and an azimuthal angle based on the optical axis of the camera. The x-axis value of the projected pixel coordinate may be proportional to the azimuthal angle of the intersection point, and the y-axis value of the projected pixel coordinate may be proportional to the longitudinal position of the intersection point.

[0022] The optical axes of the fisheye lenses may be substantially parallel to each other and substantially perpendicular to the vertical axis.

[0023] In accordance with a second aspect of the present disclosure, there is provided an autonomous robot. The autonomous robot may comprise a prime mover, the imaging system according to the first aspect of the present disclosure, and a navigation control unit. The navigation control unit may be configured to control the prime mover using depth information output by the imaging system. In accordance with a third aspect of the present disclosure, there is provided a computer- implemented image processing method. The computer-implemented image processing method may comprise: receiving a first image from a first fisheye camera; calculating an incident light ray vector for each image pixel in the first image using calibrated intrinsic parameters of the first camera; projecting each incident light ray vector to determine a mapping for each first image pixel onto its projected first pixel coordinate by determining a point of intersection between the incident light ray vector and a virtual unit cylinder centred on the optical centre of the respective camera and aligned with the vertical axis; and transforming the first image into a first projected image using the determined mapping.

[0024] The point of intersection may be defined in a cylindrical coordinate system having a longitudinal axis aligned with the vertical axis and an azimuthal angle based on the optical axis of the camera.

[0025] The x-axis value of the projected pixel coordinate may be proportional to the azimuthal angle of the intersection point, and the y-axis value of the projected pixel coordinate may be proportional to the longitudinal position of the intersection point.

[0026] The computer-implemented method may further comprise: receiving a second image at least partially overlapping with the first image from a second fisheye camera, wherein an optical centre of the second fisheye camera is spaced apart from an optical centre of the first fisheye camera along a substantially vertical axis; calculating an incident light ray vector for each image pixel in the second image using calibrated intrinsic parameters of the second camera; projecting each incident light ray vector to determine a mapping for each second image pixel onto its projected second pixel coordinate; transforming the second image into a second projected image using the determined mapping; and generating depth information for pixels in an overlapping region of the first and second images.

[0027] The step of generating the depth information may comprise: matching each first image pixel in the overlapping region with a corresponding second image pixel; determining a disparity of the matched pixels; and calculating a depth value for the first image pixel, where the depth value is inversely proportional to the disparity. The step of calculating each incident light ray vector may comprise: determining an initial light ray vector by using the camera's calibrated intrinsic parameters; virtually rotating the camera to a new coordinate system where the y-axis of the camera is parallel to a baseline connecting the optical centres of the first and second cameras; and calculating the incident light ray vector based on the transformed camera.

[0028] The first and second cameras' calibrated intrinsic parameters may be based on a double sphere model of a fisheye camera.

[0029] The optical axes of the fisheye lenses may be substantially parallel to each other and substantially perpendicular to the vertical axis.

[0030] In accordance with a fourth aspect of the present disclosure, there is provided a computer- readable medium comprising instructions which, when executed by a processor, cause the processor to perform the method in accordance with the third aspect of the present disclosure.

[0031] BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to describe the manner in which the above-recited and other advantages and features of the disclosure can be obtained, a more particular description of the principles briefly described above will be rendered by reference to specific embodiments thereof which are illustrated in the appended Figures. Understanding that these Figures depict only exemplary embodiments of the disclosure and are not therefore to be considered to be limiting of its scope, the principles herein are described and explained with additional specificity and detail through the use of the accompanying Figures.

[0033] Preferred embodiments of the present disclosure will be explained in further detail below by way of examples and with reference to the accompanying Figures, in which:-

[0034] Fig. 1A shows a prior art approach to generate depth information.

[0035] Fig. IB shows a prior art approach to generate depth information. Fig. 2 shows a schematic diagram of an imaging system having two fisheye cameras, according to an embodiment.

[0036] Fig. 3 shows a side view of the mounting arrangement of the imaging system of Fig. 2, according to an embodiment.

[0037] Fig. 4 shows a perspective view of a mounting arrangement of the imaging system of Fig. 2, according to an embodiment.

[0038] Fig. 5 shows a schematic diagram of a double sphere projection model.

[0039] Fig. 6 is a schematic diagram showing the rectification of the camera coordinate systems of the two fisheye cameras.

[0040] Fig. 7A and Fig. 7B show two images derived from the two fisheye cameras.

[0041] Fig. 8 shows a schematic diagram of the imaging system of Fig. 2 in a world coordinate system.

[0042] Fig. 9 shows a schematic diagram of the imaging system of Fig. 2 in a camera coordinate system.

[0043] Fig. 10 shows a schematic diagram of a cylindrical projection model.

[0044] Figs. 11A to 11H show the depth estimation results of the imaging system of Fig. 2 and a prior art binocular camera.

[0045] Fig. 12 shows a schematic diagram of an autonomous robot, according to an embodiment.

[0046] Fig. 13 shows a method of obtaining a projected pixel coordinate for each image pixel derived from a fisheye camera.

[0047] Fig. 14 shows a method of generating depth information based on image pixels derived from two fisheye cameras.

[0048] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Various embodiments of the disclosure are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components and configurations may be used without departing from the scope of the disclosure. Referring to the drawings, there is shown in Fig. 2 an imaging system 10 for determining depth information of a scene. The imaging system comprises a first fisheye camera 12, a second fisheye camera 14, and a processor 16.

[0049] The first fisheye camera 12 is configured to capture a first image of the scene. The first fisheye camera 12 may include an ultra-wide fisheye lens. A field of view of the first fisheye camera 12 may be in the range of 110 degrees to 280 degrees. In an example implementation, the field of view (FOV) of the first fisheye camera 12 may be 220 degrees.

[0050] The second fisheye camera 14 is configured to capture a second image at least partially overlapping with the first image. The second fisheye camera may include an ultra-wide fisheye lens. A field of view of the second fisheye camera may be in the range of 110 degrees to 280 degrees. In an example implementation, the field of view of the second fisheye camera may be 220 degrees.

[0051] Fig. 3 shows a side view of the mounting arrangement of the imaging system of Fig. 2, according to an embodiment.

[0052] In some examples, substantially all of the second image may overlap the first image. For example, 90-100% of the second image may overlap with the first image, if the images are aligned. In other examples, the first and second images may overlap only in a small region of interest, for example, 40-60% of the second image may overlap with the first image.

[0053] The first fisheye camera 12 and second fisheye camera 14 may be configured to operate with global exposure settings. The first and second fisheye cameras may be hardware-synchronized. The first and second images simultaneously from both cameras. The images may be captured at substantially the same moment using, for example, MIPI, USB, Ethernet, wireless network, or another suitable means. Each of the first and second fisheye cameras may be configured to output a stream of images. Each frame in each of the respective image streams may be synchronised with a corresponding frame in the other image stream.

[0054] Fig. 4 shows a perspective view of a mounting arrangement of the imaging system of Fig. 2, according to an embodiment.

[0055] An optical centre of the second fisheye camera 14 is spaced apart from an optical centre of the first fisheye camera 12 along a substantially vertical axis. That is, the second fisheye camera 14 may be arranged vertically above the first fisheye camera 12, or vice versa.

[0056] The optical axes of the fisheye lenses may be substantially parallel to each other and substantially perpendicular to the vertical axis.

[0057] In this way, the images from the first fisheye camera 12 and the second fisheye camera 14 can be overlapped. Substantially all of the second image can be overlapped with the first image.

[0058] It can be understood that other arrangements of the two fisheye cameras are also available. For example, the second fisheye camera may be offset from the first fisheye camera, or may be oriented differently. In some implementations, the overlapping portion may be reduced to increase the overall field of vision, or to alter the shape of the overlapping portion for a specific purpose or use case. In some examples, the relative arrangement of the first and second fisheye cameras may be adjustable or moveable to achieve different fields of view and different overlapping regions according to the situation.

[0059] The processor 16 as illustrated in Fig. 2 is configured to receive the first image and second image and generate depth information for pixels in an overlapping region of the first and second images.

[0060] The vertically arranged dual-fisheye camera system of the disclosure can provide depth information over a horizontal field of view which is significantly greater than existing pinhole camera systems. The vertical arrangement of cameras can utilize the cameras' large field of view while also minimizing algorithm complexity and enhancing algorithm accuracy as much as possible. As descripted in more detail below, this vertical arrangement can improve the identification of epipolar lines in the first and second images. The vertical arrangement enables new rectification algorithms, which can significantly reduce complexity and computational demand. The effects of lens distortion inherent to fisheye cameras can be significantly reduced by such algorithms. This can enable the application of pixel matching algorithms which are less computationally intensive, such as SGM. As such, the overall algorithm complexity to generate depth information can be significantly reduced. In some cases, the arrangement can allow the adaption of existing models to the fisheye camera-based system, reducing development costs.

[0061] As compared to existing imaging systems with multiple conventional stereo cameras, the imaging system with two cameras of the disclosure is simple but can still achieve a wide field of view, for example 220 degrees or more. The overall assembly procedure can be simplified accordingly, and manufacturing costs can be reduced.

[0062] The imaging system can provide mobile robots with improved obstacle avoidance. The wide field of view provided can ensure that these robots have a comprehensive understanding of their environment over a wide field of view, improving the ability of the robot to navigate and avoid obstacles.

[0063] It can be understood that the vertical arrangement for reducing lens distortion can also be applied in other scenarios, such as drone piloting, lawn mowers, vacuum cleaners, food delivery robots, quadruped robots, humanoid robots, etc.

[0064] To generate the depth information, the processor may be configured to match each first image pixel in the overlapping region with a corresponding second image pixel.

[0065] The first fisheye camera 12 may capture the first image from a first location O1. The second fisheye camera 14 may capture the second image from a second location O2. A point in the scene may be imaged as a pixel p1in the first image. The same point may be imaged as a pixel p2in the second image. Theoretically, as the pixel p1and the pixel p2rePreser|t the same point, the pixel features of the pixel p1and the pixel p2should be similar. To identify a matched image pixel in the second image for each pixel in the first image, the processor may be configured to calculate a similarity of each pixel in the first image with all the pixels in the second image. For example, the processor may calculate the similarly of the pixel ^with every pixel in the second image. The processor may be configured to identify the pixel in the second image with the highest similarity as the matched image pixel for the pixel in the first image. The pixel p2should have the greatest similarity to the pixel p±out of the plurality of pixels in the second image. This approach may be referred to as a "global matching" approach.

[0066] The processor may be configured to match pixels using a similarity in any of the pixel values or parameters. The processor may be configured to match an image pixel based on a similarity of neighbouring pixels. The processor may be configured to search a limited area of the second image based on an expected disparity range. In some examples, the processor may be configured to match pixels using a pre-trained model which trained with a machine learning process using pairs of images with pre-labelled pixels as training data.

[0067] The processor may be configured to use an epipolar constraint between the two images. As shown described with respect to Fig. 1A, for the pixel p±in the first image, its matched pixel p2\n the second image must be on the epipolar line. When searching for the pixel p2, the processor may be configured to compare the pixel p with only the pixels on the epipolar line in the second image. In this way, the processor can reduce the computing workload of the above "global matching" approach. The processor may be configured to calculate the epipolar line based on the extrinsic parameters of the first and second cameras. For example, the processor may be configured to calculate the epipolar line based on the relative rotation and relative translation of the cameras.

[0068] The processor may be configured to determine a disparity of the matched pixels. The disparity d may be calculated by the processor from the coordinate differences of the matched pixels of the first image and the second image.

[0069] The disparity d may be derived by the below formula: where the focal length f and the baseline length b are constants for a given stereo system.

[0070] As such, the processor may be configured to calculate a depth value for the first image pixel, where the depth value is inversely proportional to the disparity.

[0071] The disparity d may be inversely proportional to the distance of the object from the camera in the z-direction. In other words, disparity is only related to the object's distance and is independent of other factors. A smaller disparity means a greater distance between the point on an object and the camera.

[0072] Before calculating the depth information, the processor may be configured to transform the first and second images into first and second projected images respectively.

[0073] The transforming may be based on a mapping for each image pixel onto its projected pixel coordinate.

[0074] To determine the mapping, the processor 16 may be configured to calculate an incident light ray vector for each image pixel in each of the first and second images. The processor may be configured to calculate the incident light ray vectors using calibrated intrinsic parameters of the respective cameras.

[0075] For example, the first and second cameras' calibrated intrinsic parameters may be based on a double sphere model of a fisheye camera. Fig. 5 shows a schematic diagram of an exemplary double sphere model.

[0076] As shown, the refraction process of the fisheye lenses is approximated by two spherical surfaces. An illustration of the ray tracing of the double sphere model is shown in Fig. 5. As shown, a pixel p can be associated with an incident light ray vector L. An associated incident light ray vector may be calculated for each image pixel. Each of the incident light ray vectors may be a unit vector or a vector of arbitrary length, terminating as a spatial (3D) point P (X, Y, Z) in the camera coordinate system.

[0077] It can be understood that there are different Fisheye camera projection models for simplifying and approximating the actual camera optics of fisheye cameras. From the experimentation, the approximation method of the double sphere model can provide a better performance as compared to other fisheye models.

[0078] However, other models are also possible, such as Unified Camera Model (UCM), Kannala Brandt Camera Model (KBCM), Extended Unified Camera Model (EUCM), Triple Sphere Camera Model (TSCM), etc.

[0079] To calculate each incident light ray vector, the processor may be configured to determine an initial light ray vector using the camera's calibrated intrinsic parameters. The processor may be configured to virtually rotate each camera to a new coordinate system. In the new coordinate system, the y-axis of each camera may be parallel to a baseline connecting the optical centres of the first and second cameras.

[0080] Fig. 6 is a schematic diagram showing the rectification of the camera coordinate systems of the two fisheye cameras. This virtually rotation may be known as epipolar rectification.

[0081] When the two cameras with the optical centres 0 and 02a reparallel and only have relative displacement in the vertical direction, the epipolar lines in each image can be made vertical. The horizontal displacement of the line in the second image in the x-direction may be consistent with the x-coordinate of the point p in the first image. In other words, the processor can assume that the x-coordinate of the point p2in the second image is always the same as the x-coordinate of point p in the first image.

[0082] Due to mechanical installations and other factors, it is unlikely that the orientations of the two cameras will be perfectly parallel. When the two cameras are not parallel, the epipolar lines may be slanted, and it is necessary to compute the epipolar line equation for each pixel as described above. By virtually transforming the cameras to make them parallel, the processor can reduce computational time. The processor may be configured to match image pixels by searching pixels along the y-axis only. The processor may be configured to determine a disparity of the matched pixels. From a data structure perspective, the epipolar rectification can prevent discontinuous array index accesses, further reducing processing time.

[0083] An epipolar rectification algorithm may define a coordinate system as follows: the y-axis of the new coordinate system may be made parallel to the baseline (the line connecting the optical centres of the two camera centres); the x-axis of the new coordinate system may be made perpendicular to the y-axis; the z-axis of the new coordinate system may be made perpendicular to the xy-plane. The equation for calculating the new coordinate system may be as follows. where x1,y1, z1) represents the vectors of the new coordinate system's three axes.

[0084] Epipolar rectification algorithms designed for left-right stereo configurations may be used for the vertical configuration of this disclosure. To adapt an epipolar rectification algorithm to the vertical configuration, modifications may be made including defining the coordinate system as above. An example of such an epipolar rectification method is the Fusiello method, but any suitable method can be applied.

[0085] The processor may be configured to calculate the incident light ray vector based on the transformed camera.

[0086] For each camera, there may be the original coordinate system T and the new coordinate system T (i.e., the coordinate system after epipolar rectification). For each pixel, the corresponding initial light ray vector calculated using the camera's calibrated intrinsic parameters may be transformed from the coordinate system T to the coordinate system T1. The spatial point P’(X', Y', Z') associated with each initial light ray vector may be transformed to a spatial point P (X, Y, Z) associated with the incident light ray vector. The incident light ray vector may be visualised as a vector connecting the origin of the camera coordinate system and the spatial point P (X, Y, Z).

[0087] Fig. 7A and Fig. 7B show two images derived from the two fisheye cameras, demonstrating the result of epipolar rectification. After epipolar rectification, it can be observed that the corresponding points for each pixel are aligned along the same vertical line, indicating that the epipolar lines are now vertical.

[0088] This allows the use of stereo matching algorithms, such as the Semi-Global Matching (SGM) algorithm. The SGM algorithm is designed for left-right stereo configuration and includes three main steps: cost computation, cost aggregation, and left-right consistency check. Given a pair of left and right images, after rectification, for each pixel in the left image, the similarity with pixels in the same row of the right image may be calculated to find the most similar pixel p2. Then, the difference in x-coordinates between p and p2may be calculated, which is referred to as disparity. Smaller disparities may correspond to greater distances between objects and the camera.

[0089] The SGM algorithm may be modified and adapted for stereo matching in the vertical arrangement. In the modified SGM algorithm, the matching point search direction in the cost computation step may be changed from the horizontal direction to the vertical direction. The remaining steps remain consistent with the original SGM algorithm.

[0090] Other stereo matching algorithms may also be implemented but may lead to higher computational demands and longer processing times.

[0091] For projecting each incident light ray vector, the processor may be configured to use a preferred projection model to obtain a projected pixel coordinate for each image pixel.

[0092] One well-known projection model is a pinhole mode. The preferred projection model may be a pinhole projection model. However, the pinhole mode cannot represent images with a large field of view. When the field of view approaches 180 degrees, the image plane would become nearly infinite in size, which is not achievable in reality. Generally, the field of view by a single pinhole projection is limited to, for example, 100 degrees. The preferred projection model may be a multiple pinhole projection model. A single fisheye image may be projected into multiple pinhole camera images oriented in different directions. The depth may be estimated for each pinhole image separately. However, this method results in lower depth estimation quality at the boundaries between different pinhole camera regions. This method also needs extra processing. In the pinhole projection model, the disparity is solely dependent on the object's z-coordinate in the camera coordinate system, independent of the x and y coordinates.

[0093] The preferred projection model may be a spherical equidistant projection model. This projection model can be visualized as unwrapping a spherical surface, with the horizontal direction ranging from 0 to 360 degrees (longitude) and the vertical direction ranging from -90 to 90 degrees (latitude). To project each incident light ray vector, the processor may be configured to determine a point of intersection between the incident light ray vector and a virtual unit sphere centred on the optical centre of the respective camera.

[0094] As shown in Fig. 8, the imaging system of Fig. 2 may be represented as two unit spheres in a world coordinate system. Each of the fisheye cameras may be represented as one of the unit spheres. The camera's optical centre may be the sphere's centre. A spatial point P may be projected onto each one of the unit spheres by calculating the incident light ray vector and determining the point of intersection between the incident light ray vector and the unit sphere.

[0095] As shown, the longitudes of the two points of intersection of the two cameras are consistent. This is a result of the vertical arrangement of cameras. Conversely, when the cameras are placed leftright, the latitudes of the two points of intersection are consistent only when a spatial point lies on the equatorial plane. Otherwise, the points of intersection of a spatial point on the two unit spheres will have different longitudes and latitudes.

[0096] In the spherical equidistant projection model, the point of intersection may be defined in a latitude-longitude coordinate system. The latitude-longitude coordinate system may have a polar axis aligned with the vertical axis and a reference meridian aligned with the optical axis of the camera. The x-axis value of the projected pixel coordinate may be proportional to the longitude of the intersection point. In this way, the x-coordinates of the two projection points which correspond to the same 3D point in the scene may be identical in the sphere equidistant projection model.

[0097] The y-axis value of the projected pixel coordinate may be proportional to the latitude of the intersection point.

[0098] The x-axis value and the y-axis value of the projected pixel coordinate may also be represented in a spherical coordinate system. The spherical coordinate system may be defined to have a polar angle based on the vertical axis and an azimuthal angle based on the optical axis of the camera. The x-axis value of the projected pixel coordinate may be proportional to the longitude of the intersection point. The y-axis value of the projected pixel coordinate may be proportional to the latitude of the intersection point.

[0099] By projecting the images using the spherical equidistant projection model, the processor can mitigate the distortion inherent to fisheye images. In the spherical equidistant projection model, the epipolar lines that are curved due to distortion can be transformed into straight lines. In this way, the processing required to calculate the epipolar lines can be significantly reduced.

[0100] In combination with the epipolar rectification, the epipolar lines can be made parallel to the baseline of the two cameras. The straight epipolar lines can be made parallel to the image edges. The epipolar lines can be aligned with the lines of longitude. The pixel matching searching can be reduced from two dimensional to one dimensional by using the epipolar lines.

[0101] The combination of vertical arrangement and spherical equidistant projection allows SGM on the longitudinal axis. The processes for epipolar rectification and image undistortion can be combined into a single step, as both processes involve transforming one projection model into another while also rotating the camera's coordinate system.

[0102] Fig. 9 shows a schematic diagram of the imaging system of Fig. 2 in a camera coordinate system.

[0103] In the sphere equidistant projection model using a latitude-longitude coordinate system, pixel coordinates may be directly proportional to latitude and longitude. Latitude and longitude may refer to the coordinates of the intersection points of incoming light rays on the spherical surface, where the camera's optical centre is the sphere's centre. The y-coordinate may correspond to the latitude, representing the angle between the incident ray and the y-axis.

[0104] Pixel coordinates may be proportional to the angles at which light rays enter the camera. The disparity may be proportional to the angular difference between light rays. Consequently, the disparity in the image coordinate system may be equivalent to the angular difference of the incident rays.

[0105] In the example shown in Fig.9, two spatial points, and P2, are located at the same distance from the camera in the z-direction. For the point the angle of the incoming light ray may be defined as the angle between the incoming light ray and the negative half of the x-axis, denoted as (9Xand 02in the figure. The difference in the angles of incidence is 86 = 9±— 62. In this example, the difference in the angles of incidence for P2is greater than that for Px.

[0106] In the equidistant model, disparity is not only related to the object's distance in the z-direction but also to the object's position in the baseline direction. Moreover, objects located further from the camera in the baseline direction and further from the camera in the z-direction result in smaller disparities. It can be understood that when the incident ray's angle is too large or too small, the angular difference becomes difficult to distinguish. Since disparity is calculated using pixel coordinates, disparities that are too small (e.g., less than 1 pixel) can become difficult to distinguish. This can affect the quality of the final depth estimation.

[0107] In the equidistant projection model, the longitude can express a maximum field of view of 180 degrees. The areas near 180 degrees may suffer from distortion. For example, the areas near the poles in the images from the spherical equidistant projection may have severe stretching.

[0108] In some examples, the equidistant projection model may be modified to at least solve the above identified problems, such as the distortion.

[0109] The preferred projection model may be a cylindrical equidistant projection model. The cylindrical equidistant projection model may combine equidistant projection in the x-direction with pinhole projection in the y-direction. The vertical direction of the spherical equidistant projection may be replaced with a pinhole projection.

[0110] The pinhole projection in the y-direction has fewer factors affecting the magnitude of disparities which may be more algorithmically friendly and the equidistant projection in the x-direction can represent a larger field of view, resulting in a combination of the advantages of the pinhole model and the equidistant model.

[0111] In the cylindrical equidistant projection model, to project each incident light ray vector, the processor may be configured to determine a point of intersection between the incident light ray vector and a virtual unit cylinder. The virtual unit cylinder may be centred on the optical centre of the respective camera and aligned with the vertical axis. The imaging system may be represented as two unit cylinders in a world coordinate system. Each of the fisheye cameras may be represented as one of the unit cylinders. The camera's optical centre may be the cylinder's centre. Each of the virtual unit cylinder may be aligned with the baseline between the two cameras.

[0112] Fig. 10 shows a schematic diagram of a cylindrical equidistant projection model.

[0113] As shown in Fig. 10, the projection process of the cylindrical equidistant projection model can be visualized as projecting points in space onto a unit cylinder. This can be considered a combination of the pinhole model projection onto a unit plane, and the spherical equidistant model projection onto a unit sphere.

[0114] The point of intersection may be defined in a cylindrical coordinate system. The cylindrical coordinate system may have a longitudinal axis aligned with the vertical axis and an azimuthal angle based on the optical axis of the camera. The x-axis value of the projected pixel coordinate may be proportional to the azimuthal angle of the intersection point. The y-axis value of the projected pixel coordinate may be proportional to the longitudinal position of the intersection point. An undistorted image can be obtained by unwrapping the cylinder. In the cylindrical projection model, the lines of longitude can still be made straight and the epipolar lines can be aligned with the lines of longitude.

[0115] The projection formula of the cylindrical projection model may be expressed as follows: where, (x,y) represents the projected image coordinates, (X, Y, Z) denotes the spatial coordinates of the point P in the camera coordinate system, f represents the camera focal length.wimg represents the image width. cyrepresents the origin vertical offset same as pinhole model. This formula can model the process of projecting a 3D scene onto a 2D image plane of the cylindrical projection model.

[0116] In back projection or reverse projection (also known as inverse projection, the formulas are as follows:- where, (x, y) represents the projected image coordinates, (X, K, Z) denotes the spatial coordinates of the point in the camera coordinate system, f represents the camera focal length.wimg represents the image width. cyrepresents the origin vertical offset same as pinhole model.

[0117] The cylindrical projection model can provide accurate disparities at both large and small angles. The overall quality of the depth estimation can be enhanced. It can provide a more effective way for depth estimation in ultra-wide-angle fisheye camera systems. The cylindrical projection model can also be used in other fisheye camera systems with different arrangements.

[0118] Figs. 11A to 11H show the depth estimation results of the imaging system of Fig. 2 and a prior art binocular camera.

[0119] Fig. 11A an d Fig. 11B d epict a set of images captured by the imaging system comprising two fisheye cameras as shown in Fig. 2- The image of Fig. 11A is captured by the first (top) camera 12 of the imaging system 10 of Fig. 2- The image of Fig. 11B is captured by the second (bottom) camera 14 of the imaging system 10 of Fig. 2. The imaging system may have a horizontal field of view of 220 degrees.

[0120] Fig. lie an d Fig. HD are images obtained by the processor using the cylindrical projection model. These images may be undistorted. As discussed above, the lines of longitude in these images are still all straight and the epipolar lines are aligned with the lines of longitude.

[0121] Fig. HE an d Fig. 11F are images obtained by an imaging system with two pinhole cameras arranged in a left-right arrangement.

[0122] Fig. 11G is a disparity map obtained by the left-right arrangement of pinhole cameras. Fig. 11H is disparity map obtained by the vertical arrangement of fisheye cameras. As the colours shift gradually from light to dark, the disparity decreases, suggesting that the distance from the camera is getting larger. The vertical arrangement can provide a wider angle of view and more depth information.

[0123] Fig. 12 shows a schematic diagram of an autonomous robot 20, according to an embodiment.

[0124] The autonomous robot 20 may comprise a prime mover 22, an imaging system 24 and a navigation control unit 26.

[0125] The prime mover 22 may generate motion and enable the robot to move, allowing the robot to navigate its environment, perform tasks, and respond to its surroundings. In some examples, the imaging system 24 in the robot may include two fisheye cameras arranged in vertical arrangement as shown in Fig. 2. The imaging system 24 may capture visual information from the robot's environment. The imaging system allows the robot to perceive and understand the world around it through images or video. The imaging system 24 may capture images and / or video from the robot's environment, to allow perception, sensing, object recognition and tracking, obstacle avoidance, and navigation and mapping, and the like.

[0126] The navigation control unit 26 of the robot 20 may control and coordinate the robot's movement and navigation within its environment. The navigation control unit 26 may include various components and algorithms that enable the robot to plan and execute its motion, avoid obstacles, and reach desired locations. The prime mover, imaging system, and navigation control unit may work collaboratively to enable the robot's autonomous operation. The imaging system may provide visual input to the navigation control unit, which processes this data to make informed decisions about the robot's motion and navigation. For example, the navigation control unit can identify obstacles or plan a path to a specific location based on the visual information captured by the imaging system. It then sends control signals to the prime mover to execute the planned movements, allowing the robot to navigate autonomously while avoiding obstacles and reaching its intended destinations.

[0127] It should be understood that the robot may also have networking capabilities, to enable remote control and monitoring, data transmission and analysis, cloud integration and collaboration, Internet of Things (loT) integration, etc. The autonomous robot may be, for example, a manufacturing robot, a household robot such as a robotic vacuum / mop or lawnmower, a mobile utility robot for varied tasks such as delivery, maintenance, personal transit, disposal etc. The autonomous robot may be fixed in a location or may be mobile on the ground (e.g. with wheels / tracks) or in the air (e.g. with wings and / or one or more rotors). By implementing the autonomous robot with the imaging system, the autonomous robot can be provided with improved depth vision over a wider field of view. This can allow the autonomous robot to process the surrounding environment more efficiently, navigate more precisely and effectively, and accomplish a task more quickly and safely. Fig. 13 shows a method of obtaining a projected pixel coordinate for each image pixel derived from a fisheye camera. The method starts at step SOI.

[0128] At step S02, a first image is received from a first fisheye camera.

[0129] At step S03, an incident light ray vector for each image pixel in the first image is calculated using calibrated intrinsic parameters of the first camera.

[0130] At step S04, each incident light ray vector is projected to determine a mapping for each first image pixel onto its projected first pixel coordinate. The projecting may comprise determining a point of intersection between the incident light ray vector and a virtual unit cylinder centred on the optical centre of the respective camera and aligned with the vertical axis.

[0131] The point of intersection may be defined in a cylindrical coordinate system having a longitudinal axis aligned with the vertical axis and an azimuthal angle based on the optical axis of the camera. The x-axis value of the projected pixel coordinate may be proportional to the azimuthal angle of the intersection point, and the y-axis value of the projected pixel coordinate may be proportional to the longitudinal position of the intersection point.

[0132] At step S05, the first image is transformed into a first projected image by using the determined mapping.

[0133] The method finishes at step S06.

[0134] Fig. 14 shows a method of generating depth information based on image pixels derived from two fisheye cameras. The method starts at step Sil.

[0135] At step S12, a first image is received from a first fisheye camera.

[0136] At step S13, a second image at least partially overlapping with the first image is received from a second fisheye camera. An optical centre of the second fisheye camera is spaced apart from an optical centre of the first fisheye camera along a substantially vertical axis. At step S14, an initial light ray vector for each image pixel in the first and second images may be determined by using calibrated intrinsic parameters of the first camera and the second camera respectively.

[0137] At step S15, the first and second cameras may be virtually rotated to a new coordinate system respectively. In the new coordinate system, the y-axis of each camera may be parallel to a baseline connecting the optical centres of the first and second cameras.

[0138] At step S16, an incident light ray vector may be calculated for each image pixel in the first and second images based on the respective transformed camera.

[0139] At step S17, each incident light ray vector may be projected to determine a mapping for each image pixel onto its projected pixel coordinate.

[0140] The projecting may comprise determining a point of intersection between the incident light ray vector and a virtual unit cylinder which is centred on the optical centre of the respective camera and aligned with the vertical axis.

[0141] The point of intersection may be defined in a cylindrical coordinate system having a longitudinal axis aligned with the vertical axis and an azimuthal angle based on the optical axis of the camera.

[0142] The x-axis value of the projected pixel coordinate may be proportional to the azimuthal angle of the intersection point, and the y-axis value of the projected pixel coordinate may be proportional to the longitudinal position of the intersection point.

[0143] At step S18, the first and second images are transformed into a first projected image and a second projected image by using the respective determined mapping, respectively.

[0144] At step S19, each image pixel ("first image pixel") of the first projected image in the overlapping region is matched with a corresponding image pixel ("second image pixel") of the second projected image.

[0145] At step S20, a disparity of the matched pixels is determined. At step S21, a depth value for the first image pixel is calculated, where the depth value is inversely proportional to the disparity.

[0146] The method finishes at step S22-

[0147] The above embodiments are described by way of example only. Many variations are possible without departing from the scope of the disclosure as defined in the appended claims.

[0148] For clarity of explanation, in some instances the present technology may be presented as including individual functional blocks including functional blocks comprising devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software.

[0149] Methods according to the above-described examples can be implemented using computerexecutable instructions that are stored or otherwise available from computer readable media. Such instructions can comprise, for example, instructions and data which cause or otherwise configure a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during methods according to described examples include magnetic or optical disks, flash memory, Universal Serial Bus (USB) devices provided with non-volatile memory, networked storage devices, and so on.

[0150] Devices implementing methods according to these disclosures can comprise hardware, firmware and / or software, and can take any of a variety of form factors. Typical examples of such form factors include laptops, smart phones, small form factor personal computers, personal digital assistants, and so on. Functionality described herein also can be embodied in peripherals or add-in cards. Such functionality can also be implemented on a circuit board among different chips or different processes executing in a single device, by way of further example. The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are means for providing the functions described in these disclosures.

[0151] Although a variety of examples and other information was used to explain aspects within the scope of the appended claims, no limitation of the claims should be implied based on particular features or arrangements in such examples, as one of ordinary skill would be able to use these examples to derive a wide variety of implementations. Further and although some subject matter may have been described in language specific to examples of structural features and / or method steps, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to these described features or acts. For example, such functionality can be distributed differently or performed in components other than those identified herein. Rather, the described features and steps are disclosed as examples of components of systems and methods within the scope of the appended claims.

Claims

CLAIMS1. An imaging system for determining depth information of a scene, comprising: a first fisheye camera configured to capture a first image of the scene; a second fisheye camera configured to capture a second image at least partially overlapping with the first image, wherein an optical centre of the second fisheye camera is spaced apart from an optical centre of the first fisheye camera along a substantially vertical axis; and a processor configured to: receive the first image and second image; and generate depth information for pixels in an overlapping region of the first and second images.

2. The imaging system of claim 1, wherein the processor is configured to generate the depth information by: matching each first image pixel in the overlapping region with a corresponding second image pixel; determining a disparity of the matched pixels; calculating a depth value for the first image pixel, where the depth value is inversely proportional to the disparity.

3. The imaging system of claim 1 or claim 2, wherein the processor is further configured to: transform the first and second images into first and second projected images respectively; wherein a mapping for each image pixel onto its projected pixel coordinate is determined by: calculating an incident light ray vector for each image pixel in each of the first and second images using calibrated intrinsic parameters of the respective cameras; and projecting each incident light ray vector using a preferred projection model to obtain a projected pixel coordinate for each image pixel.

4. The imaging system of claim 3, wherein calculating each incident light ray vector comprises:determining an initial light ray vector using the camera's calibrated intrinsic parameters; virtually rotating the camera to a new coordinate system where the y-axis of the camera is parallel to a baseline connecting the optical centres of the first and second cameras; and calculating the incident light ray vector based on the transformed camera.

5. The imaging system of claim 3 or claim 4, wherein the first and second cameras' calibrated intrinsic parameters are based on a double sphere model of a fisheye camera.

5. The imaging system of any one of claims 3 to 5, wherein projecting each incident light ray vector using a preferred projection model comprises: determining a point of intersection between the incident light ray vector and a virtual unit sphere centred on the optical centre of the respective camera, wherein the point of intersection is defined in a latitude-longitude coordinate system having a polar axis aligned with the vertical axis and a reference meridian aligned with the optical axis of the camera; wherein the x-axis value of the projected pixel coordinate is proportional to the longitude of the intersection point, and the y-axis value of the projected pixel coordinate is proportional to the latitude of the intersection point.

7. The imaging system of any one of claims 3 to 5, wherein projecting each incident light ray vector using a preferred projection model comprises: determining a point of intersection between the incident light ray vector and a virtual unit cylinder centred on the optical centre of the respective camera and aligned with the vertical axis, wherein the point of intersection is defined in a cylindrical coordinate system having a longitudinal axis aligned with the vertical axis and an azimuthal angle based on the optical axis of the camera; wherein the x-axis value of the projected pixel coordinate is proportional to the azimuthal angle of the intersection point, and the y-axis value of the projected pixel coordinate is proportional to the longitudinal position of the intersection point.

8. The imaging system of any preceding claim, wherein the optical axes of the fisheye lenses are substantially parallel to each other and substantially perpendicular to the vertical axis.

9. An autonomous robot comprising: a prime mover; an imaging system of any preceding claim; and a navigation control unit configured to control the prime mover using depth information output by the imaging system.

10. A computer-implemented image processing method, comprising: receiving a first image from a first fisheye camera; calculating an incident light ray vector for each image pixel in the first image using calibrated intrinsic parameters of the first camera; projecting each incident light ray vector to determine a mapping for each first image pixel onto its projected first pixel coordinate by determining a point of intersection between the incident light ray vector and a virtual unit cylinder centred on the optical centre of the respective camera and aligned with the vertical axis; and transforming the first image into a first projected image using the determined mapping, wherein the point of intersection is defined in a cylindrical coordinate system having a longitudinal axis aligned with the vertical axis and an azimuthal angle based on the optical axis of the camera; wherein the x-axis value of the projected pixel coordinate is proportional to the azimuthal angle of the intersection point, and the y-axis value of the projected pixel coordinate is proportional to the longitudinal position of the intersection point.

11. The computer-implemented method of claim 10, further comprising: receiving a second image at least partially overlapping with the first image from a second fisheye camera, wherein an optical centre of the second fisheye camera is spaced apart from an optical centre of the first fisheye camera along a substantially vertical axis;calculating an incident light ray vector for each image pixel in the second image using calibrated intrinsic parameters of the second camera; projecting each incident light ray vector to determine a mapping for each second image pixel onto its projected second pixel coordinate; transforming the second image into a second projected image using the determined mapping; and generating depth information for pixels in an overlapping region of the first and second images.

12. The computer-implemented method of claim 11, wherein generating the depth information comprises: matching each first image pixel in the overlapping region with a corresponding second image pixel; determining a disparity of the matched pixels; and calculating a depth value for the first image pixel, where the depth value is inversely proportional to the disparity.

13. The computer-implemented method of claim 11 or claim 12, wherein calculating each incident light ray vector comprises: determining an initial light ray vector using the camera's calibrated intrinsic parameters; virtually rotating the camera to a new coordinate system where the y-axis of the camera is parallel to a baseline connecting the optical centres of the first and second cameras; and calculating the incident light ray vector based on the transformed camera.

14. The computer-implemented method of any one of claims 11 to 13, wherein the first and second cameras' calibrated intrinsic parameters are based on a double sphere model of a fisheye camera.

15. The computer-implemented method of any one of claims 11 to 14, wherein the optical axes of the fisheye lenses are substantially parallel to each other and substantially perpendicular to the vertical axis.

16. A computer-readable medium comprising instructions which, when executed by a processor, cause the processor to perform the method of any one of claims 10 to 15.

Images