Depth map generation for 2D panoramic images

Abstract

Techniques for generating a depth map for a 2D panoramic image are provided. A method performed by an image processing apparatus. The method includes obtaining a depth map for a 2D panoramic image of a 3D environment. The depth map is generated from a 3D point cloud of the 3D environment. The depth map includes some pixels having unknown depth values. The method includes generating an edge map for the depth map. The edge map is generated from the depth map and indicates a position in the 3D point cloud of an edge in the depth map having an unknown depth value. The method includes calculating a respective depth value for each of the pixels having an unknown depth value in the depth map. Each respective depth value is calculated as a weighted interpolation of a known depth value in an area in the depth map surrounding the respective unknown depth value, where the known depth value does not intersect any of the edges in the edge map.

Description

Technical Field

[0001] Embodiments presented herein relate to a method, an image processing device, a computer program and a computer program product for generating a depth map for a two-dimensional (2D) panoramic image. Background Art

[0002] Generally speaking, in the process of three-dimensional (3D) reconstruction, the scene geometry can be represented by a 3D point cloud. In this regard, a 3D point cloud (denoted as Ω) can be viewed as an unstructured set of K points in 3D space (with dimensions X, Y, Z).

[0003]

[0004] 3D point clouds can be used to capture scene geometry and scale, thereby representing 3D structures from the physical world.

[0005] 3D point clouds can be generated by passive scanning (e.g., registering multiple 2D images of a scene and estimating depth values ​​through triangulation) or active scanning (e.g., Light Detection and Ranging (LIDAR), where depth values ​​are estimated by measuring the time of flight of emitted light).

[0006] Since the physical scene to be scanned may be large or have complex geometry, the scanning device is usually placed on a tripod to perform the scan. The scanner is then moved to a new position, where a new scan is performed. In each of these positions, the scanning device rotates and performs a 360-degree scan of the environment. Therefore, a scan performed at one position is called a sweep. A sweep for a given position consists of a 3D point cloud generated from the given scan position, parameters for the given scan position, and a set of 2D images collected at the given scan position.

[0007] Users can explore the point cloud directly using different types of software tools. However, it can be cumbersome for users to navigate and perform measurements directly in a 3D point cloud. An alternative approach to enabling navigation and measurement in a 3D reconstructed scene is to render a 2D panoramic image in a skybox image rendering environment based on the underlying 3D point cloud. In this approach, the user sees a panoramic image (and hence the term skybox) projected onto the face of a cube. In general, the source of the skybox can be any form of texture, including a photograph, a hand-drawn image, or pre-rendered 3D geometry. The following assumes that the source of the skybox is a 3D point cloud, and that the 3D point cloud is projected as a panoramic image that is created and aligned in 6 directions with a 90-degree viewing angle (which covers the 6 faces of the cube). This can be achieved through cube mapping. In general, cube mapping is a technique for creating pre-rendered panoramic sky images that are then rendered by a graphics engine as faces of a cube at almost infinity with the viewpoint at the center of the cube. A skybox is formed from a 2D panoramic image 110 obtained from a single sweep of the 3D point cloud. In Figure 1 FIG. 1 shows a skybox 100 ( Figure 1 (a)) of an example skybox image rendering environment, wherein a 2D panoramic image 110 is rendered on each face of a skybox 100 ( Figure 1 (b)) A single image I n,s ={I n,-90 ,I n,0 ,I n,+90 ,I n,360 ,I n,上 ,I n,下}.

[0008] Navigation in the 3D reconstructed scene is then enabled by letting the user move from one cube to another (which corresponds to jumping from one sweep to another). Furthermore, measurement is enabled by exploiting the correspondence between image pixels and corresponding 3D points in the 3D point cloud. In this way, the user can perform measurements in the scene by clicking on pixels, but the actual size or distance is calculated based on the underlying 3D point cloud. That is, the actual measurement is performed on the 3D point cloud (i.e., between points in 3D space), for which depth information is required.

[0009] One problem with existing techniques for generating depth maps is that the generated depth maps may be sparse and, therefore, incomplete. That is, the depth map may lack depth values ​​at certain pixel locations. This, in turn, hinders accurate measurements based on the depth map.

[0010] Generally speaking, there are two common reasons for an incomplete depth map. These are discussed next. The first is that the resolution of a 3D point cloud is typically lower than that of a 2D image. Therefore, when the 3D point cloud is projected onto a 2D image to create a depth map, there are gaps of missing data between the depth points. The second reason is that certain surfaces and edges in the scanned environment do not properly reflect the scanner beam back to the scanner. This can happen because a beam that contacts the edge of an object can return two reflections: one from the object and one from behind the object. This forces the scanner to discard that beam entirely. Figure 2 An example of this is shown in . Figure 2 A schematic diagram of why most object edges tend to be between measurement points for active scanning scenarios is provided in . Figure 2 An example setup 210 is shown in (a), where an active scanner (cylinder) 215 emits laser rays (arrows) 225, 230 that either illuminate an object (cube) 220 or the scene background above / behind the object. Figure 2 (b) shows the Figure 2 Approximate depth map 250 with measured depth values ​​for the example setup 210 in (a). Each laser ray returns a distance, and thus a depth value. Low amplitude depth values ​​are shown as x:es 265, and high amplitude depth values ​​are shown as +:es 260. Due to the limitation of the active scan resolution, most rays ( Figure 2 The solid arrow 225 in (a) will not hit the edge of the object 220, but will hit it below or above. If the ray does hit the edge ( Figure 2 (a) dashed arrow 230), multiple distances will be measured along the ray, and the active scanning device 215 will discard the measurement value and mark it as an "unknown" or "null" depth value in the depth map, such as Figure 2 Thus, in depth map 250, the true location of edge 255 of object 220 is not at measured depth points 260 and 265, but in the space between them.

[0011] The presence of missing depth values ​​does not allow or enable the use of the depth map to accurately calculate the distance between two points in the captured 3D space. Since a 2D color image of the captured scene is assumed to be available, the user can inspect the projection of the captured 3D scene onto the 2D image plane directly or through an immersive skybox viewer. However, if the user chooses that the start and / or end point of the measurement is an image pixel that does not have a corresponding depth due to an incomplete depth map, the measurement function will not be able to return any meaningful value.

[0012] Techniques exist to fill in incomplete parts of depth maps (in other words, to make the resolution of the depth map comparable to the resolution of the corresponding 2D image of the scene). One approach involves estimating the missing depth value as the average of nearby depth values. However, this approach can lead to inaccuracies around the edges of objects and is therefore not suitable for enabling precise measurements. Another approach involves using a trained neural network to estimate the missing depth values. However, this requires a good match between the test data and the training data, as well as sufficient processing power on the device to run the neural network operations.

[0013] Therefore, there still exists a need for improved depth map generation. Summary of the Invention

[0014] The embodiments herein aim to address the above-mentioned problems.

[0015] A specific object is to provide a technique for generating a depth map which is complete in the sense that the depth map is not missing depth values ​​at pixels corresponding to edges.

[0016] According to a first aspect, these and more objects are solved by a method for generating a depth map for a 2D panoramic image. The method is performed by an image processing device. The method includes obtaining a depth map for a 2D panoramic image of a 3D environment. The depth map is generated from a 3D point cloud of the 3D environment. The depth map includes some pixels with unknown depth values. The method includes generating an edge map for the depth map. The edge map is generated from the depth map and indicates the locations of edges with unknown depth values ​​in the depth map in the 3D point cloud. The method includes calculating a corresponding depth value for each pixel in the depth map with unknown depth values. Each corresponding depth value is calculated as a weighted interpolation of known depth values ​​in an area in the depth map surrounding the corresponding unknown depth value, wherein the known depth value does not intersect any edge in the edge map.

[0017] According to a second aspect, these and more objects are solved by an image processing device for generating a depth map for a 2D panoramic image. The image processing device includes processing circuitry. The processing circuitry is configured to cause the image processing device to obtain a depth map for a 2D panoramic image of a 3D environment. The depth map is generated from a 3D point cloud of the 3D environment. The depth map includes some pixels with unknown depth values. The processing circuitry is configured to cause the image processing device to generate an edge map for the depth map. The edge map is generated from the depth map and indicates the locations of edges with unknown depth values ​​in the depth map in the 3D point cloud. The processing circuitry is configured to cause the image processing device to calculate a corresponding depth value for each pixel in the depth map with unknown depth values. Each corresponding depth value is calculated as a weighted interpolation of known depth values ​​in an area in the depth map surrounding the corresponding unknown depth value, wherein the known depth value does not intersect any edge in the edge map.

[0018] According to a third aspect, these and more objects are solved by an image processing device for generating a depth map for a 2D panoramic image. The image processing device includes an acquisition module, which is configured to obtain a depth map for a 2D panoramic image of a 3D environment. The depth map is generated from a 3D point cloud of the 3D environment. The depth map includes some pixels with unknown depth values. The image processing device includes a generation module, which is configured to generate an edge map for the depth map. The edge map is generated from the depth map and indicates the positions of edges with unknown depth values ​​in the depth map in the 3D point cloud. The image processing device includes a calculation module, which is configured to calculate a corresponding depth value for each pixel in the depth map with unknown depth values. Each corresponding depth value is calculated as a weighted interpolation of known depth values ​​in an area surrounding the corresponding unknown depth value in the depth map, wherein the known depth value does not intersect any edge in the edge map.

[0019] According to a fourth aspect, these and more objects are solved by a computer program for generating a depth map for a 2D panoramic image. The computer program includes computer code that, when run on processing circuitry of an image processing device, causes the image processing device to perform actions. One action includes: the image processing device obtains a depth map for a 2D panoramic image of a 3D environment. The depth map is generated from a 3D point cloud of the 3D environment. The depth map includes some pixels with unknown depth values. One action includes: the image processing device generates an edge map for the depth map. The edge map is generated from the depth map and indicates the locations of edges with unknown depth values ​​in the depth map in the 3D point cloud. One action includes: the image processing device calculates a corresponding depth value for each pixel in the depth map with unknown depth values. Each corresponding depth value is calculated as a weighted interpolation of known depth values ​​in an area in the depth map surrounding the corresponding unknown depth value, wherein the known depth value does not intersect any edge in the edge map.

[0020] According to a fifth aspect, these and further objects are solved by a computer program product comprising the computer program according to the fourth aspect and a computer readable storage medium on which the computer program is stored. The computer readable storage medium may be a non-transitory computer readable storage medium.

[0021] Advantageously, these aspects address the above-mentioned problems when generating depth maps.

[0022] Advantageously, these aspects enable the generation of a complete depth map in the sense that the depth map does not lack depth values ​​at pixels corresponding to edges.

[0023] Advantageously, these aspects enable depth maps with missing depth values ​​to be complete, at least with respect to filling in unknown depth values ​​at pixels corresponding to edges.

[0024] Advantageously, these aspects enable accurate measurement in a skybox image rendering environment.

[0025] Enabling accurate measurements in a skybox image rendering environment makes the skybox image rendering environment suitable for industrial use cases, such as digital twins.

[0026] These advantages are enabled by the precise location of edges, and thus accurate filling of incomplete depth maps.

[0027] Other objectives, features and advantages of the accompanying embodiments will be apparent from the following detailed disclosure, from the attached dependent claims as well as from the drawings.

[0028] In general, unless otherwise expressly defined herein, all terms used in the claims should be interpreted according to their ordinary meaning in the technical field. Unless otherwise expressly stated, all references to "a / an / the element, device, component, member, module, step, etc." should be openly interpreted as referring to at least one instance of the element, device, component, member, module, step, etc. Unless otherwise expressly stated, the steps of any method disclosed herein do not have to be performed in the exact order disclosed. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present inventive concept will now be described by way of example with reference to the accompanying drawings, in which:

[0030] Figure 1 schematically illustrates a skybox image rendering environment and a panoramic image according to an example;

[0031] Figure 2 Schematically illustrates why for an actively scanned scenario most object edges tend to be between measurement points according to an example;

[0032] Figure 3 A block diagram schematically illustrates an image processing apparatus according to an embodiment;

[0033] Figure 4 is a flow chart of a method according to an embodiment;

[0034] Figure 5 schematically illustrates an example edge map according to an embodiment;

[0035] Figure 6 schematically illustrates a search for finding the closest known depth value in a depth map according to an embodiment;

[0036] Figure 7 schematically illustrates an iteration of traveling rays between the positions of two points according to an embodiment;

[0037] Figure 8 is a schematic diagram illustrating functional units of an image processing apparatus according to an embodiment;

[0038] Figure 9 is a schematic diagram illustrating functional modules of an image processing apparatus according to an embodiment; and

[0039] Figure 10 An example of a computer program product including a computer-readable storage medium according to an embodiment is shown. DETAILED DESCRIPTION

[0040] The present invention will now be described more fully below with reference to the accompanying drawings, which illustrate certain embodiments of the present invention. However, the present invention may be embodied in many different forms and should not be construed as limited to the embodiments described herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete and fully convey the scope of the present invention to those skilled in the art. Throughout this specification, like numbers refer to like elements. Any steps or features shown by dashed lines should be considered optional.

[0041] As mentioned above, there is still a need for improved depth map generation.

[0042] Therefore, embodiments disclosed herein relate to techniques for generating a depth map for a 2D panoramic image 110. To achieve such techniques, an image processing device, a method performed by the image processing device, and a computer program product containing code (e.g., in the form of a computer program) that, when executed on the image processing device, causes the image processing device to perform the method are provided.

[0043] exist Figure 3 A block diagram of an image processing device 300 according to an embodiment is shown in FIG. 3D to 2D projection block 310 is configured to generate a (sparse) depth map 315 (and a 2D color image 330) from the 3D point cloud 305. An edge map estimation block 320 is configured to generate an edge map 325 from the (sparse) depth map 315. Optionally, the edge map estimation block 320 is further configured to generate the edge map 325 from an image edge map 340 as generated from the 2D (color) image 330 by the image edge detection block 335. A depth map completion block 345 is configured to generate a (dense) depth map 350 from the (sparse) depth map 315 and the edge map 325.

[0044] Figure 4 is a flow chart illustrating an embodiment of a method for generating a depth map for a 2D panoramic image 110. The method is performed by an image processing device 300, 800, 900. These methods are advantageously provided as a computer program 1020.

[0045] S102: The image processing device 300, 800, 900 obtains a depth map of the 2D panoramic image 110 of the 3D environment. The depth map is generated from the 3D point cloud of the 3D environment.

[0046] At least some embodiments are based on the understanding that: Figure 2 As explained, a laser beam that strikes the edge of an object will not return a measurement, which in turn will result in missing depth values. As a result, the depth map includes some pixels with unknown depth values.

[0047] Any depth value of an edge of an object in the captured scene must be between existing depth values. Therefore, incomplete regions between recorded 3D points are examined to find possible edges only at locations (or coordinates) surrounded by 3D points of different depth values ​​(i.e., depth discontinuities). In particular, the image processing device 300, 800, 900 is configured to perform action S104.

[0048] S104: The image processing device 300, 800, 900 generates an edge map 500 for the depth map. The edge map 500 is generated from the depth map and indicates positions of edges with unknown depth values ​​in the depth map in the 3D point cloud.

[0049] Thus, the method can find regions with unknown depth values ​​where the unknown depth value lies between two sufficiently different known depth values. This results in an accurate and complete edge map. Figure 5 , Figure 5 An example edge map 500 according to an embodiment is schematically illustrated. In the illustrated edge map 500, locations of edges in the 3D point cloud that have unknown depth values ​​in the depth map are shown in white.

[0050] Then, for each pixel in the depth map having an unknown depth value, a corresponding depth value may be calculated as in action S106 .

[0051] S106: The image processing device 300, 800, 900 calculates a corresponding depth value for each pixel in the depth map having an unknown depth value. Each corresponding depth value is calculated as a weighted interpolation of known depth values ​​in an area surrounding the corresponding unknown depth value in the depth map, wherein the known depth value does not intersect any edge in the edge map 500.

[0052] Thus, the method enables filling in missing parts of an incomplete depth map in a precise manner that accurately locates the edges between objects in the depth map. The filled depth map, in particular using depth values ​​around object boundaries (i.e., at edges), allows for accurate distance measurements and obtaining the true size of objects (from edge to edge).

[0053] Now, we will continue to refer to Figure 4 , disclose embodiments relating to further details of generating a depth map for a 2D panoramic image 110 as performed by the image processing apparatus 300 , 800 , 900 .

[0054] Next, further aspects of how a depth map for the 2D panoramic image 110 may be obtained will be disclosed.

[0055] As mentioned above Figure 1As disclosed, in some examples, a panoramic image is composed of a set of individual images, each of which has a depth map. In particular, in some embodiments, a 2D panoramic image 110 is rendered in a skybox image rendering environment 100, wherein the panoramic image 110 is composed of a set of individual images I n,s ={I n,-90 ,I n,0 ,I n,+90 ,I n,360 ,I n,上 ,I n,下}, each surface in the skybox image rendering environment 100 has a separate image, and each separate image has a depth map.

[0056] In general, the generation of depth maps (one for each individual image) can be based on reprojecting the 3D point cloud Ω to the image I n For each image in , there is a depth map for each image. This reprojection is done with the help of the sensor pose P n is executed and results in the image I n Each image in (i.e., with Figure 1 Each face of the cube shown in FIG. 1 is associated with a depth map D n Thus, in some embodiments, the generating in action S104 and the calculating in action S106 are performed for the depth map of each individual image.

[0057] The sensor pose P in the 3D point cloud coordinate system can be expressed by its orientation (X P ,Y P ,Z P ) and direction angle Definition. The rotation matrix R is defined as follows:

[0058]

[0059] And the translation vector n is defined as follows:

[0060]

[0061] The pose P in homogeneous coordinates can be defined as:

[0062]

[0063] From the 3D point cloud Ω, point m = [X k ,Y k ,Z k ] reprojected to the camera coordinate system corresponding to pose P is given by:

[0064] m * =P Tm.

[0065] Next, Convert to 2D image coordinates (i.e. pixel coordinates) as follows:

[0066]

[0067] Among them, according to the focal length f and the principal point [s x ,s y ], using intrinsic camera parameters.

[0068] Then, the pixel position [u * ,v * The depth value d at ] is the Euclidean distance between the sensor position n and point m. That is:

[0069]

[0070] This operation is repeated for all pixels of a given image, resulting in a depth map associated with the given image on the skybox. Since 3D points are typically sparser than 2D color images, the created depth map is sparse, i.e., it contains pixels for which the depth value is unknown and no 3D points are projected into them.

[0071] Next, further aspects of how edge maps can be generated for depth maps will be disclosed. Generally, edge maps are generated only for unknown depth values. Generally, one edge map is generated for each depth map with unknown depth values. Thus, the width and height of the edge map are equal to the width and height of the corresponding depth map and image I. n width and height.

[0072] For each depth map D n , edge graph M n Initialized with a "null" value. In some examples, a "null" value is represented by a value of 0.

[0073] Then, for the edge graph M n A search is performed (as will be described in detail below) for each "null" value in the depth map that has no known depth value at the same coordinates. A search is performed in the depth map to find the closest known depth value. Furthermore, the search is limited to being performed within K sectors, where K > 3 is an even number. Furthermore, the search is limited to being performed within K / 2 pairs of subtending sectors; that is, a number of pairs (k + ,k - ), where k=1, ..., K / 2. In particular, in some embodiments, each pixel having an unknown depth value is located at a corresponding coordinate in the depth map, and the image processing device 300, 800, 900 is configured to perform (optional) action S104a as part of generating the edge map 500 in action S104.

[0074] S104a: The image processing device 300, 800, 900 determines, for each pixel with an unknown depth value in the depth map, at least one pair of known depth values ​​closest to each pixel with an unknown depth value in the depth map. Each pair of closest known depth values ​​is determined by searching, by the image processing device 300, 800, 900, the depth map for K / 2 pairs of closest known depth values ​​within subtending sectors at the corresponding coordinates, where K>3 is an even number, and where In the opposite sector k + and k - The known depth value in .

[0075] Here is the intermediate reference Figure 6 . Figure 6 A schematic diagram of a search for finding the closest known depth value in a depth map 600 having K=6 sectors is provided. The boundary between two of the six sectors is indicated by line 610. The hollow circle 620 represents a starting position with an unknown depth value (i.e., an empty pixel in the sparse depth map). The solid circles 630a and 630b represent the starting positions for two opposite sectors k in the depth map. + and k - The closest known depth value is found in the sector. If no known depth value is found after a certain distance L, the known depth value in that sector is set to "null". Therefore, the use of distance L limits the search space. This restriction can be used to improve search time and handle situations where large areas of the depth map are missing depth values.

[0076] Once the closest known depth value for each of the K / 2 pairs of opposing sectors is determined A check is then performed to see if the “null” value in the considered edge map corresponds to an edge in the 3D point cloud and therefore to a point located at the considered K / 2 pair (k + ,k - ) A (still unknown) depth value between two sufficiently different known depth values ​​in at least one of the opposite sectors.

[0077] In some examples, if all depth values ​​are If the maximum difference between at least one pair of pixels is greater than a certain threshold, then the "null" value in the considered edge map corresponds to an edge in the 3D point cloud. That is, in some examples, a pixel with an unknown depth value in the depth map is defined as an edge if the following conditions are met:

[0078]

[0079] In other words, if for some threshold θ>0, Then two known depth values is defined as being sufficiently different.

[0080] If an edge is found, then the edge graph M n The value at the corresponding coordinate in is set to a "non-null" value. In some examples, a "non-null" value is represented by a value of 1.

[0081] When the edge graph M n After this search has been performed for all the "null" values ​​in each edge graph M n is complete.

[0082] In some examples, each edge graph M n By comparing with the corresponding image I n Therefore, in some embodiments, the edge map 500 is further generated based on the edge map of the 2D panoramic image 110 .

[0083] Image I n The edge map can be generated by an edge detector, such as a Canny edge detector, a Kovalevsky edge detector, and the like.

[0084] Next, aspects of how missing depth values ​​may be calculated will be disclosed.

[0085] As described above, the missing (ie, unknown) depth values ​​in each of the depth maps are calculated as weighted interpolations of known depth values ​​in nearby areas that do not intersect edges. In other words, this is equivalent to using the edge map M n The edge in the image is used as a diffusion guide to guide the depth map D n Therefore, in some embodiments, the image processing device 300, 800, 900 is configured to perform (optional) action S106a as part of calculating the depth value in action S106.

[0086] S106a: The image processing device 300, 800, 900 diffuses the pixels with known depth values ​​in the depth map. The edges in the edge map 500 are used as diffusion guides.

[0087] In some examples, calculating the corresponding depth value includes casting a marching ray from a pixel with a known depth value to a pixel with an unknown depth value along the iteration direction in the depth map (i.e., performing a ray marching search). Then, when calculating the corresponding depth value, only the known depth values ​​for which the marching ray does not intersect any edge with an unknown depth value in the depth map are used for weighted interpolation of the known depth value. In this regard, ray marching can be considered a class of image processing methods used in 3D computer graphics, in which rays are iteratively traversed, effectively dividing each ray into smaller ray segments, and sampling a function at each step.

[0088] In some embodiments, the image processing device 300 , 800 , 900 is configured to perform (optional) actions S106 a - 1 to S106 a - 4 as part of calculating a depth value for each corresponding pixel in the depth map having an unknown depth value in action S106 .

[0089] S106a-1: The image processing device 300, 800, 900 obtains J corresponding known depth values ​​p1, p2, ..., p for pixels in a window around the pixel with an unknown depth value. J List Λ.

[0090] That is, for the depth map D n For each point p0 at coordinates [u, v] with unknown depth value in [u, v], a list Λ (Λ = [p1, p2, ..., p k ]).

[0091] Next, for each point in Λ with a known depth value, cast a traveling ray from the point with a known depth value to p0 (or from p0 to the point with a known depth value p k Cast a traveling ray—direction doesn't matter).

[0092] S106a-2: The image processing device 300, 800, 900 calculates in the depth map each known depth value p in the list Λ. j Cast a ray-travelling ray to a pixel with an unknown depth value.

[0093] The ray marching is iteratively passing through the k A way to find all coordinates [x,y] between p0 and p0.

[0094] S106a-3: If the traveling ray intersects an edge with an unknown depth value in the depth map, the image processing device 300, 800, 900 removes the known depth value p from the list Λ j .

[0095] S106a-4: The image processing device 300, 800, 900 calculates each corresponding depth value as a weighted interpolation of all remaining known depth values ​​in the list Λ, as defined by a weight kernel ψ.

[0096] Generally speaking, when analyzing the depth map D n When interpolating and calculating missing depth values ​​in the edge map M, only those known depths are used for which the ray march will not encounter n The edge of the middle.

[0097] Next, we will refer to Figure 7 Exposes an example of the process used to calculate missing depth values. Figure 7 A schematic diagram 700 of a traveling ray iteration between the positions of two points is provided in FIG; a start point 710 and an end point 720. The open circles 740 represent all coordinates that are examined during the traveling ray iteration along the point-to-point path as given by arrow 730.

[0098] If in p k If any of the positions iterated between and p0 contains a non-zero label in the edge map M, the depth value p is removed from the list Λ of nearby known depth values. k .

[0099] Once the depth values ​​that intersect the edge have been removed from the list Λ, the missing depth p0 can be computed as a weighted interpolation of the remaining depth values ​​in Λ using the weight kernel ψ:

[0100]

[0101] In general, the weight kernel ψ maps to each value (p i ) based on the distance weighting term (ψ i ), so that larger distances are assigned smaller weights, and vice versa. In some examples, ψ i is calculated as p0 and p in the depth map i The coordinates (respectively represented by pos(p0) and pos(p i ) gives the normalized inverse of the Manhattan distance or Euclidean distance between ). In some examples, ψ i Calculated from a Gaussian kernel of width x pixels:

[0102]

[0103] Therefore, in some examples, given an edge and interpolation weight kernel, the following process can be performed for each location in the depth map with a missing depth value. First, a weight from the weight kernel is assigned to each known depth value. Second, a ray-traveling search is performed in the edge map from the location of the known depth value to the location of the missing depth value. If the ray-traveling encounters an edge in the edge map, the ray-traveling is interrupted. That is, as described above, only those known depth values ​​for which the ray-traveling does not encounter an edge in the edge map are used when interpolating and calculating the missing depth value.

[0104] Figure 8 The components of the image processing apparatus 300, 800, 900 according to the embodiment are schematically shown in terms of a number of functional units. The processing circuit 810 is provided by any combination of one or more of the following: a suitable central processing unit (CPU), a multiprocessor, a microcontroller, a digital signal processor (DSP), etc., which are capable of running a computer program product 1010 (e.g., stored in the form of a storage medium 830) Figure 10 The processing circuit 810 may also be provided as at least one application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0105] In particular, the processing circuit 810 is configured to cause the image processing device 300, 800, 900 to perform a set of operations or steps as described above. For example, the storage medium 830 may store the set of operations, and the processing circuit 810 may be configured to retrieve the set of operations from the storage medium 830 to cause the image processing device 300, 800, 900 to perform the set of operations. The set of operations may be provided as a set of executable instructions.

[0106] Thus, the processing circuitry 810 is configured to perform the methods disclosed herein. The storage medium 830 may also include persistent memory, which may be, for example, any single memory or combination of magnetic memory, optical memory, solid-state memory, or even remotely mounted memory. The image processing device 300, 800, 900 may also include a communication (comm.) interface 820, which is configured to communicate with other entities, functions, nodes, and devices. Thus, the communication interface 820 may include one or more transmitters and receivers, including analog and digital components. For example, the processing circuitry 810 controls the general operation of the image processing device 300, 800, 900 by sending data and control signals to the communication interface 820 and the storage medium 830, by receiving data and reports from the communication interface 820, and by retrieving data and instructions from the storage medium 830. To avoid obscuring the concepts presented herein, other components of the image processing device 300, 800, 900 and related functionality have been omitted.

[0107] Figure 9 Components of the image processing apparatus 300 , 800 , 900 according to the embodiments are schematically shown in terms of a plurality of functional modules. Figure 9 The image processing apparatus 300, 800, 900 includes a plurality of functional modules; an obtaining module 910 configured to execute step S102, a generating module 920 configured to execute step S104, and a calculating module 940 configured to execute step S106. Figure 9 The image processing devices 300, 800, 900 may further include multiple optional functional modules, for example, a determination module 930 configured to execute step S104a, a diffusion module 950 configured to execute step S106a, an acquisition module 960 configured to execute step S106a-1, a projection module 970 configured to execute step S106a-2, a removal module 980 configured to execute step S106a-3, and a calculation module 990 configured to execute step S106a-4.

[0108] In general, each functional module 910:990 may be implemented solely in hardware in one embodiment and in software in another embodiment, i.e., the latter embodiment may have computer program instructions stored on a storage medium 830 that, when executed on a processing circuit, cause the image processing device 300, 800, 900 to perform the combined operation. Figure 9 830. The corresponding steps described above should also be mentioned. It should also be noted that even though these modules correspond to parts of a computer program, they do not have to be independent modules therein, but rather their implementation in software depends on the programming language used. Preferably, one or more or all of the functional modules 910:990 can be implemented by the processing circuit 810, possibly in cooperation with the communication interface 820 and / or the storage medium 830. Thus, the processing circuit 810 can be configured to retrieve instructions as provided by the functional modules 910:990 from the storage medium 830 and execute these instructions, thereby performing any of the steps disclosed herein.

[0109] The image processing device 300, 800, 900 may be provided as a standalone device, or may be provided as part of at least one other device. A first portion of the instructions executed by the image processing device 300, 800, 900 may be executed in a first device, and a second portion of the instructions executed by the image processing device 300, 800, 900 may be executed in a second device; the embodiments disclosed herein are not limited to any particular number of devices on which the instructions executed by the image processing device 300, 800, 900 may be run. Thus, the methods according to the embodiments disclosed herein are suitable for execution by the image processing device 300, 800, 900 located in a cloud computing environment. Thus, although Figure 8 A single processing circuit 810 is shown in FIG, but the processing circuit 810 may be distributed across multiple devices or nodes. The same applies to Figure 9 Functional modules 910:990 and Figure 10 Computer program 1020.

[0110] Figure 10 An example of a computer program product 1010 including a computer-readable storage medium 1030 is shown. A computer program 1020 may be stored on the computer-readable storage medium 1030, which may cause the processing circuit 810 and entities and devices operatively coupled thereto (e.g., the communication interface 820 and the storage medium 830) to perform methods according to the embodiments described herein. Thus, the computer program 1020 and / or the computer program product 1010 may provide means for performing any of the steps disclosed herein.

[0111] exist Figure 10 In the example of , computer program product 1010 is shown as an optical disc, such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a Blu-ray Disc. Computer program product 1010 may also be implemented as a memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM), and more specifically, may be implemented as a non-volatile storage medium of a device in an external memory, such as a USB (Universal Serial Bus) memory or a flash memory (e.g., a compact flash). Thus, although computer program 1020 is schematically shown here as a track on the depicted optical disc, computer program 1020 may be stored in any manner suitable for computer program product 1010.

[0112] The inventive concept has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the inventive concept as defined by the appended patent claims.

Claims

1. A method for generating a depth map for a two-dimensional (2D) panoramic image (110), the method being performed by an image processing device (300, 800, 900), the method comprising: Obtaining (S102) a depth map of the 2D panoramic image (110) for a three-dimensional 3D environment, wherein the depth map is generated from a 3D point cloud of the 3D environment, and wherein the depth map includes some pixels having unknown depth values; generating (S104) an edge map (500) for the depth map, wherein the edge map (500) is generated from the depth map and indicates locations of edges with unknown depth values ​​in the depth map in the 3D point cloud; and For each of the pixels in the depth map having an unknown depth value, calculating (S106) a corresponding depth value, wherein the corresponding depth value is calculated as a weighted interpolation of known depth values ​​in a region of the depth map surrounding the corresponding unknown depth value, wherein the known depth value does not intersect any of the edges in the edge map (500).

2. The method according to claim 1, wherein The 2D panoramic image (110) will be rendered in a skybox image rendering environment (100), wherein the panoramic image (110) is composed of a set of individual images I n,s ={I n,-90 ,I n,0 ,I n,+90 ,I n,360 ,I n,上 ,I n,下 }, each surface in the skybox image rendering environment (100) has a separate image, and each separate image has a depth map.

3. The method according to claim 2, wherein: The generating and the calculating are performed for the depth map of each individual image.

4. A method according to any one of the preceding claims, wherein Each pixel having an unknown depth value is located at a corresponding coordinate in the depth map, and wherein generating the edge map (500) comprises: For each pixel in the depth map having an unknown depth value, determining (S104a) at least one pair of known depth values ​​closest to each pixel in the depth map having an unknown depth value by searching for the closest known depth values ​​within K / 2 pairs of subtending sectors at the corresponding coordinates in the depth map. Where K>3 is an even number, and where In the opposite sector k + and k - The known depth value in .

5. The method according to claim 4, wherein The pixel with unknown depth value in the depth map is defined as an edge if the following conditions are met: Here, θ>0 is the threshold value.

6. A method according to any one of the preceding claims, wherein The edge map ( 500 ) is further generated based on the edge map of the 2D panoramic image 110 .

7. A method according to any one of the preceding claims, wherein Calculating the corresponding depth value includes: The pixels with known depth values ​​in the depth map are diffused (S106a), wherein the edges in the edge map (500) are used as diffusion guides.

8. A method according to any one of the preceding claims, wherein Calculating the corresponding depth value comprises casting a traveling ray from a pixel of known depth value to the pixel of unknown depth value along an iteration direction in the depth map, and wherein, in calculating the corresponding depth value, only known depth values ​​for which the traveling ray does not intersect any edge of the depth map with an unknown depth value are used for the weighted interpolation of known depth values.

9. A method according to any one of the preceding claims, wherein For each corresponding pixel in the depth map having an unknown depth value, calculating the corresponding depth value includes: For pixels within a window around the pixel with unknown depth value, obtain (S106a-1) corresponding J known depth values ​​p1, p2, ..., p J List of Λ; In the depth map, for each known depth value p in the list Λ j The pixel of the pixel with the unknown depth value projects (S106a-2) a traveling ray toward the pixel with the unknown depth value; If the traveling ray intersects an edge in the depth map with an unknown depth value, the known depth value p is removed (S106a-3) from the list Λ. j ;as well as The corresponding depth value is calculated (S106a-4) as a weighted interpolation of all remaining known depth values ​​in the list Λ, as defined by a weight kernel ψ.

10. An image processing device (300, 800, 900) for generating a depth map for a two-dimensional (2D) panoramic image (110), the image processing device (300, 800, 900) comprising a processing circuit (810) configured to cause the image processing device (300, 800, 900) to: A depth map of the 2D panoramic image (110) for a three-dimensional 3D environment is obtained, wherein: The depth map is generated from a 3D point cloud of the 3D environment, and wherein the depth map includes some pixels having unknown depth values; generating an edge map (500) for the depth map, wherein the edge map (500) is generated from the depth map and indicates locations of edges with unknown depth values ​​in the depth map in the 3D point cloud; and For each of the pixels in the depth map having an unknown depth value, calculating a corresponding depth value, wherein the corresponding depth value is calculated as a weighted interpolation of known depth values ​​in a region of the depth map surrounding the corresponding unknown depth value, wherein the known depth values ​​do not intersect any of the edges in the edge map (500).

11. An image processing device (300, 800, 900) for generating a depth map for a two-dimensional (2D) panoramic image (110), the image processing device (300, 800, 900) comprising: an obtaining module (910) configured to obtain a depth map of the 2D panoramic image (110) for a three-dimensional 3D environment, wherein the depth map is generated from a 3D point cloud of the 3D environment, and wherein the depth map includes some pixels having unknown depth values; a generating module (920) configured to generate an edge map (500) for the depth map, wherein the edge map (500) is generated from the depth map and indicates locations of edges with unknown depth values ​​in the depth map in the 3D point cloud; and A calculation module (940) is configured to calculate, for each of the pixels in the depth map having an unknown depth value, a corresponding depth value, wherein the corresponding depth value is calculated as a weighted interpolation of known depth values ​​in an area of ​​the depth map surrounding the corresponding unknown depth value, wherein the known depth values ​​do not intersect any of the edges in the edge map (500).

12. The image processing device (300, 800, 900) according to claim 10 or 11, further configured to perform the method according to any one of claims 2 to 9.

13. A computer program (1020) for generating a depth map for a two-dimensional (2D) panoramic image (110), the computer program comprising computer code which, when run on a processing circuit (810) of an image processing device (300, 800, 900), causes the image processing device (300, 800, 900) to: Obtaining (S102) a depth map of the 2D panoramic image (110) for a three-dimensional 3D environment, wherein: The depth map is generated from a 3D point cloud of the 3D environment, and wherein the depth map includes some pixels having unknown depth values; generating (S104) an edge map (500) for the depth map, wherein the edge map (500) is generated from the depth map and indicates locations of edges with unknown depth values ​​in the depth map in the 3D point cloud; and For each of the pixels in the depth map having an unknown depth value, calculating (S106) a corresponding depth value, wherein the corresponding depth value is calculated as a weighted interpolation of known depth values ​​in a region of the depth map surrounding the corresponding unknown depth value, wherein the known depth values ​​do not intersect any of the edges in the edge map (500).

14. A computer program product (1010) comprising a computer program (1020) according to claim 13 and a computer-readable storage medium (1030) on which the computer program is stored.

Images