Generating complete depth data for 6-DOF video
By analyzing depth components and selectively using a subset of cameras based on completeness analysis, the method addresses excessive computational demands in 6-DoF VR video capture, enhancing calculation efficiency and accuracy.
Patent Information
- Application Number
- JP2024514369
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-10-18
- Filing Date
- 2022-09-07
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2042-09-07
AI Technical Summary
The computational resources required for generating depth data in 6-DoF VR video capture systems are excessive due to the large number of cameras needed, especially in dynamic scenes like a football game, where one workstation may not be sufficient to calculate all depth data.
A method for generating depth data in 6-DoF video involves analyzing a first set of depth components to determine completeness or overcompleteness, selecting a second set of images and depth components based on this analysis, and using a subset of cameras to reduce the number of images and depth components, ensuring accurate depth estimation by ensuring all scene information is captured by at least two cameras.
This approach reduces computational power requirements by dynamically selecting necessary images and cameras, improving calculation efficiency while maintaining accurate depth data generation for 6-DoF video.
Smart Images

Figure 0007582554000001 
Figure 0007582554000002 
Figure 0007582554000003
Abstract
Description
[Technical field]
[0001] The present invention relates to the field of generating depth data for a scene, in particular for a 6DoF video of a scene. [Background technology]
[0002] Virtual reality (VR) video is an important application of VR technology that can bring an immersive experience of video. In general, VR video technology can be divided into 3-DoF (degrees of freedom) and 6-DoF. 3-DoF technology only allows users to freely rotate their head to see different orientations from a fixed position in the video scene. However, 6-DoF technology allows users to choose different orientations and also allows users to freely choose their position in the scene.
[0003] 3-DoF video capture requires a minimal number of cameras pointing in different directions at the scene, whereas 6-DoF video capture requires a relatively large camera array.
[0004] In practice, for 6-DoF VR videography, there are many cameras facing the scene, and every two adjacent cameras create a camera pair that is used to generate a depth image. The distance between the pair of cameras is called the baseline. The depth image and the images captured by the cameras can be used to render an image of the virtual view. Summary of the Invention [Problem to be solved by the invention]
[0005] In a VR video shooting system, all real-time images are sent to a workstation, which runs a depth estimation algorithm to generate depth images. Therefore, a camera array with a large number of cameras requires a large amount of computational resources. For example, for a football game, more than 50 cameras may be required, and therefore one workstation may not be enough to calculate all of the depth data for a football game. Clearly, there is a need to improve the computational efficiency for the generation of 6DoF videos.
[0006] US Patent Application Publication No. 2017 / 094259(A1) discloses techniques relating to 3D image capture using dynamic cameras. [Means for solving the problem]
[0007] The invention is defined by the claims.
[0008] According to an example according to an aspect of the present invention, there is provided a method for generating depth data for a six degree of freedom (6DoF) video of a scene, the method comprising: obtaining a first set of images of the scene; generating a first set of depth components based on the first set of images; analyzing the first set of depth components to determine completeness of the depth components; acquiring a second set of images of the scene; generating a second set of depth components based on the second set of images; If the analyzing step determines that the first set of depth components is overcomplete, the number of depth components in the second set is selected to be less than the number of depth components in the first set.
[0009] Acquiring 6DoF video of a dynamic scene would be difficult because the depth of objects in a dynamic scene may change at any time. For this reason, a large set of cameras must be used so that as an object approaches the cameras, for example, depth data can still be acquired for a close object so that it is still within the field of view of at least two cameras.
[0010] However, using a large number of cameras also means acquiring (and potentially transmitting) a large amount of data. It is likely that most of the time during a 6DoF video, not all cameras in the large set are needed to acquire depth data for the 6DoF video. This is especially true when objects are not close to the cameras.
[0011] Therefore, the inventors propose to analyze a first set of depth components in a frame of a 6DoF video, and based on the analysis determining that the first set of depth components is complete, generate a second set of depth components having a smaller number of depth components than the first set. The method of analysis is described below.
[0012] The completeness of the depth component refers to how much of the scene is captured in the depth component. If all of the depth information of a scene is captured in the depth component, then there is enough scene depth information to allow for accurate warping of the image, so a 6DoF frame can be rendered accurately.
[0013] Determining that the first set of depth components is complete may be defined by the first set of depth components having sufficient scene depth information to render a 6 DoF frame without missing scene depth information in the frame. Similarly, determining that the first set of depth components is overcomplete may be defined by the first set of components having more than sufficient scene depth information. In other words, an overcomplete first set of depth components may have redundant or overlapping depth information.
[0014] The assessment of completeness may be performed by assessing continuity of the depth component, identifying gaps or occlusions in the depth component, identifying missing depth information, identifying artifacts, and the like.
[0015] The first set of images and the first set of depth components may be used to render a first frame of the 6DoF video. The second set of images and the second set of depth components may be used to render a second frame of the 6DoF video, the second frame being after the first frame.
[0016] The images can be acquired from a group of cameras. The group of cameras can be selected by selecting a subset of cameras from all available cameras imaging the scene. The group of cameras can comprise one or more pairs of cameras, thereby enabling the generation of depth components from images acquired by both cameras in a camera pair. A camera pair can be formed by selecting a camera and the nearest available camera in the group of camera pairs.
[0017] The depth component may consist of a depth map (e.g., generated from a pair of images) or any other type of depth information for the scene (e.g., a 3D mesh, a point cloud, etc.) In other words, the depth component may be any form of depth information for a scene that includes multiple depth values.
[0018] For example, in a depth map, each depth value may correspond to a pixel in the depth map, in a 3D mesh, each depth value may correspond to a vertex, edge, and / or face of the 3D mesh, in a point cloud, each depth value may correspond to a point in the point cloud.
[0019] It is not necessary that a camera pair consists of two adjacent cameras. In fact, a camera pair of two cameras far from each other may be preferable, since the greater the baseline (i.e., the distance between the cameras) of the two cameras, the higher the accuracy of the depth image. Thus, for a fixed number of selected cameras, the accumulation (i.e., sum) of the baselines of the selected cameras may be maximized when satisfying the requirement that each pixel is visible by at least two cameras.
[0020] For example, cameras selected from left to right may be numbered as 1, 2, 3, 4. If the camera pairs are 1 to 3, 2 to 4, and 2 to 3, the baseline accumulation may be greater than the accumulation of camera pairs 1 to 2, 2 to 3, and 3 to 4, and therefore the accuracy of the depth component may be greater.
[0021] Analyzing the first set of depth components may include determining an indication of how close one or more of the objects in the scene are to a first group of cameras used to acquire the first set of images.
[0022] We propose to determine a proximity metric (or closeness metric) for objects in a scene, which indicates how close one or more objects are to the camera currently imaging the scene (i.e., the first group). If an object is closer to the camera than the current first camera group can obtain depth data from, then a second group of camera pairs is selected to obtain the next frame of the 6DoF video, and the second camera group can obtain depth data for the object closest to the camera.
[0023] This metric is an estimate of how close the object is physically to the first camera group. This metric can be, for example, a single value of depth or a single non-numeric metric (i.e., close, far, etc.). Alternatively, this metric can include information about the depth of the closest object and where the closest object is in the scene.
[0024] The index may also include information regarding the depth of one or more objects, and in some embodiments, information regarding the position of one or more objects.
[0025] The depth component may comprise a depth map generated by performing depth estimation on at least two images.
[0026] The depth component may have depth maps / depth images, where each depth map is generated from images acquired from a camera pair. The depth map may be obtained by performing depth estimation (e.g., via depth disparity) on two texture images acquired from the camera pair (i.e., an image from each camera in the camera pair).
[0027] In one embodiment, the proximity metric may be based on the pixel in the depth map that has the lowest depth value.
[0028] In the context of the present invention, the terms depth map and depth image are equivalent and interchangeable.
[0029] The method may further include determining whether a camera-facing surface of any one of the objects in the scene is not visible within the field of view of at least two cameras in the first camera group, and generating the second group of depth components is further based on this determination.
[0030] A camera-facing surface of an object can be a surface of the object that is imaged by at least one camera in a camera group (i.e., is visible within the field of view (FOV) of at least one camera). Ensuring that all camera-facing surfaces of an object are imaged by at least two cameras (i.e., are within the FOV of at least two cameras) allows the depth component to be generated more accurately.
[0031] For example, if one or more camera-facing surfaces are determined to be not visible in the depth component, a second camera group may be selected to include all of the cameras available to image the scene. Alternatively, one or more cameras may be identified that can image the camera-facing surfaces that are not currently visible in the FOV of at least two of the cameras.
[0032] Analyzing the first set of depth components may further include determining whether any objects in the scene are at least partially occluded in the first set of depth components.
[0033] Analyzing the first set of depth components to determine object occlusion may include identifying where the occlusion is and which camera may be used to acquire subsequent images for the next (second) frame to avoid the occlusion in subsequent depth components that are generated.
[0034] For example, if an object occlusion is determined in the depth component, the second group of cameras may be selected to include all of the available cameras. Alternatively, one or more cameras for correcting the occlusion may be identified, and the identified cameras are included in the second group.
[0035] Analyzing the first set of depth components may further include determining whether the first set of depth components has any visual and / or depth artifacts.
[0036] Visual and depth artifacts may result from the absence of a camera imaging a particular portion of a scene, an object in a scene, or a portion of an object. Artifacts may also be caused by a faulty camera (i.e., a camera that does not image the scene as intended, for example, due to a fogged lens or a malfunctioning camera sensor), and therefore a different or additional camera may be required in the second group.
[0037] Analyzing the depth components to determine whether they have visual and / or depth artifacts may further include identifying where the artifacts are in the depth components and identifying which camera can be used to acquire a subsequent second set of images of the next (second) frame in order to correct the artifacts in the subsequent depth components.
[0038] Depth estimation typically uses a feature matching algorithm between two camera images. The feature matching algorithm outputs a large matching error if it fails to match a pixel in the first image with a pixel in the second image. Therefore, artifacts can be identified using the matching error output by the feature matching algorithm during depth estimation. In addition, areas of the depth map that have artifacts can be identified by checking which pixels in the two images have large matching errors.
[0039] The method may further include selecting a group of cameras configured to acquire a second set of images based on an analysis of the first set of depth components.
[0040] We propose to analyze the first set of depth components and, based on this analysis, select which camera to use to acquire the second set of images, which may have a smaller number of images than the first set. The second set of images is used to generate the second set of depth components.
[0041] The group of cameras may be selected to reduce the number of images in the second set of images and therefore also reduce the number of depth components in the second set of depth components.
[0042] Selecting the group of cameras may include selecting a camera pair such that all objects in the scene visible from at least one camera are included within the field of view of at least two cameras in the group of cameras. The cameras may be selected such that depth can be estimated for all (visible) parts of all objects.
[0043] Selecting the group of cameras may include selecting a predefined group of cameras from a set of predefined groups of cameras, each of the predefined groups of cameras associated with a minimum distance from the predefined group of cameras at which an object in the scene is guaranteed to be within the field of view of at least two cameras in the predefined group of cameras.
[0044] From knowledge of the inter-camera distances and camera fields of view in a predefined group of cameras, it is possible to derive the closest / minimum distance at which an object can be located to the predefined group before the predefined group is unable to properly obtain depth information of the object. In other words, a minimum distance can be derived for each predefined group of cameras based on the inter-camera distances and camera fields of view in the predefined group.
[0045] Thus, each predefined group of cameras corresponds to a minimum distance (from the predefined group) at which an object is guaranteed to be within the field of view of two cameras (within the predefined group) such that depth information can be obtained for the object.
[0046] If the analysis determines that the first set of depth components is incomplete, the second set of depth components may have a larger number of depth components than the first set of depth components.
[0047] Determining that the first set of depth components is incomplete may be defined by the first set of depth components not having sufficient scene depth information to render the frame with 6 DoF, in other words, the incomplete first set of depth components may have missing scene depth information in the frame.
[0048] Determining that the first set of depth components is overcomplete can be based on a closeness metric indicating that a nearest object in the scene is farther away from the first group of cameras than a minimum distance that ensures that an object in the scene is within the field of view of at least two cameras in the second group of cameras, where the second group of cameras includes fewer cameras than the first group of cameras.
[0049] Determining that the first set of depth components is incomplete can be based on a closeness metric indicating that the nearest object in the scene is closer to the first camera group than a minimum distance that ensures that an object in the scene is within the field of view of at least two cameras in the first group of cameras.
[0050] The present invention also provides a computer program product comprising computer program code which, when executed on a computing system having a processing system, causes the processing system to perform all of the steps of a method for generating depth data for a six degree of freedom (6DoF) video of a scene.
[0051] The present invention also provides a system for generating depth data for a six degree of freedom (6DoF) video of a scene, the system comprising: Obtain a first set of images of the scene; generating a first set of depth components based on the first set of images; Analyzing the first set of depth components to determine completeness of the depth components; Obtain a second set of images of the scene; a processor configured to generate a second set of depth components based on a second set of images; If the analysis determines that the first set of depth components is overcomplete, then the number of depth components in the second set is selected to be less than the number of depth components in the first set.
[0052] The processor may be configured to analyze the first set of depth components by determining an indication of how close one or more of the objects in the scene are to the first camera group.
[0053] The processor may be further configured to determine whether a camera-facing surface of any one of the objects in the scene is not visible within the field of view of at least two cameras in the first group of cameras, and the processor may be further configured to generate a second group of depth components based on the determination.
[0054] The processor may be further configured to select a group of cameras configured to acquire a second set of images based on an analysis of the first set of depth components.
[0055] The processor can be configured to select the group of cameras by selecting a predefined group of cameras from a set of predefined groups of cameras, each of the predefined groups of cameras being associated with a minimum distance from the predefined group of cameras at which an object in the scene is guaranteed to be within the field of view of at least two cameras in the predefined group of cameras.
[0056] The system may further comprise a camera group comprising a plurality of cameras.
[0057] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief description of the drawings]
[0058] For a better understanding of the present invention and to show more clearly how it may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Figure 1] FIG. 1 shows an array of cameras. [Diagram 2]FIG. 1 shows a camera array having six camera pairs. [Diagram 3] FIG. 1 shows a camera array having four camera pairs. [Figure 4] FIG. 1 shows a camera array having three camera pairs. [Figure 5A] FIG. 1 shows an array of cameras imaging two objects with two camera pairs. [Figure 5B] FIG. 1 shows an array of cameras imaging two objects with three camera pairs. [Figure 6] 2 shows a flow chart according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0059] The present invention will now be described with reference to the drawings.
[0060] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the devices, systems and methods, are for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the devices, systems and methods of the present invention will become better understood from the following description, the appended claims, and the accompanying drawings. It should be understood that the drawings are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to denote the same or similar parts.
[0061] The present invention provides a method for generating depth data for a six degree of freedom (6DoF) video of a scene. The method includes acquiring a first set of images of the scene, generating a first set of depth components based on the first set of images, and analyzing the first set of depth components to determine completeness of the depth components. A second set of images of the scene is further acquired and a second set of depth components is generated based on the second set of images, and if the analysis determines that the first set of depth components is overcomplete, the number of depth components in the second set is selected to be less than the number of depth components in the first set.
[0062] 1 shows an array of cameras 102. Each camera 102 has a corresponding field of view 106. The field of view 106 is shown only for camera pair 104 consisting of cameras (102) 3 and 4. The shaded region 108 indicates the area of the scene from which depth data can be obtained based on images acquired by camera pair 104.
[0063] In general, the number of camera pairs 104 used to acquire depth data of a scene not only depends on the size of the space in which the video is shot, but also on how close all objects in the scene are to the camera pairs 104. If the objects are very close to the camera pairs 104, some cameras 102 may have to be moved closer to other cameras 102 to properly estimate the depth data. In other words, the baseline of the camera pairs 104 should be short when the objects are close. However, if the objects are far from the camera pairs 104, the baseline may be longer. All other conditions being equal, a longer baseline would be preferable because it provides more accuracy in depth estimation when using feature matching / disparity.
[0064] In dynamic scenes (e.g., a football game), the background is usually very far from the array of cameras 102. However, players will be moving, sometimes closer to the array of cameras 102 and sometimes farther away from them. For this reason, it is not necessary to have all the cameras 102 close to each other at all times. However, during VR filming, all the cameras 102 should preferably be fixed, and considering the worst case, enough cameras 102 should be used, and they should be close enough to each other to obtain depth data of the objects that are close.
[0065] Once the cameras 102 are set up, the number of cameras 102 and their relative positions are likely to be fixed. However, it is recognized that in a dynamic scene, it is not always necessary to use all images acquired by the cameras 102 to estimate the depth data. Therefore, the algorithm can select a subset of the acquired images, and this subset can be used to estimate the depth data according to the detected scene. This can save computational power and improve computational efficiency.
[0066] In summary, the claimed invention proposes a method to dynamically select the required images from the array of cameras 102 to estimate the depth data of the scene, thereby reducing the required computational power.
[0067] The array of cameras 102 can be part of a 6-DoF VR video capture system that further includes a workstation and cables for transmitting all images from the cameras 102 to the workstation. The workstation can include a processor configured to generate depth data for each camera pair 104 for a first frame. The workstation can then select the required images from a particular camera pair 104 that are required to further estimate the depth data of a subsequent frame. The selection of the required images is based on analyzing the depth data of the previous frame.
[0068] Analyzing the depth data of the previous frame (e.g., via an algorithm) may include detecting and removing redundant camera pairs 104 according to the previous estimated depth data. Optionally, more camera pairs 104 may be added if the analysis detects that the currently selected camera pairs 104 are not sufficient to estimate the depth data. The depth data may be a depth image or a depth map.
[0069] The selection of the camera pair 104 can be based on previous depth data. The selection of the camera pair 104 can further be based on detection of occlusion of objects in the scene. In addition, the selection of the camera pair 104 can be based on checking for failure of depth estimation in a particular area of the scene. When estimating depth data based on images from a particular camera pair 104, if depth and / or occlusion in a particular region cannot be estimated, more (or different) camera pairs 104 that can obtain depth data for that particular region may be required. Methods for analyzing depth data are further detailed below.
[0070] The selection of the camera pair 104 defines which images (i.e., from which camera pair 104) are used for depth estimation. Thus, the analysis of the depth estimation may be used to select which images are used for depth estimation for a second frame based on the depth data of a first frame. In this case, all images may be acquired from all of the cameras 102 for the second frame, but only a certain subset of images are selected based on the selected camera pair. In general, the depth data of a frame is used to select which images are used for depth estimation in a subsequent frame.
[0071] 2 shows an array of cameras 102 with six camera pairs 104 (a-f). To estimate the depth of an object, the object must be present within the field of view 106 of at least two cameras 102 of a camera pair 104. The depth of a pixel can be estimated by disparity calculation between the two images captured by the camera pair 104.
[0072] The shaded region 108 defines the region where depth can be accurately estimated based on the six camera pairs 104a-f. The dashed line 202 defines how close an object can be to the six camera pairs 104a-f before the estimated depth data is no longer complete. By using six camera pairs 104a-f, an object can be relatively close to the camera array while ensuring that each part of the object can be observed in at least one camera pair 104.
[0073] 3 shows an array of cameras 102 with four camera pairs 104a, 104c, 104e and 104f. Similar to FIGS. 1 and 2, the shaded region 108 defines the region of the scene where depth can be accurately estimated based on a given camera pair 104a, 104c, 104e and 104f. A line 302 defines how close an object can be to the four camera pairs 104a, 104c, 104e and 104f before the estimated depth data is no longer complete.
[0074] By using only four camera pairs 104a, 104c, 104e and 104f, objects cannot be as close to the camera array as when six camera pairs 104a-f are used (see FIG. 2), but depth data only needs to be estimated for four camera pairs 104a, 104c, 104e and 104f instead of six camera pairs 104a-f.
[0075] Figure 4 shows an array of cameras 102 with three camera pairs 104g, 104h and 104i. Similar to Figures 1, 2 and 3, the shaded region 108 defines the region of the scene where depth can be accurately estimated based on a given camera pair 104g, 104h and 104i. Line 402 defines how close an object can be to the three camera pairs 104g, 104h and 104i before the estimated depth data is no longer complete.
[0076] By using only three camera pairs 104g, 104h and 104i, objects cannot get as close to the camera array as when four or six camera pairs 104 are used (see Figures 2 and 3), but depth data only needs to be estimated for the three camera pairs 104g, 104h and 104i instead of the four or six camera pairs 104.
[0077] Therefore, based on Figures 2, 3 and 4, there is a clear relationship between the number of camera pairs 104 for which depth data is acquired and how close an object can be to the array of cameras 102. For example, it is clear that the computational resources required to estimate depth data are reduced when an object moves farther away from the array of cameras 102. If depth data in a dynamic scene is always estimated for the six camera pairs 104a-f shown in Figure 2, this is likely a waste of computational resources, since it is unlikely that there is always an object close to the array of cameras 102.
[0078] In practice, the distance of an object to the array of cameras 102 is known (e.g., based on depth data from a previous frame), and therefore a way is needed to find which camera pair 104 to select (i.e., which image to use for depth estimation in the next frame).
[0079] At the start of a 6-DoF video, the system may have no idea about the scene it observes. Therefore, it may be advantageous to use multiple camera pairs 104 to estimate depth data for the first frame (e.g., using every two adjacent cameras for the first frame, as shown in FIG. 2). The depth data for the first frame can then be used to select a minimum number of camera pairs 104 that ensures that each part of the scene is seen by at least one camera pair 104.
[0080] In addition, the larger the baseline (i.e., the distance between the two cameras) of the camera pair 104, the more accurate the depth data. Thus, the accumulation (i.e., the sum) of all the baselines of all the camera pairs 104 used may be maximized when selecting the camera pair 104. Any kind of optimization method (or brute force method) may be used to select the cameras to satisfy the two conditions (i.e., each part of the scene imaged by at least one camera pair 104 and the sum of the maximized baselines). After the camera pair 104 is selected, the selected camera pair 104 may be used to estimate the depth data of the next frame. This method may be used successively for each subsequent frame. This method is applicable when some objects are moving away from the camera 102.
[0081] Once a camera pair 104 is selected and used for depth data estimation, more camera pairs 104 may need to be selected for the next frame if some objects are moving towards some cameras. Optionally, the depth data of all selected camera pairs 104 may be stitched together. Stitching depth data from multiple camera pairs 104 involves all of the depth data being combined (i.e., stitched) into a single component (e.g., a depth image). Stitching algorithms are known, for example, for stitching depth images and / or depth maps. In some cases, it may not be possible to stitch all the depth data together, for example, as one depth image by a stitching algorithm.
[0082] If an error is detected during stitching of the depth data, one or more camera pairs corresponding to the depth gap may be added. Alternatively, all camera pairs 104 (e.g., FIG. 2) may be used to estimate the depth data of the second frame, and then the estimated depth data of the second frame may be used to reselect the camera pair 104 for the third frame. The selection of the camera pair 104 may be based on the depth data acquired by the camera pair 104 and / or stitched depth components generated from the depth data.
[0083] 5A and 5B show an array of cameras 102 imaging two objects 502 and 504 by two and three camera pairs 104, respectively. The array of cameras 102 in FIGS. 5A and 5B includes only four cameras 102 for simplicity. In FIG. 5A, the object 502 partially occludes the object 504 to the selected camera pairs 104a and 104b. Region 506 shows the region of the scene occluded by the object 502. Thus, a part of the object 504 is occluded by the object 502 to both camera pairs 104a and 104b. If the object 502 were not in the scene, all depth data of the object 504 could be estimated by both camera pairs 104a and 104b. However, due to the occlusion caused by the object 502, depth data can no longer be estimated for all of the object 504 using these camera pairs.
[0084] 5B, camera pair 104c is also selected. Object 502 does not occlude object 504 to camera pair 104c, and therefore depth data can be acquired by camera pair 104c. Therefore, object occlusion may also need to be considered when selecting camera pair 104.
[0085] FIG. 6 shows a flow chart according to an embodiment of the present invention. At the beginning of a 6-DoF video (i.e., for the first frame of the video), all camera pairs can be used (step 602). In step 604, images from all camera pairs are received and used to estimate depth data (e.g., depth image / depth map) of the scene in step 606. As shown in FIG. 2, images from every two adjacent cameras can be used to generate depth data. In step 608, the depth data of each selected camera pair can be stitched together by a stitching algorithm.
[0086] In step 610, the stitched depth data can be analyzed to determine whether there are any "errors". Alternatively, the depth data can be analyzed (610) (before stitching 608). If the analysis 610 identifies one or more errors, it means that the selected camera pairs are not sufficient to obtain a complete set of depth data for the scene. In other words, the depth data can be incomplete. Errors include an object not being imaged by at least two cameras in the camera array, an object being occluded, or the depth data is not complete for a camera pair (e.g., the camera lenses are dirty / damaged). If an error occurs, in step 612, all camera pairs can be reselected and used for the next frame.
[0087] If no errors are detected in the stitched depth data, this may mean that the depth data acquired for the scene in the current frame is overcomplete (i.e., there is more depth data than is needed for the scene / some of the depth data is redundant or duplicated). In this case, an optimization method 614 can be used to select a camera pair that satisfies three conditions: the number of selected cameras is minimal, the part of the real-world scene corresponding to each pixel in the depth data is physically observed by at least two cameras (occlusions are taken into account), and the baseline accumulation of the selected camera pair is maximal. The optimization method 614 is used to select a camera pair for the next frame (step 616). For each frame, the output may be a stitched depth image.
[0088] Obviously, the above camera pair selection is used to acquire image pairs for depth estimation. However, the camera pair selection may involve selecting which cameras acquire images of the scene or selecting which camera's images are used for depth estimation. More than two cameras can be selected for depth estimation (e.g., three cameras can be selected for more accurate depth data instead of a camera pair).
[0089] Those skilled in the art can easily develop a processor to perform any of the methods described herein. Thus, each step of the flowchart may represent a respective operation performed by a processor and may be executed by a respective module of the processor.
[0090] As described above, the system utilizes a processor to process data. The processor may be implemented in a variety of ways using software and / or hardware to perform the various functions required. The processor typically uses one or more microprocessors that are programmed using software (e.g., microcode) to perform the functions required. The processor may also be implemented as a combination of dedicated hardware to perform some functions and one or more programmed microprocessors and associated circuitry to perform other functions.
[0091] Examples of circuitry that may be used in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).
[0092] In various implementations, the processor may be associated with one or more storage media, e.g., volatile and non-volatile computer memories such as RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that, when executed on one or more processors and / or controllers, perform the required functions. The various storage media may be attached within a processor or controller, or may be transportable such that one or more programs stored on the storage media are readable by a processor.
[0093] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.
[0094] A single processor or other unit may fulfill the functions of several items recited in the claims.
[0095] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.
[0096] The computer program may be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.
[0097] When the term "adapted for" appears in the claims or the specification, it is meant to be synonymous with the term "configured to."
[0098] Any reference signs in the claims should not be construed as limiting the scope.
Claims
1. 1. A method for generating depth data for a six degree of freedom (6DoF) video of a scene, comprising: acquiring a first set of images of the scene; generating a first set of depth components based on the first set of images, each depth component having a plurality of depth values; analyzing the first set of depth components to determine completeness of the depth components; acquiring a second set of images of the scene; generating a second set of depth components based on the second set of images; having If the analysis determines that the first set of depth components is overcomplete, a number of depth components in the second set is selected that is less than the number of depth components in the first set, such that the overcomplete first set of depth components has more depth information of the scene than is sufficient to render a 6 DoF frame; If the analysis determines that the first set of depth components is incomplete, then the number of depth components in the second set is selected to be greater than the number of depth components in the first set, and some of the depth information of the scene is missing from the incomplete first set of depth components.
2. 2. The method of claim 1 , wherein the first set of images is acquired by a first group of cameras, and analyzing the first set of depth components includes determining an indication of how close one or more objects in the scene are to the first set of cameras.
3. The method of claim 1 or 2, wherein the depth component comprises a depth map generated by performing depth estimation based on at least two images.
4. 4. The method of claim 1, further comprising determining whether a camera-facing face of any object in the scene is visible within the field of view of at least two cameras corresponding to the first set of images, and wherein generating a second set of depth components is further based on said determining.
5. 5. The method of claim 1, wherein analysing the first set of depth components comprises determining whether any objects in the scene are at least partially occluded in the first set of depth components.
6. The method of claim 1 , wherein analysing the first set of depth components comprises determining whether the first set of depth components includes visibility and / or depth artifacts.
7. 7. The method of claim 1, wherein the first set of images is acquired by a first group of cameras, the method further comprising the step of selecting a second group of cameras configured to acquire the second set of images based on an analysis of the first set of depth components.
8. 8. The method of claim 7, wherein selecting a group of cameras comprises selecting a predetermined camera group from a set of predetermined camera groups, each predetermined camera group being associated with a minimum distance from the predetermined camera group that ensures that an object in the scene is within the field of view of at least two cameras in the predetermined camera group.
9. A computer program which, when executed by a computer, causes the computer to carry out the method according to any one of claims 1 to 8.
10. 1. A system for generating depth data for a six degree of freedom (6DoF) video of a scene, the system comprising: acquiring a first set of images of the scene; generating a first set of depth components based on the first set of images, each depth component having a plurality of depth values; analyzing the first set of depth components to determine completeness of the depth components; acquiring a second set of images of the scene; generating a second set of depth components based on the second set of images; a processor configured to: If the analysis determines that the first set of depth components is overcomplete, a number of depth components in the second set is selected that is less than the number of depth components in the first set, such that the overcomplete first set of depth components has more depth information of the scene than is sufficient to render a 6 DoF frame; If the analysis determines that the first set of depth components is incomplete, a number of depth components in the second set is selected to be greater than the number of depth components in the first set, and some depth information of the scene is missing from the incomplete first set of depth components.
11. 11. The system of claim 10, wherein the first set of images is acquired by a first group of cameras, and the processor is configured to analyze the first set of depth components by determining an indication of how close one or more objects in the scene are to the first set of cameras.
12. 12. The system of claim 10 or 11, wherein the processor is further configured to determine whether a camera-facing face of any object in the scene is visible within the field of view of at least two cameras corresponding to the first set of images, and the processor is further configured to generate a second set of depth components based on the determination.
13. 13. The system of claim 10, wherein the first set of images is acquired by a first group of cameras, and the processor is further configured to select a group of cameras configured to acquire the second set of images based on an analysis of the first set of depth components.
14. 14. The system of claim 13, wherein the processor is configured to select a group of cameras by selecting a predetermined camera group from a set of predetermined camera groups, each predetermined camera group being associated with a minimum distance from the predetermined camera group that ensures that an object in the scene is within the field of view of at least two cameras in the predetermined camera group.
Citation Information
Patent Citations
Image processing method, image processing system and computer program
JP2012194751A
Apparatus and method for generating a representation of a scene
JP2020529685A
Method and system of 3D image capture with dynamic cameras
US20170094259A1
Video display method and video display device
US20180343442A1
Configuration settings of a digital camera for depth map generation
US9307222B1
