Shape-dependent novel view compositing

Abstract

A method is proposed for rendering a target view image of a scene from a target viewpoint. This method involves obtaining at least two source view images of the scene taken from different locations and at least one depth map of the scene. The surface shape in the scene is estimated using the source view images and / or depth map, and the surface shape indicates the gradient of the surfaces in the scene relative to the location where the source view images were captured. Based on the surface shape and the locations where the first and second source view images were captured, a first portion of the scene to be rendered using the first source view image is determined, and a second portion of the scene to be rendered using the second source view image is determined. The target view image of the scene is rendered from the target viewpoint using at least one depth map, the first source view image for the first portion of the scene, and the second source view image for the second portion of the scene.

Description

Technical Field

[0001] The present invention relates to rendering an image in a multi-view video from a target viewpoint. In other words, the present invention relates to a novel view synthesis using a multi-camera capture system.

Background Art

[0002] Synthesis of a novel view from a plurality of source view cameras is an area of active research and development. Novel view synthesis is particularly difficult when the number of cameras is limited or when the goal is to create a large display space where virtual cameras can be placed at various positions within the scene. One solution to the above problems is to add a depth sensor to form a hybrid camera rig composed of both a color sensor and a depth sensor. Using a depth sensor generally enables a wider display space.

[0003] Existing methods for novel view synthesis are very sensitive to calibration errors and depth errors. These errors usually cause visible artifacts. Furthermore, view blending techniques usually reduce sharpness, while neural network-based techniques require appropriate calibration to learn an excellent prediction model.

[0004] It has been found that artifacts are a serious problem when using a combination of a small number of cameras and a large display zone. However, adding cameras is often not an option. Furthermore, calibration errors are difficult to solve without using a specific calibration pattern, and the depth obtained via a time-of-flight depth sensor is biased and may generate noise.

Summary of the Invention

Problems to be Solved by the Invention

[0005] All these issues often result in new views being composited / rendered with noticeable artifacts, impacting the viewing experience. Therefore, an improved method is needed to composite new views while minimizing visual artifacts. [Means for solving the problem]

[0006] The present invention is defined by the claims.

[0007] According to an embodiment of one aspect of the present invention, a method is provided for rendering a target view image of a scene at a target viewpoint, and this method is: The steps include obtaining at least two source view images of the scene captured from different locations and at least one depth map of the scene, A step of estimating the surface shape in a scene using a source view image and / or depth map, wherein the surface shape indicates the gradient of the surface in the scene relative to the location where the source view image was captured. The steps include determining the first part of the scene to be rendered using the first source view image and the second part of the scene to be rendered using the second source view image, based on the surface shape and the positions where the first and second source view images were captured, The method comprises the steps of rendering a target view image of the scene at a target viewpoint using at least one depth map, a first source view image for the first part of the scene, and a second source view image for the second part of the scene.

[0008] This method provides a shape-dependent view assignment of textures instead of blending textures from different views. This reduces artifacts caused by calibration and depth errors, and improves the sharpness of the newly rendered view. This is especially important when capturing source view images using a small number of cameras.

[0009] As a result, new views are rendered at the target viewpoint with accurate texture values, eliminating the need for highly precise camera calibration. Therefore, new views with accurate texture values ​​can be generated with a small number of cameras. This is because the gradient of the surface in the scene relative to the camera is taken into account when assigning a source view to each part.

[0010] By determining the first and second parts using the surface shape, a shape-dependent view assignment is provided. In other words, determining the first and second parts can be considered as assigning weights to the source view that depend on an index of the gradient (i.e., surface shape) of the surfaces in the scene (compared to conventional view blending). Preferably, at least a portion of the first part should have a weight of 100% for the first image, and at least a portion of the second part should have a weight of 100% for the second image.

[0011] Surface shape estimation involves assuming a surface shape based on the type of object in the scene (for example, a human head can be assumed to be convex).

[0012] Determining a portion means assigning a source view camera to each portion based on the surface shape relative to the position of each source view camera. Then, the corresponding portion is rendered using the source view image from the assigned source view camera.

[0013] This method may further include using the surface shape to determine which source-view camera is most orthogonal to one or more surfaces of the portion. In this case, the determination of the source-view image to be used for a portion is based on the source-view camera that is most orthogonal to one or more surfaces of that portion.

[0014] Images can be acquired from a multi-camera capture system consisting of two or more source-view cameras that capture the scene.

[0015] In this embodiment, two source view images and two source view depth maps are used. Therefore, the target view can be rendered by warping the first source view image using the first depth map and warping the second view image using the second depth map. Then, a view assignment map is used to select the warped view, which is then selected at the target image coordinates.

[0016] In another example, the source view image can be captured from a single camera that moves from one position to another, assuming the scene is static (i.e., not moving).

[0017] Surface shape estimation involves identifying one or more types of objects and / or object features within a scene, and assigning a predetermined surface shape to one or more objects and / or features based on the identified types of objects and / or object features.

[0018] Certain types of objects are known to have common shapes. For example, the surface of a human head can generally be defined as convex. Therefore, by identifying the type of object in a scene, a given surface shape can be assigned to that object.

[0019] Similarly, it is known that the types of features of an object (e.g., nose, eyebrows, wrinkles, etc.) also have common shapes.

[0020] This simplifies the process of estimating surface shape compared to, for example, directly calculating the surface gradient by analyzing a depth map.

[0021] The predetermined surface shape can be convex, concave, or flat.

[0022] Generally, by defining the surface shape as convex, concave or flat, a simple format is provided for determining which source view should be used to render which part of the surface.

[0023] For example, in the case of a concave shape, the left side of the shape faces the right side of the multi-camera capture system, and vice versa. Therefore, the rightmost camera can best convey the texture of the left side of the surface, and the leftmost camera can best convey the texture of the right side of the surface.

[0024] Identifying one or more types of objects in a scene can include applying an object detection algorithm to an image and / or a depth map.

[0025] Identifying one or more types of objects in a scene can include applying a keypoint detection algorithm to an image.

[0026] Keypoint detection is usually performed for a specific object category (such as a human). Furthermore, keypoint detection can be the basis for assumptions about richer surface shape information. For example, using a human keypoint detection algorithm, specific body parts (such as the upper left arm) can be identified.

[0027] Estimating the surface shape can include calculating the normal vector of the surface in the depth map and determining the angle between the vector between the position where the source view image was captured and the surface and the normal vector.

[0028] Determining the first part and the second part can include comparing the angles determined for different source view images and assigning source view images to each part based on the comparison of the angles.

[0029] In some cases, instead of assuming the surface is convex / concave / planar, for example, a pixel-level indicator of the gradient may be desirable to more accurately represent the shape. Calculating the normal vector yields a more accurate surface shape, thereby more precisely dividing the surface into first and second parts.

[0030] Preferably, determining the first and second parts involves assigning a camera to each pixel based on the camera having the smallest angle between the normal vector at the pixel's position and the vector between the camera and the pixel.

[0031] In some examples, determining the first and second parts involves using the sum of line-weighted image coordinates to determine the stitch lines that separate the first and second parts.

[0032] This method simplifies the determination of the first and second parts because it eliminates the need to determine the surface normal vectors. This method can be particularly advantageous in video call applications that want to composite new views in real time.

[0033] The sum of the weighted image coordinates for each line can be weighted according to the normalized depth from the target viewpoint.

[0034] This method may further include applying a low-pass filter to the depth map and using the filtered depth map to estimate the surface shape.

[0035] Applying a low-pass filter reduces noise in the depth map, decreasing the number of undesirable view assignments caused by noise.

[0036] In some cases, it may be desirable to apply a different low-pass filter to pixels belonging to the same depth surface to avoid averaging over abrupt depth transitions (for example, between different surfaces of different depths). This is a nonlinear filtering operation.

[0037] Applying a low-pass filter may involve applying the low-pass filter individually to two or more surfaces within the depth map in order to generate a filtered depth map.

[0038] This method can also include identifying surfaces in the depth map.

[0039] This method may further include identifying one or more objects and / or object features in the scene, and adapting the first and second parts if an object and / or object feature is identified between the first and second parts.

[0040] The area or line between the first and second parts can be called the stitching area / line. When this area / line touches a specific object or feature of an object, it can cause unwanted artifacts because there are slight differences in the texture changes captured by the camera (for example, light reflections may differ between cameras due to different positions). Therefore, artifacts in the target view can be reduced by adjusting the parts and moving the stitching line / area.

[0041] Object features are the characteristics of an object (such as a nose or eyebrows). These features can influence the general shape of an object (such as a head). Of course, it will be understood that the distinction between objects and object features can be arbitrarily chosen. For example, if the expected scene is of a person, the head, arms, and body can be treated as objects, while the nose and eyebrows can be treated as object features. However, if the expected scene includes a crowd, not only the nose and eyebrows, but also the head, arms, and body can be treated as object features. The precise distinction between objects and object features can be based on the scene being captured and personal preference.

[0042] This method may further include calculating the color difference between the corresponding colors in the first and second source views within the stitched region (the stitched region being the portion of the target view image that overlaps both the first and second parts), and adjusting the colors within the stitched region based on the color difference so as to gradually change the colors from the first and second parts.

[0043] The stitched area may overlap less than 20%, 15%, 10%, or 5% of the first part, and less than 20%, 15%, 10%, or 5% of the second part. The stitched area may not overlap all of the first part or all of the second part.

[0044] The present invention also provides a computer program carrier that, when executed on a computer, causes the computer to perform all steps according to any of the methods described above.

[0045] A computer program carrier can include computer memory (e.g., random access memory) and computer storage (e.g., hard drives, solid-state drives, etc.). A computer program carrier can also be a bitstream that transmits computer program code.

[0046] The present invention also provides a system for rendering a target view image of a scene from a target viewpoint, and the system is Obtain at least two source view images of the scene captured from different locations, and at least one depth map of the scene. The surface shape in the scene is estimated using the source view image and / or depth map (the surface shape indicates the gradient of the surface in the scene relative to the location where the source view image was captured), Based on the surface shape and the positions where the first and second source view images were captured, the first part of the scene to be rendered using the first source view image and the second part of the scene to be rendered using the second source view image are determined. The system includes at least one depth map and a processor configured to render a target view image of a scene at a target viewpoint, using a first source view image for a first part of the scene and a second source view image for a second part of the scene.

[0047] The processor is configured to identify one or more types of objects and / or object features in the scene and to estimate surface shapes by assigning predetermined surface shapes to one or more objects and / or features based on the types of the identified objects and / or object features.

[0048] The predetermined surface shape can be convex, concave, or flat.

[0049] The processor can be configured to identify one or more types of objects in a scene by applying an object detection algorithm to the image and / or depth map.

[0050] The processor can be configured to identify one or more types of objects in a scene by applying a keypoint detection algorithm to the image.

[0051] The processor can be configured to estimate the surface shape by calculating the normal vectors of the surface in the depth map and determining the angle between the normal vectors and the vector between the surface and the location where the source view image was captured.

[0052] The processor can be configured to determine the first and second parts by comparing the angles determined for different source view images and assigning the source view images to each part based on the angle comparison.

[0053] The processor can be further configured to apply a low-pass filter to the depth map and use the filtered depth map to estimate the surface shape.

[0054] The processor can be configured to apply a low-pass filter by individually applying a low-pass filter to two or more surfaces within the depth map in order to generate a filtered depth map.

[0055] The processor may be further configured to identify one or more objects and / or object features in the scene, and to adapt the first and second parts if the objects and / or object features are identified between the first and second parts.

[0056] The processor can be further configured to calculate the color difference between the corresponding colors in the first and second source views within the stitched area (the stitched area being the portion of the target view image that overlaps both the first and second parts), and to adjust the colors within the stitched area based on the color difference so as to gradually change the colors from the first and second parts.

[0057] These and other aspects of the present invention will become apparent from and be explained with reference to the embodiments described below. [Brief explanation of the drawing]

[0058] To better understand the present invention and to more clearly illustrate how it can be put into practice, refer to the accompanying drawings as merely examples. [Figure 1] A diagram showing two source-view cameras imaging a sloping surface. [Figure 2] A diagram showing two cameras imaging a second inclined surface. [Figure 3] A diagram illustrating view assignment to a convex surface using two cameras. [Figure 4] A diagram illustrating view assignment to a concave surface using two cameras. [Figure 5] A diagram illustrating view assignment in a video call scenario involving one person. [Figure 6] A diagram illustrating the adjustment of stitch lines by identifying a human nose in a new view. [Figure 7] A diagram showing the view assignments of three cameras arranged in a triangular configuration. [Figure 8] A diagram showing a target view image with two sections and a stitching region. [Figure 9] A diagram illustrating how to render a target view image of a scene from a target viewpoint using a multi-camera capture system. [Modes for carrying out the invention]

[0059] The present invention will be described with reference to the drawings.

[0060] The detailed descriptions and specific examples illustrate exemplary embodiments of the apparatus, systems, and methods, but should be understood to be for illustrative purposes only and not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, systems, and methods of the invention will be better understood from the following description, the appended claims, and the appended drawings. The drawings are for illustrative purposes only and are not drawn to a specific scale. Also, the same reference numerals are used throughout the drawings to indicate the same or similar parts.

[0061] The present invention provides a method for rendering a target view image of a scene at a target viewpoint. This method includes obtaining at least two source view images of the scene taken from different locations and at least one depth map of the scene. Surface geometry in the scene is estimated using the source view images and / or depth map, and the surface geometry indicates the gradient of the surfaces in the scene relative to the location where the source view images were captured. Based on the surface geometry and the locations where the first and second source view images were captured, a first portion of the scene to be rendered using the first source view image is determined, and a second portion of the scene to be rendered using the second source view image is determined. The target view image of the scene is rendered at the target viewpoint using at least one depth map and the first source view image for the first portion of the scene and the second source view image for the second portion of the scene.

[0062] When compositing from a single source-view camera using a depth map, the quality of the resulting view is notably higher compared to view blending methods. However, resolving occlusion typically requires a second or third camera equipped with a depth sensor. Since blending is not a suitable option, the question becomes which source view contribution to select for each pixel or region of the target view. Surface shape and orientation have been found to be good characteristics to base the selection on. Generally, to improve image quality, for a particular region of the target view, it is necessary to select a source-view camera that views that region as orthogonal as possible to the 3D surface.

[0063] Therefore, it has been proposed to let the surface shape determine which source view contribution is selected for each pixel in the target view. This shape-dependent view assignment borrows from the conventional image stitching concept, but differs in that the image is warped to the target view and assigned using the surface shape criterion. The effect of this approach is that pixels are optimally selected from the source view to maintain resolution, and the associated stitch artifacts are far less noticeable than blend artifacts.

[0064] Figure 1 shows two source-view cameras 102 and 104 that image the inclined surface 101. In this example, the entire inclined surface 101 is oriented towards source-view camera 104 rather than source-view camera 102. Therefore, the combined view (for example, at the target viewpoint between cameras 102 and 104) is rendered with the image captured from source-view camera 104.

[0065] The orientation / gradient of the inclined surface 101 can be determined by comparing the angle between the normal vector from the inclined surface 101 and the vectors from cameras 102 and 104 to the inclined surface 101.

[0066] For example, at point 106 on the inclined surface 101, the angle θ1 between the normal vector 108 from point 106 and the vector 110 between camera 102 and point 106 is greater than the angle θ2 between the normal vector 108 from point 106 and the vector 112 between camera 104 and point 106. Therefore, point 106 on the inclined surface is facing camera 104 rather than camera 102. The vector between the camera and a point on the surface can be called the source-view direction of a particular pixel / region.

[0067] Similarly, at point 114 on the inclined surface 101, the angle θ1 between the normal vector 116 from point 114 and the vector 120 between camera 102 and point 114 is greater than the angle θ2 between the normal vector 116 from point 114 and the vector 118 between camera 104 and point 114. Therefore, point 114 on the inclined surface is facing camera 104 rather than camera 102. Thus, it can be assumed that all points on surface 101 can be assigned to the image from camera 102.

[0068] For a surface assumed to be planar, it is understood that a single normal vector can be used because the normal vector can be translated across the entire surface.

[0069] Generally, angles θ1 and θ2 indicate the slope of the surface relative to the camera. In this specification, the index of the surface slope relative to the camera is referred to as the surface shape.

[0070] Figure 2 shows two cameras 202 and 204 imaging the second inclined surface 201. Figure 2 shows a situation similar to Figure 1, except that the view assignment on the surface 201 has been changed. The gradient of the surface 201 relative to cameras 202 and 204 is determined using the θ1 angle between the source view direction from camera 202 and the normal vector from the surface 201 (e.g., normal vectors 206, 208, and 210), and can further be determined using the θ2 angle between the source view direction from camera 204 and the normal vector from the surface 201.

[0071] For the first portion 212 of the surface 201 where θ1 < θ2, pixels are selected from camera 202. However, for the second portion 216 where θ1 > θ2, pixels are selected from camera 204. There may also be a third portion 214 called the stitched portion, where the angle difference is smaller than a threshold. In the stitched portion 214, pixels can be obtained from either camera 202 or camera 204, or new colors, transparencys, or depth values ​​can be created by combining pixel color, transparency, and depth values ​​from both cameras 202 and 204. This can be achieved using filtering operations.

[0072] Figures 1 and 2 show planes. However, many surfaces in a scene are not planes, or cannot be assumed to be planes. In such cases, a pixel-level surface shape analysis can be performed using a depth map to determine the pixel-level normal vectors. However, this analysis may be susceptible to noise in the depth map.

[0073] Therefore, for certain object categories, such as human subjects, the surface shape of the object can be assumed to be convex or concave to avoid potentially noisy surface shape analysis. For example, in the case of a human subject, most body parts can be assumed to give a convex depth profile. Examples of such objects are the head, neck, shoulders, and arms. However, a counterexample would be a hand open facing a depth camera, where the depth surface is concave.

[0074] Figure 3 shows the assignment of views to a convex surface 301 using two cameras 302 and 304. In this case, the normal vector 308 from point 306 is equal in angle to the source view direction from cameras 302 and 304. Thus, the left side 310 of the surface 301 to the left of point 306 can be assigned to camera 302, and the right side 312 to the right of point 306 can be assigned to camera 304.

[0075] In some applications, it is not necessary to calculate the normal vector 308 because point 306 can be assumed to be at the geometric center of surface 301. For example, in a video call, it can be assumed that the head of the human subject facing the camera is convex, and that the nose faces the camera precisely between cameras 302 and 304. In this case, the center of the face can be assumed to be the stitch line (i.e., the line between the first and second parts). Of course, determining normal vectors with equal angles can provide a more robust image novel view composite, assuming there is little noise in the depth map.

[0076] Figure 4 shows the view assignment of the concave surface 401 using two cameras 402 and 404. The normal vector 408 from point 406 can be seen to have an equal angle between the normal vector 408 and the source view direction vectors from cameras 402 and 404. Thus, the right side 412 of the surface 401 to the right of point 406 is assigned to camera 402, and the left side 410 to the left of point 406 is assigned to camera 404.

[0077] Concave and convex surfaces can be assumed, for example, by using object detection and classification algorithms on an image and / or depth map to identify one or more types of objects. Certain types of objects (such as heads, shoulders, and arms) can be assumed to be either convex, concave, or planar. Therefore, convex, concave, or planar surfaces can be fitted to the surface of a depth map.

[0078] Another approach is to use a keypoint detection algorithm on the image to identify the type of object. Thus, the shape of the object of that type can be inferred. Keypoint detection can be used to identify keypoints on objects within an image. For example, a human keypoint detection algorithm can be used to detect humans in an image. Keypoints (e.g., left knee, right elbow, left eye) can be used to adapt convex or concave surfaces to specific human features (e.g., a convex surface that fits the head).

[0079] Figure 5 shows a view assignment 506 in a video call scenario involving one person. In this case, person 506 is captured using two cameras, and two source view images 502 and 504 are acquired from the cameras. Then, a new view 508 is synthesized from the two source view images 502 and 504 at the target viewpoint. In this case, it can be assumed that the head, neck, and body of person 506 are all convex.

[0080] To construct the view assignment, the pixels of source views 502 and 504 are first warped to the target view using the source view depth map. At the target view image coordinates (u, v), the stitch line 510 can be determined by calculating the sum of the line-weighted image coordinates. The stitch line is the line that separates the first region 512, where the texture values ​​of source view 502 are used, from the second region 514, where the texture values ​​of source view 504 are used.

[0081] The position of stitch line 510 can be determined by calculating the following: TIFF2026515778000002.tif20118

[0082] Here, u stitch(v) is the position of the stitch line in the target view image coordinates, w(u, v) is a weighting coefficient that depends on the normalized depth z(u, v) observed from the target viewpoint, and w(u, v) ≡ (1 - z(u, v)). The normalized depth z(u, v) can be obtained by warping one or both of the source view depth maps to the target viewpoint. The depth map can be obtained, for example, by determining the disparity map between images 502 and 504 and converting it to a depth map. The depth map can also be obtained using a depth sensor.

[0083] Directly calculating the position of the stitch line 510 means that it is not necessary to calculate the normal vectors of the surfaces involved. Instead, if all of the region 512 to the left of the stitch line 510 is assigned to source view 502, and all of the region 514 to the right of the stitch line 510 is assigned to source view 504, then it can be assumed that all surfaces are convex.

[0084] Assuming that all surfaces are concave, it is understood that region 512 uses the texture from source view 504 and region 514 uses the texture from source view 502.

[0085] In some cases, it may be preferable to reposition the stitch line 510 based on the object features within the image. For example, it has been found that if the stitch line does not pass through the center of the nose, variations in depth around the nose can cause artifacts.

[0086] Figure 6 shows the adjustment of the stitch line 510 by identifying the nose 602 of person 506 in a new view 508. Figure 6a shows the stitch line 510 positioned next to the nose 602. Such placement of the stitch line 510 has been found to potentially make artifacts more prominent in the stitch line 510 near the nose 602 (for example, due to occlusion in one of the source views, or due to the fact that the surface on both sides of the nose is actually concave rather than convex locally). In this case, the stitch line 510 can be adjusted to cross the center of the nose 602, as shown in Figure 6b. This can be achieved, for example, by detecting the position of the nose 602 in the new view 508 (e.g., via object detection, keypoint detection, etc.) and fitting the position of the stitch line to cross the center of the nose 602. It may be more efficient and simpler to implement by detecting the location of noise in one or both source views, warping this location to the target view (using one or more depth maps), and then modifying the stitch line in the target view.

[0087] Therefore, knowing that the stitch line 510 is likely to cross the center of the face region of person 506, a positional shift of the stitch line 510 can be introduced such that, for example, the stitch line 510 aligns with a more favorable position than crossing the cheeks of person 506.

[0088] Other objects may cause more visible artifacts when the stitch line 510 crosses the center of the object (for example, due to large depth variations at the center). Thus, in other cases, it may be preferable to position the stitch line 510 away from the center of the object.

[0089] More generally, a shift in stitch line 510 is an adaptation of sections 512 and 514. In this example, section 512 is extended closer to the face to include half of the nose 602, while section 514 is reduced to include only half of the nose 602.

[0090] Objects can be identified in the source view, the warped source view (i.e., the target viewpoint), the depth map, and / or the warped depth map.

[0091] The concepts described here can also be extended to more cameras. Surface geometry analysis, which shows the surface gradient, can be generalized to multiple source-view cameras.

[0092] Figure 7 shows the view assignments for three cameras 702, 704, and 706 arranged in a triangular configuration. Figure 7a shows the view assignments for a convex surface 701a imaged by cameras 702, 704, and 706. In this case, region 708a is assigned to camera 702, region 710a is assigned to camera 704, and region 712a is assigned to camera 706.

[0093] Figure 7b shows the view assignments for the concave surface 701b as captured by cameras 702, 704, and 706. In this case, region 708b is assigned to camera 702, region 710b is assigned to camera 704, and region 712b is assigned to camera 706.

[0094] The concepts described herein enable the creation of algorithms suitable for real-time implementation of view assignment as part of a renderer running on a consumer device's graphics computing unit (GPU). The algorithm, for example, warps each source view depth map to a target viewpoint, combines the depth maps in the target view, selects the closest depth value for each pixel in the target view, filters the combined depth map with a low-pass filter, calculates the normal vector for each pixel in the target view, calculates the cosine of the angle between the normal vector and the source view direction, and assigns the source view corresponding to the smallest angle for each pixel in the target view.

[0095] The filtering step is included to avoid situations where the view assignment (and therefore the position of the portion) becomes sensitive to depth noise. For example, a Gaussian low-pass filter with a large sigma (e.g., >30 pixels) can be used. The normal vector per pixel is the 3D position x of the center pixel. i It can be calculated by constructing two 3D world vectors, with a starting point and ending at the 3D positions of different adjacent pixels. To avoid the case where the two vectors are parallel, they can be selected at a 90-degree angle from a regular image grid. These two vectors are denoted as a1 and b2. Assuming that the depth surface can be locally approximated by a plane, the normal vector n can be approximated using the cross product as follows: TIFF2026515778000003.tif22122

[0096] For each source view i, the angle θ between the surface normal vector n and the source view direction. i The cosine of is cos(θ) i ) = n·d i It can be calculated as follows.

[0097] Here, d i is the normalized source view direction vector for source view i, which is calculated as follows: TIFF2026515778000004.tif21129

[0098] Here, t i t is the camera position of the source view in world space. i This may be determined by calibration of external parameters (for example, using the structure of a motion algorithm), or it may be known (for example, in the case of a camera integrated into a single device). For a particular pixel, the view assignment can be defined as follows: TIFF2026515778000005.tif25165

[0099] In all the examples above, only one source view is used for each portion of the target view. However, depending on the camera setup, it is possible to blend source views for each portion. For example, a camera setup could include two sets of cameras: a first set for capturing the right side of an object and a second set for capturing the left side of the object. If the cameras in the sets are close to each other, we can assume that the texture values ​​between the cameras are similar (for example, the reflections between the cameras in the set will not differ significantly). In this case, we can use a blend of the first set of cameras to render the right side of the target view, and similarly, use a blend of the second set of cameras to render the left side of the target view.

[0100] Figure 8 shows a target view image 802 having two parts 804 and 806 and a stitching region 810. The left part 804 of the target view image 802 is rendered using a first source view image warped to the target viewpoint, and the right part 806 is rendered using a second source view image warped to the target viewpoint. The line separating the left part 804 and the right part 806 is the stitching line 808. The stitching region 810 is defined around the stitching line 810 and overlaps with parts of the first part 804 and parts of the second part 806.

[0101] If the source view images used for sections 804 and 806 are captured by different cameras and / or from different positions, the different observation angles of the source view cameras relative to the dominant light source may cause visible color transitions in the stitch line 808. Therefore, it is proposed to modify the color intensity of the warped source view in the stitch region 810 as a function of the observed color related to the position (or more generally, the view assignment map) of the stitch line 808.

[0102] Blending contributions from both the first and second warped source views in the stitched region 810 is one solution. However, this may result in blurring of the stitched region 810. Therefore, it is further proposed to calculate the average color difference in the sub-region 812 of the stitched region 810 across multiple adjacent pixels and gradually adjust the color of one or more source views as they approach the stitch line 808. The effect is that the original source view color of the stitched region 810 gradually changes in the left portion 804 and the right portion 806 to account for the difference between colors observed in the corresponding source view images.

[0103] For example, in subregion 812, the corresponding pixels of the warped first source view can be compared with the corresponding pixels of the warped second source view. The average color difference between these pixels can be calculated and used to apply the color of the pixel at the center of subregion 812. This can be done for all pixels in stitch region 810 so that the colors within stitch region 810 match along the stitch lines.

[0104] The color of stitch region 810, which is close to either the left region 804 or the right region 806, requires minimal adjustment to match adjacent pixels that are not present in stitch region 810. Similarly, the color of the center of stitch region 810 will likely be the most adapted.

[0105] Color adjustments can be split between source views. In other words, half of the color difference measured at stitch line 808 can be added (or subtracted) to the first source view image, and half can be subtracted (or added) to the second source view image. The effect of these adjustments is that the color difference between the source view images becomes less noticeable at stitch line 808.

[0106] The adjustment can be linearly increased for each source view image, starting from zero correction at the boundary of the stitching region 810 and increasing towards the stitching line 808. This can be implemented by calculating the shortest distance to each pixel up to the stitching line 808 and dividing it by half the width of the stitching region 810.

[0107] Color adaptation can be applied by using the same correction factor for each color channel (red, green, blue) or by applying it in a converted color space (such as YUV).

[0108] Figure 9 shows how to render a target view image of a scene at a target viewpoint using a multi-camera capture system. At least two source view images of the scene are acquired from different cameras of the multi-camera capture system, and at least one depth map of the scene is also acquired in step 902. The surface shape in the scene is estimated in step 904 using the source view images and / or depth map, where the surface shape represents the gradient of the surfaces in the scene relative to the camera. The surface shape is estimated by identifying one or more types of objects and / or object features in the scene and assigning a predetermined surface shape to one or more objects and / or object features based on the type of identified objects and / or object features. In step 906, the first part of the scene to be rendered using the first source view image and the second part of the scene to be rendered using the second source view image are determined based on the surface shape and the camera positions corresponding to the first and second source view images. Therefore, in step 908, the target view image of the scene can be rendered from the target viewpoint using the depth map, the first source view image for the first part of the scene, and the second source view image for the second part of the scene.

[0109] Those skilled in the art can easily develop a processor to perform any of the methods described herein. Thus, each step in the flowchart may represent a different action performed by the processor, which may be performed by each module of the processing processor.

[0110] One or more steps of any method described herein may be performed by one or more processors. A processor consists of electronic circuits suitable for processing data. Any method described herein may be computer-implemented, where computer implementation means that the steps of the method are performed by one or more computers, where a computer is defined as a device suitable for data processing. A computer is suitable for processing data according to given instructions.

[0111] As described above, the system utilizes a processor to perform data processing. The processor is implemented in various ways using software and / or hardware to perform the various functions required. The processor typically uses one or more microprocessors programmed to perform the required functions using software (e.g., microcode). The processor may also be implemented as a combination of dedicated hardware for performing some functions and one or more programmed microprocessors and associated circuits for performing other functions.

[0112] Examples of circuits used in various embodiments of this disclosure include, but are not limited to, conventional microprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0113] In various implementations, the processor may be associated with one or more storage media, which are volatile and non-volatile computer memories such as RAM, PROM, EPROM, and EEPROM. These storage media may be encoded with one or more programs that perform the required functions when executed on one or more processors and / or controllers. The various storage media may be mounted within the processor or controller, or they may be transportable so that one or more programs stored in the storage media can be loaded into the processor.

[0114] Modifications of the disclosed embodiments can be understood and implemented by those skilled in the art in carrying out the claimed invention, based on a review of the drawings, disclosures, and appended claims. In the claims, the words “comprising” do not exclude other components or steps, and the indefinite articles “a” or “an” do not exclude plurality.

[0115] A single processor or other unit can perform the functions of several of the items listed in the claims.

[0116] The mere fact that certain means are described in mutually different dependent claims does not indicate that combinations of these means cannot be used advantageously.

[0117] Computer programs can be stored / distributed on suitable media such as optical or solid-state media supplied together with or as part of other hardware, but they can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0118] When the term "adapt" is used in a claim or specification, it means that the term "adapt" is equivalent to the term "constituted".

[0119] No reference numeral in a claim should be construed as limiting the scope.

[0120] Any method described herein excludes methods of performing mental acts.

Claims

1. A method for rendering a target view image of a scene from a target viewpoint, The steps include obtaining at least two source view images of the scene captured from different locations, and at least one depth map of the scene. A step of estimating the surface shape in the scene using the source view image and / or the depth map, wherein the surface shape represents the slope of the surface in the scene with respect to the location where the source view image was captured, and the estimation of the surface shape includes identifying one or more types of objects and / or object features in the scene, and assigning a predetermined surface shape to the one or more types of objects and / or object features based on the type of the identified objects and / or object features, Based on the surface shape and the positions where the first source view image and the second source view image were captured, The first part of the scene rendered using the first source view image, and The second part of the scene rendered using the second source view image, The steps to determine, The steps include rendering a target view image of the scene at the target viewpoint using the at least one depth map, the first source view image for the first part of the scene, and the second source view image for the second part of the scene, A method of having.

2. The method according to claim 1, wherein the predetermined surface shape is a convex surface, a concave surface, or a flat surface.

3. The method according to claim 1 or 2, wherein identifying one or more types of objects in the scene includes applying an object detection algorithm to the image and / or the depth map.

4. The method according to claim 1 or 2, wherein identifying one or more objects of a certain type in the scene includes applying a keypoint detection algorithm to the image.

5. The estimation of the surface shape includes calculating the normal vector of the surface in the depth map and determining the angle between the normal vector and the vector between the position where the source view image was captured and the surface. The method according to claim 1 or 2, wherein determining the first and second parts includes comparing the angles determined for different source view images and assigning the source view images to each part based on the comparison of the angles.

6. The method according to claim 1 or 2, wherein the determination of the first and second parts includes determining a stitch line separating the first and second parts using the sum of line-weighted image coordinates.

7. The method according to claim 6, wherein the sum of the weighted image coordinates for each line is weighted according to the normalized depth from the target viewpoint.

8. The method according to claim or claim 2, comprising applying a low-pass filter to the depth map and estimating the surface shape using the filtered depth map.

9. The method according to claim 8, wherein the application of the low-pass filter includes applying the low-pass filter separately to two or more surfaces in the depth map in order to generate a filtered depth map.

10. The method according to claim 1 or 2, comprising identifying one or more objects and / or object features in the scene, and adapting the first and second parts if objects and / or object features are identified between the first and second parts.

11. A step of calculating the color difference between corresponding colors in the first source view and the second source view in a stitched region, wherein the stitched region is the portion of the target view image that overlaps both the first and second portions. A step of adjusting the color of the stitch area based on the color difference so as to gradually change the color from the first part and the second part, The method according to claim 1 or 2, further comprising:

12. A computer program that, when executed on a computer, causes the computer to perform the method according to claim 1 or 2.

13. A system for rendering a target view image of a scene from a target viewpoint, The steps include obtaining at least two source view images of the scene captured from different locations, and at least one depth map of the scene. A step of estimating the surface shape in the scene using the source view image and / or the depth map, wherein the surface shape represents the slope of the surface in the scene with respect to the location where the source view image was captured, and the estimation of the surface shape includes identifying one or more types of objects and / or object features in the scene, and assigning a predetermined surface shape to the one or more types of objects and / or object features based on the type of the identified objects and / or object features, Based on the surface shape and the positions where the first source view image and the second source view image were captured, The first part of the scene rendered using the first source view image, and The second part of the scene rendered using the second source view image, The steps to determine, The steps include rendering a target view image of the scene at the target viewpoint using the at least one depth map, the first source view image for the first part of the scene, and the second source view image for the second part of the scene, A system having a processor that performs [a certain action].

14. The processor is configured to estimate the surface shape by calculating the normal vector of the surface in the depth map and determining the angle between the normal vector and the vector between the position where the source view image was captured and the surface. The system according to claim 13, wherein the processor is configured to determine the first and second parts by comparing the angles determined for different source view images and assigning the source view images to each part based on the comparison of the angles.

15. The aforementioned processor, The color difference between the corresponding colors in the first source view and the second source view is calculated in the stitch region, which is the portion of the target view image that overlaps with both the first and second portions. The color of the stitch area is adjusted based on the color difference so that the color from the first part and the second part is gradually changed. The system according to claim 13 or 14, configured as follows.

Images