MÉTODO E SISTEMA PARA RENDERIZAR UMA IMAGEM DE VISTA ALVO DE UMA CENA EM UM PONTO DE VISTA ALVO
By employing surface shape-dependent view assignment, the method addresses calibration and depth errors in innovative view synthesis, producing high-quality rendered views with reduced artifacts using a small number of cameras.
Patent Information
- Application Number
- BR112025024267
- Authority / Receiving Office
- BR · BR
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-11
- Filing Date
- 2024-05-07
- Publication Date
- 2026-11-10
AI Technical Summary
Existing methods for innovative view synthesis are sensitive to calibration errors and depth errors, leading to visible artifacts, especially when using a small number of cameras to create a large viewing space, and are difficult to resolve without specific calibration standards.
A method for rendering a target view image using surface shape-dependent view assignment, where surface gradients are used to determine portions of the scene to be rendered from different source view images, reducing artifacts and increasing sharpness by assigning weights based on surface shapes and orientations relative to camera positions.
This approach generates an innovative rendered view with accurate texture values, reducing the need for precise camera calibration and minimizing artifacts, particularly effective when using a small number of cameras.
Abstract
Description
1 / 25 “METHOD AND SYSTEM FOR RENDERING A TARGET VIEW IMAGE OF A SCENE AT A TARGET VIEWPOINT” FIELD OF THE INVENTION
[001] The invention relates to rendering an image in a multiview video from a target viewpoint. In other words, the invention relates to an innovative view synthesis using a multi-camera capture system. BACKGROUND OF THE INVENTION
[002] Innovative view synthesis from multiple source view cameras is an active area of research and development. Innovative view synthesis is especially difficult when the number of cameras is limited or when the goal is to create a large viewing space where the virtual camera can be positioned in a wide variety of positions in the scene. One solution to the above problem is to add depth sensors and form a hybrid camera rig consisting of both color and depth sensors. The use of depth sensors allows for a larger viewing space overall.
[003] Existing methods for innovative view synthesis are very sensitive to calibration errors and depth errors. These errors typically result in visible artifacts. Additionally, view fusion methods typically reduce sharpness, while neural network-based methods need good calibration to learn a good predictive model.
[004] Artifacts have proven to be a significant issue when using a small number of cameras in combination with a large viewing area. However, adding more cameras is generally not an option. Furthermore, calibration errors are difficult to resolve without the use of specific calibration standards, and depth obtained through time-of-flight depth sensors can be biased and noisy.
[005] All these issues typically lead to an innovative view being synthesized / rendered with perceptible artifacts, which affects the experience of Petition 870250101956, dated 07 / 11 / 2025, pp. 76 / 112 2 / 25 visualization. As such, there is a need for an improved method to synthesize innovative visualizations with reduced visual artifacts. SUMMARY OF THE INVENTION
[006] The invention is defined by the claims.
[007] According to examples conforming to an aspect of the invention, a method is provided for rendering a target view image of a scene at a target viewpoint, wherein the method comprises: Obtain at least two source view images of the captured scene from different positions and at least one depth map of the scene; Estimate surface shapes within the scene using source view images and / or the depth map, where surface shapes are indicative of surface gradients in the scene relative to the positions where the source view images were captured; Determine, based on the surface shapes and the positions from which a first source view image and a second source view image were captured: A first portion of the scene to be rendered using the first source view image; and a second portion of the scene to be rendered using the second source view image; and render a target view image of the scene at the target viewpoint using at least one depth map and the first source view image for the first portion of the scene and the second source view image for the second portion of the scene.
[008] The method provides a view assignment dependent on the texture format instead of merging the texture from multiple views. This reduces artifacts caused by calibration errors and depth errors, as well as increasing sharpness in the view. Petition 870250101956, dated 07 / 11 / 2025, pp. 77 / 112 3 / 25 innovative rendering. This is especially important when a small number of cameras are used to capture source view images.
[009] The result is an innovative rendered view at a target viewpoint with accurate texture values that do not require very precise camera calibration. Thus, an innovative view with accurate texture values can be generated with a small number of cameras. This is because surface gradients in the scene, relative to the cameras, are taken into account when assigning a source view to each portion.
[010] Determining the first and second portions using surface shapes provides shape-dependent view assignment. In other words, determining the first and second portions can be seen as assigning weights to the source view (compared to conventional view merging) that are dependent on an indication of the gradient of surfaces in the scene (i.e., the surface shapes). Preferably, at least part of the first portion should have a weight of 100% for the first image and at least part of the second portion should have a weight of 100% for the second image.
[011] Estimating surface shapes can include assuming surface shapes based on the type of objects in the scene (for example, it is possible to assume that a person's head is convex).
[012] Determining the portions means assigning a source view camera to each portion based on the surface shapes relative to the positions of each source view camera. The source view images from the assigned source view cameras are then used to render the corresponding portions.
[013] The method may further comprise determining which source view camera is most orthogonal to one or more surfaces in said portion using the surface shapes. In this case, the determination of which source view image to use for a portion occurs based on the source view camera that is most orthogonal to one or more surfaces in said portion. Petition 870250101956, dated 07 / 11 / 2025, pp. 78 / 112 4 / 25
[014] Images can be obtained from a multi-camera capture system comprising two or more source view cameras imaging a scene.
[015] In one example, two source view images and two source view depth maps are used. The target view can thus be rendered by distorting the first source view image using the first depth map and distorting the second view image using the second depth map. A view assignment map is then used to select which distorted view is selected at the target image coordinate.
[016] In another example, source view images can be obtained from a single camera moved from one position to another, assuming that the scene is static (i.e., without movement).
[017] The estimation of surface shapes involves identifying one or more object types and / or object features in the scene and assigning predetermined surface shapes to one or more objects and / or features based on the identified object type and / or object feature.
[018] Certain types of objects are known to have a general shape. For example, the surface of a person's head can generally be defined as convex. Thus, by identifying the type of object in the scene, a predetermined surface shape can be assigned to the object.
[019] Similarly, it is also known that object feature types (e.g., nose, eyebrows, wrinkles, etc.) have a general shape.
[020] This simplifies the step of estimating surface shapes compared to, for example, analyzing the depth map and calculating surface gradients directly.
[021] The predetermined surface shapes can be convex, concave or flat. Petition 870250101956, dated 07 / 11 / 2025, pp. 79 / 112 5 / 25
[022] In general, defining surface shapes as convex, concave, or planar provides a simple way to determine which source view should be used to render which part of the surface.
[023] For example, for concave shapes, a left portion of the shape is facing toward the right side of the multi-camera capture system and vice versa. Thus, a camera further to the right can better transmit the texture of the left portion of the surface and a camera further to the left can better transmit the texture of the right portion of the surface.
[024] Identifying one or more types of objects in the scene may involve applying an object detection algorithm to the images and / or the depth map.
[025] Identifying one or more types of objects in the scene may involve applying a keypoint detection algorithm to the images.
[026] Keypoint detection is typically performed for a given object category (e.g., a human). Furthermore, keypoint detection could be the basis for richer surface shape information assumptions. For example, using a human keypoint detection algorithm, a specific body part (e.g., upper left arm) can be identified.
[027] Estimating surface shapes can involve calculating surface normal vectors on the depth map and determining angles between normal vectors and vectors between the position from which source view images were captured and the surfaces.
[028] Determining the first portion and the second portion may involve comparing the angles determined for different source view images and assigning a source view image to each portion based on the angle comparison.
[029] In some cases, it may be preferable to have a pixel-by-pixel indication of the gradient to have a more accurate representation of the shapes rather than, by Petition 870250101956, dated 07 / 11 / 2025, pages 80 / 112 6 / 25 For example, assume the surface is convex / concave / planar. Calculating the normal vectors provides a more accurate surface shape, thus providing a more precise division between the first and second portions.
[030] Preferably, the determination of a first and a second portion comprises assigning a camera, per pixel, based on the camera with the smallest angle between the normal vector at the pixel's position and the vector between the camera and the pixel.
[031] In some examples, determining the first portion and the second portion involves determining a seam line that separates the first portion and the second portion using a line-weighted sum of image coordinates.
[032] This method simplifies the determination of the first and second portions, as there is no need to determine the normal vectors of the surfaces. The method can be particularly advantageous in video call applications, where there is a desire to synthesize an innovative view in real time.
[033] The sum of line-weighted image coordinates can be weighted according to the normalized depth of the target viewpoint.
[034] The method may additionally comprise applying a low-pass filter to the depth map and using the filtered depth map to estimate surface shapes.
[035] Applying a low-pass filter reduces noise in the depth map to reduce unwanted vision attributions caused by noise.
[036] In some cases, it may be preferable to apply a separate low-pass filter over pixels belonging to the same depth surface to average over abrupt depth transitions (e.g., between different surfaces at different depths). This would become a non-linear filter operation.
[037] Applying a low-pass filter can involve applying a low-pass filter separately to two or more surfaces in the depth map to generate the filtered depth map. Petition 870250101956, dated 07 / 11 / 2025, pages 81 / 112 7 / 25
[038] The method may also involve identifying surfaces on the depth map.
[039] The method may further comprise identifying one or more objects and / or object features in the scene and adapting the first portion and the second portion if an object and / or object feature is identified between the first and second portions.
[040] The region, or line, between the first and second portions may be called a stitch region / line. If this region / line falls over certain objects or object features, it may result in unwanted artifacts due to small differences in texture variations captured by the cameras (e.g., light reflections may be different between cameras due to their different positions). Thus, moving the stitch line / region by adapting the portions may reduce artifacts in the target view.
[041] Object features are features of an object (e.g., nose, eyebrows, etc.). These features can affect the overall shape of the object (e.g., head). Naturally, it will be appreciated that the distinction between an object and an object feature can be chosen arbitrarily. For example, when the expected scene is of one person, the head, arms, body, etc., can be treated as objects and the nose, eyebrows, etc., can be treated as object features. However, when the expected scene contains many people, the head, arms, body, etc., as well as the nose and eyebrows, can be treated as object features. The exact distinction between an object and an object feature can be based on the scene being photographed and / or personal preference.
[042] The method may further comprise calculating the color differences between the corresponding colors in the first source view and the second source view in a stitch region, wherein the stitch region is a portion of the target view image that overlaps both the first and second portions, and adjusting the colors in the stitch region based on the color differences to gradually change the colors of the first and second portions. Petition 870250101956, dated 07 / 11 / 2025, pages 82 / 112 8 / 25
[043] The seam region may overlap less than 20%, 15%, 10% or 5% of the first portion and less than 20%, 15%, 10% or 5% of the second portion. The seam region does not overlap the entire first portion and the entire second portion.
[044] The invention also provides a computer program carrier comprising computer program code which, when executed on a computer, causes the computer to execute all steps in accordance with any of the methods mentioned above.
[045] The computer program carrier may comprise computer memory (e.g., random access memory), computer storage (e.g., hard disks, solid-state drives, etc.). The computer program carrier may be a bitstream carrying the computer program code.
[046] The invention also provides a system for rendering a target view image of a scene at a target viewpoint, wherein the system comprises a processor configured to: Obtain at least two source view images of the captured scene from different positions and at least one depth map of the scene; Estimate surface shapes within the scene using source view images and / or the depth map, where surface shapes are indicative of surface gradients in the scene relative to the positions where the source view images were captured; Determine, based on the surface shapes and the positions from which a first source view image and a second source view image were captured: a first portion of the scene to be rendered using the first source view image; and a second portion of the scene to be rendered using the second source view image; and Petition 870250101956, dated 07 / 11 / 2025, pages 83 / 112 9 / 25 Render a target view image of the scene at the target viewpoint using at least one depth map and the first source view image for the first portion of the scene and the second source view image for the second portion of the scene.
[047] The processor is configured to estimate surface shapes by identifying one or more object types and / or object features in the scene and assigning predetermined surface shapes to one or more objects and / or features based on the identified object type and / or object feature.
[048] The predetermined surface shapes can be convex, concave or flat.
[049] The processor can be configured to identify one or more types of objects in the scene by applying an object detection algorithm to the images and / or the depth map.
[050] The processor can be configured to identify one or more types of objects in the scene by applying a keypoint detection algorithm to the images.
[051] The processor can be configured to estimate surface shapes by calculating surface normal vectors on the depth map and determining angles between normal vectors and vectors between the position from which the source view images were captured and the surfaces.
[052] The processor can be configured to determine the first portion and the second portion by comparing the angles determined for different source view images and assigning a source view image to each portion based on the angle comparison.
[053] The processor can be further configured to apply a low-pass filter to the depth map and use the filtered depth map to estimate surface shapes.
[054] The processor can be configured to apply the low-pass filter by separately applying a low-pass filter to two or more surfaces in the depth map to generate the filtered depth map. Petition 870250101956, dated 07 / 11 / 2025, pages 84 / 112 10 / 25
[055] The processor can be further configured to identify one or more objects and / or object features in the scene and adapt the first portion and the second portion if an object and / or object feature is identified between the first and second portions.
[056] The processor can be further configured to calculate the color differences between the corresponding colors in the first source view and the second source view in a stitch region, wherein the stitch region is a portion of the target view image that overlaps both the first and second portions, and adjust the colors in the stitch region based on the color differences to gradually change the colors of the first and second portions.
[057] These and other aspects of the invention will become evident and will be elucidated with reference to the embodiment(s) described further on. BRIEF DESCRIPTION OF THE DRAWINGS
[058] To ensure a better understanding of the invention, and to show more clearly how it can be carried out, reference will now be made, by way of example only, to the attached drawings, in which: Figure 1 illustrates two source-view cameras imaging an inclined surface; Figure 2 illustrates two cameras imaging a second inclined surface; Figure 3 illustrates the assignment of views to a convex surface using two cameras; Figure 4 illustrates the assignment of views to a concave surface using two cameras; Figure 5 illustrates view assignment in a video call scenario involving a single person; Figure 6 illustrates an adjustment of the stitching line identifying the person's nose in the innovative view; Petition 870250101956, dated 07 / 11 / 2025, pages 85 / 112 11 / 25 Figure 7 illustrates the view assignment for three cameras positioned in a triangular configuration; Figure 8 shows a target view image with two portions and a stitching region; and Figure 9 shows a method for rendering a target view image of a scene at a target viewpoint using a multi-camera capture system. DETAILED DESCRIPTION OF THE MODALITIES
[059] The invention will be described with reference to the figures.
[060] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the apparatus, systems and methods, are intended for illustrative purposes only and are not intended to limit the scope of the invention. These and other features, aspects and advantages of the apparatus, systems and methods of the present invention will be better understood from the following description, the appended claims and the accompanying drawings. It should be understood that the figures are merely schematic and are not drawn to scale. It should also be understood that the same reference numbers are used in all figures to indicate equal or similar parts.
[061] The invention provides a method for rendering a target view image of a scene at a target viewpoint. The method comprises obtaining at least two source view images of the scene captured from different positions and at least one depth map of the scene. Surface shapes within the scene are estimated using the source view images and / or the depth map, where the surface shapes are indicative of the surface gradients in the scene relative to the positions from which the source view images were captured. Based on the surface shapes and the positions from which a first source view image and a second source view image were captured, a first portion of the scene to be rendered using the first Petition 870250101956, dated 07 / 11 / 2025, pages 86 / 112 12 / 25 The source view image is determined, and a second portion of the scene to be rendered using the second source view image is also determined. A target view image of the scene is rendered at the target viewpoint using at least one depth map and the first source view image for the first portion of the scene and the second source view image for the second portion of the scene.
[062] It was noted that when synthesizing from a single-source view camera with a depth map, the innovative view quality remains superior compared to view fusion methods. However, a second or third depth-sensing camera is typically required to resolve the occlusion. Since fusion is not a good option, the question becomes which source view contribution should be selected for each pixel or region in the target view. Surface shape and orientation were found to be good properties to base the selection on. Generally, for better image quality and for a given region in the target view, a source view camera that views that region as orthogonally as possible to the 3D surface should be selected.
[063] Thus, it is proposed to let the surface shape determine, for each pixel in the target view, which source view contribution is selected. This shape-dependent view assignment takes advantage of traditional image stitching concepts, but differs in that images are distorted in a target view and assigned using surface shape criteria. The effect of this approach is that pixels are ideally selected from among source views to maintain resolution and that the associated stitching artifacts are much less visible than the fusion artifacts.
[064] Figure 1 illustrates two source view cameras 102 and 104 imaging an inclined surface 101. In this example, the entire inclined surface 101 is oriented more towards the source view camera 104 than towards the source view camera 102. Thus, a synthesized view (e.g., at a target viewpoint between Petition 870250101956, dated 07 / 11 / 2025, page 87 / 112 13 / 25 as cameras 102 and 104) would be rendered with images captured from the viewport camera 104.
[065] The orientation / gradient of the inclined surface 101 can be determined by comparing the angles between the normal vectors of the inclined surface 101 and the vectors of cameras 102 and 104 with the inclined surface 101.
[066] For example, at point 106 on inclined surface 101, the angle 01 between the normal vector 108 from point 106 and the vector 110 between camera 102 and point 106 is greater than the angle 02 between the normal vector 108 from point 106 and the vector 112 between camera 104 and point 106. Thus, point 106 on the inclined surface faces camera 104 more than camera 102. A vector between a camera and a point on the surface can be called a source view direction of a given pixel / region.
[067] Similarly, at point 114 on inclined surface 101, the angle 01 between the normal vector 116 from point 114 and the vector 120 between camera 102 and point 114 is greater than the angle 2 between the normal vector 116 from point 114 and the vector 118 between camera 104 and point 114. Thus, point 114 on the inclined surface faces camera 104 more than camera 102. As such, it can be assumed that all points on surface 101 can be assigned to images from camera 102.
[068] It will be appreciated that, for surfaces assumed to be planar, a single normal vector can be used as the normal vector can be translated over the entire surface.
[069] In general, angles 01 and 02 provide an indication of the surface gradient relative to the cameras. In the present invention, any indications of the gradient of a surface relative to the cameras are called surface shapes.
[070] Figure 2 illustrates two cameras 202 and 204 imaging a second inclined surface 201. Figure 2 illustrates a situation similar to that in Figure 1. However, in this case, the view assignment changes over surface 201. The gradient Petition 870250101956, dated 07 / 11 / 2025, pages 88 / 112 14 / 25 of surface 201 relative to cameras 202 and 204 can be determined using angles 01 between the source view directions from camera 202 and the normal vectors from surface 201 (e.g., normal vectors 206, 208, and 210) and additionally using angles 02 between the source view directions from camera 204 and the normal vectors from surface 201.
[071] For a first portion 212 of surface 201, where 01 < 02, pixels are selected from camera 202. However, for a second portion 216, where 01 > 02, pixels are selected from camera 204. There may also be a third portion 214, called a stitch portion, where the difference between the angles is less than a threshold. In stitch portion 214, pixels may be taken from camera 202 or camera 204, or the color, transparency, and pixel depth values from both cameras 202 and 204 may be combined to create a new color, transparency, or depth value. This can be achieved using filter operations.
[072] Figures 1 and 2 show planar surfaces. However, many surfaces in a scene will not be planar or cannot be considered planar. In these cases, it is possible to perform a surface shape analysis per pixel using the depth map to determine a normal vector per pixel. However, this analysis may be susceptible to noise in the depth map.
[073] Thus, for some object categories, such as human individuals, it is possible to assume that object surface shapes are convex or concave to circumvent a potentially noisy surface shape analysis. For example, for human individuals, it is possible to assume that most body parts have a convex depth profile. Examples of such objects are the head, neck, shoulders, and arms. However, a contrary example is an open hand facing the depth camera which results in a concave depth surface.
[074] Figure 3 illustrates the assignment of view to a convex surface 301 using two cameras 302 and 304. In this case, a normal vector 308 from point 306 can Petition 870250101956, dated 07 / 11 / 2025, pp. 89 / 112 15 / 25 can be found so that the angles between the normal vector 308 and the source view directions of cameras 302 and 304 are equal. As such, a left portion 310 of surface 301, to the left of point 306, can be assigned to camera 302 and a right portion 312, to the right of point 306, can be assigned to camera 304.
[075] In some applications, it can be assumed that point 306 will fall at the geometric center of surface 301, and thus there would be no need to calculate normal vector 308. For example, in video calls, it can be assumed that the head of a human individual facing the cameras is convex and is oriented towards the cameras so that the nose is directed exactly between cameras 302 and 304. Thus, in this case, it can be assumed that the center of the face is the seam line (i.e., the line between the first and second portions). Obviously, determining the normal vector where the angles are equal can provide a more robust, innovative image view synthesis, assuming little noise in the depth map.
[076] Figure 4 illustrates the view assignment for a concave surface 401 using two cameras 402 and 404. A normal vector 408 from point 406 can be found such that the angles between the normal vector 408 and the source view direction vectors of cameras 402 and 404 are equal. As such, a right portion 412 of surface 401, to the right of point 406, can be assigned to camera 402 and a left portion 410, to the left of point 406, can be assigned to camera 404.
[077] Concave and convex surfaces can be assumed, for example, using object detection and classification algorithms in images and / or depth maps and identifying one or more object types. Certain object types (e.g., head, shoulders, arms, etc.) can be considered convex or concave, or even planar. A convex, concave, or planar surface can thus be fitted to the surfaces in the depth map.
[078] Another approach may be to use keypoint detection algorithms in images to identify object types. Thus, the shapes of said object types can be assumed. Keypoint detection can be used to Petition 870250101956, dated 07 / 11 / 2025, pages 90 / 112 16 / 25 Identify key points on objects in an image. For example, human key point detection algorithms can be used to detect a human in the image. The key points (e.g., left knee, right elbow, left eye, etc.) can then be used to fit a convex or concave surface to certain human features (e.g., suitable convex surface for the head).
[079] Figure 5 illustrates view assignment in a video call scenario involving a single person 506. In this case, person 506 is imaged using two cameras, and two source view images 502 and 504 are obtained from the cameras. The two source view images 502 and 504 are then used to synthesize an innovative view 508 into a target viewpoint. In this case, the head, neck, and body of person 506 can be considered convex.
[080] To construct the view assignment, the pixels of source views 502 and 504 are first distorted to the target view using the source view depth maps. At the target view image coordinates (u, v), a line-weighted sum of image coordinates can be calculated to determine the stitch line 510. The stitch line is a line that separates a first region 512, for which the texture values of source view 502 are used, and a second region 514 for which the texture values of source view 504 are used.
[081] The position of stitch line 510 can be determined by calculating: T,Uw(mv)u Zuw(W) . UstitchW =
[082] When ustitch(v) is the position of the stitch line in the target view image coordinates and w(u, v) is a weighting factor that depends on the normalized depth z(u, v) as observed from the target viewpoint: w(u,v) = (1 — z(u, v)).
[083] The normalized depth z(u, v) can be obtained by distorting one or both of the depth-of-view maps from the source to the target viewpoint. Depth maps can be obtained, for example, by determining a map of Petition 870250101956, dated 07 / 11 / 2025, pp. 91 / 112 17 / 25 discrepancy between images 502 and 504 and transforming it into a depth map. A depth sensor can also be used to obtain the depth map.
[084] Calculating the position of seam line 510 directly means that there is no need to calculate the normal vectors for the surfaces involved. Instead, it can be assumed that the entire region 512 to the left of seam line 510 can be assigned to source view 502 and the entire region 514 to the right of seam line 510 can be assigned to source view 504 insofar as all surfaces are assumed to be convex.
[085] It will be appreciated that, if all surfaces could be assumed to be concave, region 512 would use the texture of source view 504 and region 514 would use the texture of source view 502.
[086] In some cases, it may be preferable to reposition the 510 stitch line based on object features in the images. For example, depth variation near the nose region has been found to cause artifacts when the stitch line does not pass through the center of the nose.
[087] Figure 6 illustrates an adjustment of the 510 stitch line identifying the nose. 602 of person 506 in innovative view 508. Figure 6a shows the stitch line 510 positioned next to the nose 602. It was found that such positioning of the stitch line 510 can make the artifacts more visible on the stitch line 510 closest to the nose 602 (for example, due to occlusion of one of the source views or due to the fact that locally, on both sides of the nose, the surface is actually concave rather than convex). In this case, the stitch line 510 can be adjusted to cross the center of the nose 602, as shown in Figure 6b. This can be achieved, for example, by detecting the position of nose 602 in the innovative view 508 (e.g., through object detection, keypoint detection, etc.) and adapting the position of the stitching line to cross the center of nose 602. Note that this may be more efficient and easier to implement than detecting a noise position in one or both source views and distorting that position to Petition 870250101956, dated 07 / 11 / 2025, pages 92 / 112 18 / 25 the target view (using one or more depth maps) and then modify the stitch line in the target view.
[088] Thus, knowing that seam line 510 will likely cross the center of person 506’s facial region, the offset for positioning seam line 510 can be introduced so that seam line 510 is in a more favorable position than, for example, crossing person 506’s cheek.
[089] Other objects may cause more visible artifacts when stitch line 510 crosses the center of said objects (for example, due to high depth variation at the center). Therefore, in other cases, it may be preferable to move stitch line 510 in the opposite direction to the center of said objects.
[090] More generally, the offset of seam line 510 is an adaptation of portions 512 and 514. In this example, region 512 is expanded near the face to include half of the nose 602 and region 514 is reduced to include only half of the nose 602.
[091] It will be recognized that objects can be identified in source views, distorted source views (i.e., target viewpoint), depth map(s), and / or distorted depth map(s).
[092] The concepts discussed here can also be expanded to a larger number of cameras. The analysis of surface shapes indicative of surface gradients generalizes to multiple source view cameras.
[093] Figure 7 illustrates the view assignment for three cameras 702, 704, and 706 positioned in a triangular configuration. Figure 7a shows the view assignment for a convex surface 701a imaged by cameras 702, 704, and 706. In this case, region 708a would be assigned to camera 702, region 710a would be assigned to camera 704, and region 712a would be assigned to camera 706.
[094] Figure 7b shows the view assignment for a concave surface 701 b imaged by cameras 702, 704 and 706. In this case, region 708b would be assigned to camera 702, region 710b would be assigned to camera 704 and region 712b would be assigned to camera 706. Petition 870250101956, dated 07 / 11 / 2025, pages 93 / 112 19 / 25
[095] The concepts described here enable the creation of an algorithm suitable for the real-time implementation of view assignment as part of a renderer that runs on the graphics computing unit (GPU) of a consumer device. Said algorithm could, for example, distort each source view depth map to the target viewpoint, combine the depth maps in the target view and select, for each pixel in the target view, the closest depth value, filter the combined depth map with a low-pass filter, compute a normal vector for each pixel in the target view, calculate the cosine of the angle between the normal vector and the source view direction and, for each pixel in the target view, assign the source view that corresponds to the minimum angle.
[096] The filtering step is included to avoid the case where the view assignment (and thus the portion positions) becomes sensitive to depth noise. For example, a Gaussian low-pass filter with a large sigma (e.g., > 30 pixels) can be used. The normal vector per pixel can be computed by constructing two 3D world vectors that have the 3D position of the central pixel as a starting point and each different neighboring pixel 3D position as endpoints. To avoid the case where the two vectors are parallel, they can be selected at 90-degree angles from the regular image grid. Let these two vectors be denoted by a1 and b2. If it is assumed that the depth surface can be locally approximated by a planar surface, the normal vector n can be approximated using the cross product: _a1xb2 |aixb2|.
[097] For each source view i, the cosine of the angle θι between the surface normal vector n and the source view direction can be calculated as follows: cos(^) = n · df. where is the normalized source view direction vector for source view i, which is calculated as follows: tt -Xj ltj—ΧίΓ Petition 870250101956, dated 07 / 11 / 2025, pages 94 / 112 20 / 25 where t(is the source view camera position in the world. The camera positions t(can be determined through calibration of extrinsic parameters (e.g., using the structure of motion algorithms) or may already be known (e.g., for cameras integrated into a single device). For a given pixel, the view assignment can now be defined as follows: i min = argminCcosCO. i
[098] In all the examples given above, only one source view is used per portion of the target view. However, it will be recognized that it may be possible to merge the source views per portion depending on the camera configuration. For example, a camera configuration might include two sets of cameras, a first set for imaging the right side of an object and a second set for imaging the left side of the object. If the cameras in a set are close to each other, it can be assumed that the texture values between the cameras will be similar (for example, reflections between cameras in a set will not differ significantly). In this case, it may be possible to render the right portion of the target view using a merge of the cameras in the first configuration and, similarly, render a left portion of the target view using a merge of the cameras in the second configuration.
[099] Figure 8 shows a target view image 802 with two portions 804 and 806 and a stitch region 810. The left portion 804 of the target view image 802 is rendered using a first distorted source view image for the target viewpoint, and the right portion 806 is rendered using a second distorted source view image for the target viewpoint. The line separating the left portion 804 and the right portion 806 is the stitch line 808. A stitch region 810 is defined around the stitch line 810 and overlaps a portion of the first portion 804 and a portion of the second portion 806.
[100] When the source view images used for portions 804 and 806 are taken with different cameras, and / or from different positions, the different viewing angles of the source view cameras, relative to a light source Petition 870250101956, dated 07 / 11 / 2025, pages 95 / 112 Dominant 21 / 25, can cause a visible color transition in stitch line 808. Thus, it is proposed to modify the color intensity of the distorted source views in the stitch region 810 as a function of the observed colors in relation to the position of stitch line 808 (or, more generally, to the view assignment map).
[101] Merging the contributions of both the first and second distorted source views in the stitch region 810 is one solution. However, this will likely introduce blur in the stitch region 810. Thus, it is additionally proposed to calculate the average color differences in sub-regions 812 of the stitch region 810 over multiple neighboring pixels and to adjust one or more source view colors gradually as one moves closer to the stitch line 808. The effect is that the original source view colors in the stitch region 810 are gradually altered in the left portion 804 and the right portion 806 as they represent the observed differences between colors in their corresponding source view images.
[102] For example, in sub-region 812, the corresponding pixels of the first distorted font view can be compared with the corresponding pixels of the second distorted font view. The average color difference between these pixels can be calculated and then used to adapt the pixel color in the center of sub-region 812. This can be done for all pixels in stitch region 810, so that the colors in stitch region 810 match at the stitch line.
[103] It will be recognized that the colors in the stitch region 810 closest to the left region 804 or the right region 806 will have the smallest adjustments to match neighboring pixels that are not in the stitch region 810. Similarly, the colors in the center of the stitch region 810 will likely be more adapted.
[104] The color adjustments can be split between the source views. In other words, half of the color difference measured at stitch line 808 can be added (or subtracted) to the first source view image and half can be subtracted (or added) to the second source view image. The effect Petition 870250101956, dated 07 / 11 / 2025, pages 96 / 112 22 / 25 of these adjustments means that the color difference between the source view images will be less visible on the 808 stitch line.
[105] The adjustments can increase linearly from zero offset at the edge of stitch region 810 towards stitch line 808, for each source view image. This can be implemented by calculating the shortest distance for each pixel to stitch line 808 and dividing that by half the width of stitch region 810.
[106] Color adaptation can use the same correction factor for each color channel (red, green, blue) or it can be applied in a transformed color space (e.g., YUV or other).
[107] Figure 9 shows a method for rendering a target view image of a scene at a target viewpoint using a multi-camera capture system. At least two source view images of the scene are obtained from different cameras of the multi-camera capture system, and at least one depth map of the scene is also obtained in step 902. Surface shapes within the scene are estimated using the source view images and / or the depth map in step 904, where the surface shapes are indicative of the surface gradients in the scene relative to the cameras. The surface shapes are estimated by identifying one or more object types and / or object features in the scene and assigning predetermined surface shapes to the one or more objects and / or object features based on the identified object type and / or object feature.In step 906, a first portion of the scene, to be rendered using a first source view image, and a second portion of the scene, to be rendered using a second source view image, are determined based on the surface shapes and the position of the cameras corresponding to the first source view image and the second source view image. A target view image of the scene can then be rendered at the target viewpoint in step 908 using the depth map and the first source view image for the first portion of the scene and the second source view image for the second portion of the scene. Petition 870250101956, dated 07 / 11 / 2025, pages 97 / 112 23 / 25
[108] Someone skilled in the art would readily be able to develop a processor to execute any method described here. Thus, each step of a flowchart can represent a different action performed by a processor, and can be executed by a respective processing module.
[109] One or more steps of any of the methods described herein may be performed by one or more processors. A processor comprises an electronic circuit suitable for processing data. Any method described herein may be computer-implemented, wherein computer-implemented means that the steps of said methods are performed by one or more computers and wherein a computer is defined as a device suitable for data processing. A computer may be suitable for processing data in accordance with prescribed instructions.
[110] As discussed above, the system makes use of the processor to perform data processing. The processor can be implemented in various ways, with software and / or hardware, to perform the various functions required. The processor can typically employ one or more microprocessors that can be programmed using software (e.g., microcode) to perform the necessary functions. The processor can be implemented as a combination of dedicated hardware to perform some functions and one or more programmed microprocessors and associated circuits to perform other functions.
[111] Examples of circuit assemblies that can be employed in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).
[112] In various implementations, the processor can be associated with one or more storage media, such as volatile and non-volatile computer memory, such as random access memory (RAM), programmable read-only memory (PROM), memory Petition 870250101956, dated 07 / 11 / 2025, pages 98 / 112 24 / 25 erasable programmable read-only memory (EPROM) and electrically-erasable programmable read-only memory (EEPROM). Storage media can be encoded with one or more programs that, when executed on one or more processors and / or controllers, perform the necessary functions. Various storage media can be fixed inside a processor or controller, or they can be portable, so that the one or more programs stored on them can be loaded into a processor.
[113] Variations of the embodiments presented may be understood and carried out by those skilled in the art in the practice of the claimed invention, based on a study of the drawings, disclosure and appended claims. In the claims, the expression that includes does not exclude other elements or steps, and the indefinite article a or an does not exclude a plurality.
[114] A single processor or other unit may perform the functions of several items mentioned in the claims.
[115] The mere fact that certain measures are mentioned in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.
[116] A computer program can be stored / distributed on suitable media, such as optical storage media or solid-state media, supplied together with other hardware, or as part of it, but it can also be distributed in other ways, such as via the Internet or other wired or wireless telecommunication systems.
[117] If the term adapted for is used in the claims or description, it should be noted that the term adapted for is intended to be equivalent to the term configured for.
[118] Any reference marks in the claims should not be interpreted as limiting the scope. Petition 870250101956, dated 07 / 11 / 2025, pages 99 / 112 25 / 25
[119] Any methods described here exclude a method for performing intellectual acts or activities such as these. Petition 870250101956, dated 07 / 11 / 2025, pages 100 / 112
Claims
1 / 4 CLAIMS 1. A method for rendering a target view image of a scene at a target viewpoint, wherein the method is characterized by comprising: obtaining (902) at least two source view images of the scene captured from different positions and at least one depth map of the scene; estimating (904) surface shapes within the scene using the source view images and / or the depth map, wherein the surface shapes are indicative of the surface gradients in the scene relative to the positions at which the source view images were captured, wherein the estimation of the surface shapes comprises identifying one or more object types and / or object features in the scene and assigning predetermined surface shapes to one or more objects and / or object features based on the identified object type and / or object feature;determine (906), based on the surface shapes and the positions from which a first source view image and a second source view image were captured: a first portion of the scene to be rendered using the first source view image; and a second portion of the scene to be rendered using the second source view image; and render (908) a target view image of the scene at the target viewpoint using at least one depth map and the first source view image for the first portion of the scene and the second source view image for the second portion of the scene.; 2. A method according to claim 1, characterized in that the predetermined surface shapes are convex, concave, or flat.
3. Method, according to claim 1 or 2, characterized in that the identification of one or more types of objects in the scene comprises applying an object detection algorithm to the images and / or the depth map. Petition 870250101956, dated 07 / 11 / 2025, pp. 101 / 112 2 / 4 4. A method, according to any one of claims 1 to 3, characterized by the identification of one or more types of objects in the scene comprising applying a keypoint detection algorithm to the images.
5. Method, according to any one of claims 1 to 4, characterized by: the estimation of surface shapes comprising calculating the normal vectors of surfaces in the depth map and determining angles between the normal vectors and vectors between the position at which the source view images were captured and the surfaces, and the determination of the first portion and the second portion comprising comparing the angles determined for different source view images and assigning a source view image to each portion based on the comparison of the angles.
6. A method, according to any one of claims 1 to 4, characterized in that the determination of the first portion and the second portion comprises determining a seam line separating the first portion and the second portion using a line-weighted sum of image coordinates.
7. Method, according to claim 6, characterized in that the sum of line-weighted image coordinates is weighted according to the normalized depth from the target viewpoint.
8. A method, according to any one of claims 1 to 7, characterized by further comprising applying a low-pass filter to the depth map and using the filtered depth map to estimate surface shapes.
9. Method, according to claim 8, characterized by the application of the low-pass filter comprising separately applying a low-pass filter to two or more surfaces in the depth map to generate the filtered depth map. Petition 870250101956, dated 07 / 11 / 2025, pp. 102 / 112 3 / 4 10. A method, according to any one of claims 1 to 9, characterized by further comprising identifying one or more objects and / or object features in the scene and adapting the first portion and the second portion if an object and / or object feature is identified between the first and second portions.
11. A method according to any one of claims 1 to 10, characterized by further comprising: calculating color differences between corresponding colors in the first source view and the second source view in a stitch region, wherein the stitch region is a portion of the target view image that overlaps both the first and second portions; and adjusting the colors in the stitch region based on the color differences to gradually alter the colors of the first and second portions.
12. Computer program carrier characterized by comprising computer program code that, when executed on a computer, causes the computer to execute all the steps as defined in any one of claims 1 to 11.
13. System for rendering a target view image of a scene at a target viewpoint, the system being characterized by comprising a processor configured to: obtain (902) at least two source view images of the scene captured from different positions and at least one depth map of the scene; estimate (904) surface shapes within the scene using the source view images and / or the depth map, wherein the surface shapes are indicative of the surface gradients in the scene relative to the positions at which the source view images were captured, wherein the processor is configured to estimate the surface shapes by identifying one or more object types and / or object features in the scene and assigning predetermined surface shapes to one or more objects and / or object features based on the identified object type and / or object feature; Petition 870250101956, dated 07 / 11 / 2025, p.103 / 112 4 / 4 determine (906), based on the surface formats and the positions from which a first source view image and a second source view image were captured: a first portion of the scene to be rendered using the first source view image; and a second portion of the scene to be rendered using the second source view image; and render (908) a target view image of the scene at the target viewpoint using at least one depth map and the first source view image for the first portion of the scene and the second source view image for the second portion of the scene.
14. System according to claim 13, characterized in that: the processor is configured to estimate surface shapes by calculating surface normal vectors in the depth map and determining angles between the normal vectors and vectors between the position where the source view images were captured and the surfaces, and the processor is configured to determine the first portion and the second portion by comparing the angles determined for different source view images and assigning a source view image to each portion based on the angle comparison.
15. System, according to any one of claims 13 to 14, characterized in that the processor is further configured to: calculate color differences between corresponding colors in the first source view and the second source view in a stitching region, wherein the stitching region is a portion of the target view image that overlaps both the first and second portions; and adjust the colors in the stitching region based on the color differences to gradually alter the colors of the first and second portions. Petition 870250101956, dated 07 / 11 / 2025, pp. 104 / 112
