Image generation

Abstract

The device includes a receiver 601 that receives captured video data for a real-world scene, the video data being linked to a capture pose region. A store 615 stores a 3D mesh model of the real-world scene. A renderer 605 generates an output image for a viewport for a viewing pose. The renderer 605 includes a first circuit 607 configured to generate first image data for the output image by projecting the captured video data onto the viewing pose, and a second circuit configured to determine second image data for the first region of the output image in response to the 3D mesh model. A third circuit 611 generates the output image to include at least a portion of the first image data and the second image data for the first region. A fourth circuit 613 determines the first region based on a deviation of the viewing pose from the capture pose region.

Description

Technical Field

[0001] The present invention relates to an image generation approach, and more particularly, but not exclusively, to the generation of images for three-dimensional video signals for different viewpoints.

Background Art

[0002] Recently, new services and methods for consuming images and videos have been continuously developed and introduced, and the diversity and scope of image applications and video applications have been significantly increasing.

[0003] For example, as one increasingly popular service, it is possible for a viewer to change the position or direction being viewed in a scene so that the presented video adapts to present a view from a changed position or direction. There are those that provide an image sequence in such a manner that the viewer can interact actively and dynamically with the view.

[0004] The capture, distribution, and presentation of three-dimensional video have become increasingly popular and attractive in some applications and services. A particular approach is known as immersive video and typically involves providing a real-world scene that allows for small viewer movements, such as relatively small head movements and rotations, and often a view of real-time events. For example, a real-time video broadcast, such as a sports event that allows for local client-based view generation following a small movement of the viewer's head, provides the user with the impression of sitting in a stand watching the sports event. The user will have a natural experience similar to that of a spectator present at that position in the stand, such as being able to look around. Recently, the popularity of display devices equipped with applications that support position tracking and 3D interaction based on 3D capture of real-world scenes has been expanding. Such display devices are particularly suitable for immersive video applications that provide an enhanced three-dimensional user experience.

[0005] To provide such services for real-world scenes, the scene is typically captured from different positions and using different camera capture poses. As a result, the relevance and importance of multi-camera capturing and 6DoF (6 degrees of freedom) processing are rapidly increasing. Applications include live concerts, live sports, and telepresence. The freedom to choose one's own viewpoint enriches these applications by increasing the sense of presence compared to regular video. Furthermore, immersive scenarios become conceivable, in which case the observer can navigate and interact with the captured live scene. For broadcast applications, this requires real-time depth assessment on the production side and real-time field of view synthesis on the client device. Both depth assessment and field of view synthesis introduce errors, and these errors depend on the details of the algorithm implementation used. In many such applications, 3D scene information is often provided that allows for the synthesis of high-quality field of view images for viewpoints relatively close to a reference (multiple) viewpoint, but if the viewpoint deviates too much from the reference viewpoint, it degrades the synthesis of high-quality field of view images.

[0006] For example, to provide 3D image data in the form of multiple 2D images from offset positions and / or as image data plus depth data, a pair of video cameras offset from each other capture the scene. A rendering device dynamically processes this 3D data to generate images for different changing viewpoint positions / directions. The rendering device can dynamically perform viewpoint shifts or projections to dynamically follow the user's movement.

[0007] A problem with immersive video is that the viewer's field of view—the space in which they experience sufficient image quality—is limited. As the viewer moves outside the field of view, the degradation and errors resulting from the synthesis of field-of-view images become increasingly pronounced, ultimately leading to an unacceptable user experience. Errors, artifacts, and inaccuracies in the generated field-of-view images arise, in particular, from the provision of 3D video data that does not provide sufficient information (e.g., non-occluding data) for field-of-view synthesis.

[0008] For example, when multiple cameras are typically used to capture a 3D representation of a scene, playback on a virtual reality headset tends to be spatially limited to a virtual viewpoint located near the original camera positions. This ensures that the rendering quality of the virtual viewpoint does not show artifacts, typically resulting from missing information (obstructed data) or 3D evaluation errors.

[0009] Within the so-called sweet spot field of view, rendering can be performed directly from one or more reference camera images with associated depth maps or meshes using standard texture mapping combined with field-of-view blending.

[0010] Outside the sweet spot of the field of view, image quality deteriorates, often to an unacceptable degree. In current applications, this is addressed by providing the viewer with a blurred or even black image for parts of the scene that cannot be rendered accurately enough. However, such an approach tends to be suboptimal and provides a suboptimal user experience. European Patent Application Publication 3422711A1 discloses an example of a rendering system in which blurring is introduced to bias the user away from parts of the scene that are not represented by an incomplete depiction of the scene. [Overview of the project] [Problems that the invention aims to solve]

[0011] Therefore, an improved approach may be advantageous. In particular, an approach that enables improved operation, enhanced flexibility, an improved immersive user experience, reduced complexity, ease of implementation, improved perceived and synthesized image quality, improved rendering, increased (possibly virtual) degrees of freedom of movement for the user, an improved user experience, and / or improved performance and / or operation would be advantageous.

[0012] Therefore, the present invention preferably aims to mitigate, alleviate, or eliminate one or more of the aforementioned disadvantages, either individually or in any combination. [Means for solving the problem]

[0013] According to one aspect of the present invention, a device is provided comprising: a first receiver configured to receive video data of a real-world scene, the video data being captured and linked to a capture pose region; a store configured to store a 3D mesh model of at least a portion of the real-world scene; a second receiver configured to receive a viewing pose; and a renderer configured to generate an output image for a viewport of a viewing pose, wherein the renderer comprises: a first circuit configured to generate first image data for a viewport of at least a portion of the output image by projecting the captured video data onto the viewing pose; a second circuit configured to generate second image data for an output viewport of at least a first region of the output image from the 3D mesh model; a third circuit configured to generate an output image including at least a portion of the first image data and the second image data for the first region; and a fourth circuit configured to determine the first region in accordance with the deviation of the viewing pose from the capture pose region.

[0014] The present invention provides an improved user experience in many embodiments and scenarios. It enables an improved trade-off between image quality and freedom of motion for many applications. This approach often provides a more immersive user experience and is particularly suitable for immersive video applications. This approach reduces the perceived degradation of image quality for different viewing poses. This approach provides the user with an improved experience with respect to a larger range of changes in position and / or orientation. In many embodiments, this approach relaxes the requirements for capturing real-world scenes. For example, fewer cameras are used. The requirements regarding how much of the scene is captured are relaxed. In many embodiments, this approach relaxes the requirements for data communication, for example, allowing for lower latency dialogue services.

[0015] This approach enables, for example, an improved immersive video experience.

[0016] A pose is a position and / or orientation. A pose region is a sequence of poses. A capture pose region is a region in the captured video data that provides data enabling the generation of image data that meets quality standards. An output image is an image in an image sequence, and in particular, a frame / image in a video sequence.

[0017] The 3D mesh model further includes at least one pixel map having pixel values ​​linked to the vertices of the 3D mesh of the 3D mesh model.

[0018] According to an optional feature of the present invention, the renderer is configured to determine a first region as a region in which the quality of the first image data generated by the first circuit does not meet the quality criteria.

[0019] In some embodiments, the renderer is configured to determine an intermediate image that includes first image data and to determine a first region as a region in which the quality of the image data of the intermediate image does not meet a quality criterion.

[0020] This provides improved and / or facilitated operation in many embodiments. It provides a particularly efficient approach for determining a first region that is particularly suitable for providing an appealing user experience.

[0021] According to an optional feature of the invention, the third circuit is configured to determine the first region according to a difference between a viewing pose and a capture pose region.

[0022] This provides improved and / or facilitated operation in many embodiments. It provides a particularly efficient approach for determining a first region that is particularly suitable for providing an appealing user experience.

[0023] In many embodiments, the third circuit is configured to determine the first region according to a distance between a viewing pose and a capture pose region. This distance is determined according to a suitable distance measure. This distance measure reflects the distance as the position and / or orientation of the viewing pose relative to the capture pose region.

[0024] According to an optional feature of the invention, this difference is an angular difference.

[0025] This provides improved and / or facilitated operation in many embodiments.

[0026] According to an optional feature of the invention, the renderer is configured to adapt second image data according to the captured video data.

[0027] This provides an improved user experience in many embodiments. It provides output images that are generated more consistently and less contradictorily in many scenarios, and reduces the perceived visibility of the difference between the portion of the output image generated from video data and the portion of the output image generated from the 3D mesh model.

[0028] According to an optional feature of the present invention, the renderer is configured to adapt the first image data according to the 3D mesh data.

[0029] This provides an improved user experience in many embodiments. It provides output images that are generated more consistently and less contradictorily in many scenarios, and reduces the perceived visibility of the difference between the portion of the output image generated from video data and the portion of the output image generated from the 3D mesh model.

[0030] According to an optional feature of the present invention, the renderer is configured to adapt the second image data according to the first image data.

[0031] This provides an improved user experience in many embodiments. It provides output images that are generated more consistently and less contradictorily in many scenarios, and reduces the perceived visibility of the difference between the portion of the output image generated from video data and the portion of the output image generated from the 3D mesh model.

[0032] According to an optional feature of the present invention, the renderer is configured to adapt the first image data according to the second image data.

[0033] This provides an improved user experience in many embodiments. It provides output images that are generated more consistently and less contradictorily in many scenarios, and reduces the perceived visibility of the difference between the portion of the output image generated from video data and the portion of the output image generated from the 3D mesh model.

[0034] According to an optional feature of the present invention, the renderer is configured to adapt a 3D mesh model according to a first image data.

[0035] This provides an improved user experience in many embodiments. It provides output images that are generated more consistently and without inconsistency in many scenarios, and reduces the perceived visibility of the difference between parts of the output image generated from video data and parts of the output image generated from a 3D mesh model.

[0036] According to an optional feature of the present invention, the apparatus further comprises a model generator for generating a three-dimensional mesh model in response to captured video data.

[0037] In many embodiments, this provides improved operation and / or a simplified implementation.

[0038] According to an optional feature of the present invention, the first receiver is configured to receive video data from a remote source and to further receive a 3D mesh model from the same remote source.

[0039] In many embodiments, this provides improved operation and / or a simplified implementation.

[0040] According to an optional feature of the present invention, the second circuit is configured to vary the level of detail for the first region in accordance with the deviation of the field of view pose relative to the capture pose region.

[0041] In many embodiments, this provides a further improved user experience and improved perceptual adaptation to changes in the viewer's pose.

[0042] According to an optional feature of the present invention, the first receiver is further configured to receive captured second video data for a real-world scene, the second video data being linked to a second capture pose region, the first circuit is further configured to determine third image data for at least a portion of the output image by projecting the captured second video data onto a viewing pose, and the third circuit is configured to determine the first region according to the deviation of the viewing pose from the second capture pose region.

[0043] This provides an enhanced user experience in many scenarios and embodiments.

[0044] According to one aspect of the present invention, a method is provided comprising the steps of: receiving captured video data of a real-world scene, the video data being linked to a capture pose region; storing a three-dimensional mesh model of at least a portion of the real-world scene; receiving a viewing pose; and generating an output image for a viewport of the viewing pose, wherein the step of generating the output image comprises: generating first image data for a viewport of at least a portion of the output image by projecting the captured video data onto the viewing pose; generating second image data for an output viewport of at least a first region of the output image from the three-dimensional mesh model; generating an output image so as to include at least a portion of the first image data and the second image data for the first region; and determining the first region according to the deviation of the viewing pose from the capture pose region.

[0045] These and other aspects, features, and advantages of the present invention will become apparent from and be explained using the embodiments described herein hereafter.

[0046] Implementations of the present invention will be described by reference to the following drawings, merely as examples. [Brief explanation of the drawing]

[0047] [Figure 1] This is an illustration of an example of elements of a video distribution system according to several embodiments of the present invention. [Figure 2] This is an illustration of an example of 3D scene capture. [Figure 3] This is an illustration of an example of the field of view generated for a specific viewing posture. [Figure 4] This is an illustration of an example of the field of view generated for a specific viewing posture. [Figure 5] This is an illustration of an example of the field of view generated for a specific viewing posture. [Figure 6] This is an illustration of an example of elements of a video rendering device according to several embodiments of the present invention. [Figure 7] This is an illustration of an example of the field of view generated for a specific viewing posture. [Figure 8] This is an illustration of an example of the field of view generated for a specific viewing posture. [Figure 9] This is an illustration of an example of capturing a 3D scene using two sets of capture cameras. [Modes for carrying out the invention]

[0048] The following explanation focuses on immersive video applications, but it should be understood that the principles and concepts described can be applied to many other applications and embodiments.

[0049] In many approaches, immersive video is delivered locally to the viewer by a standalone device that, for example, does not use any remote video server, or even has any access to any remote video server. However, in other applications, immersive applications may rely on data received from a remote or central server. For example, video data may be provided from a remote central server to a video rendering device and processed locally to generate the desired immersive video experience.

[0050] Figure 1 illustrates such an example of an immersive video system, in which a video rendering device 101 communicates with a remote immersive video server 103 via a network 105, such as the Internet. The server 103 is configured to support a potentially large number of video rendering devices 101 simultaneously.

[0051] The immersive video server 103 supports the immersive video experience, for example, by transmitting 3D video data that describes a real-world scene. This data specifically describes the visual features and geometric properties of the scene, which is generated from real-time capture of the real world by a pair of (possibly 3D) cameras.

[0052] For example, as illustrated in Figure 2, a pair of cameras are arranged to be individually offset (e.g., linearly) as an appropriate capture setting, and each captures an image of scene 203. The captured data is used to generate a 3D video data stream, which is sent from the immersive video server 103 to a remote video rendering device.

[0053] 3D video data is, for example, a video stream, and includes, for example, directly captured images from multiple cameras, and / or processed data such as depth data generated from the images and captured images. Many techniques and approaches for generating 3D video data are known, and it will be understood that any suitable approach and 3D video data format / representation can be used without diminishing the value of the present invention.

[0054] The immersive video rendering device 101 is configured to receive 3D video data and process the received 3D video data to generate an output video stream, in which the generated output video stream dynamically reflects changes in the user's pose, thereby providing an immersive video experience in which the provided scenery adapts to changes in the user's field of view / pose / position.

[0055] In this field, the terms "position" and "pose" are used as general terms for position and / or orientation. For example, the combination of position and orientation of an object, camera, head, or view is referred to as a pose or position. Thus, an instruction for position or pose includes six values / components / degrees of freedom, each value / component typically describing a specific property of the corresponding object's position or orientation. Of course, in many situations, for example, when one or more components are considered fixed or meaningless, the position or pose will be considered or represented using fewer components (for example, if all objects are considered to be at the same height and have a horizontal orientation, four components provide a complete representation of the object's pose). Hereafter, the term "pose" is used to refer to a position and / or orientation that can be represented by one to six values ​​(corresponding to the maximum possible degrees of freedom). The term "pose" is interchangeable with the term "position." The term "pose" is interchangeable with the term "position and / or orientation." The term "pose" can be interchanged with the terms "position and orientation" (when the pose provides information about both position and orientation), the term "position" (when the pose provides information about position, or perhaps just position), and the term "orientation" (when the pose provides information about orientation, or perhaps just orientation).

[0056] The quality of the generated view image depends on the image and depth information available for the field-of-view stacking operation. It also depends on the amount of reprojection and field-of-view shifting required.

[0057] For example, field shifting typically results in the removal of occlusion from parts of an image that are not visible in the main image used for the field shift. Such holes are filled in by data from other images if these capture the objects that are being removed from occlusion, but it is also possible that the parts of the image that are being removed from occlusion for the new viewpoint are also missing from the other source fields of view. In such cases, field stacking needs to estimate the data based on surrounding data, for example. The process of removing occlusion inherently tends to introduce inaccuracies, artifacts, and errors. Furthermore, this tends to increase with the amount of field shift, and in particular, the probability of missing data (holes) increases with increasing distance from the image's capture pose during field stacking.

[0058] Another possible source of distortion is incomplete depth information. Often, depth information is provided by depth maps generated by depth estimation (e.g., disparity estimation between source images) or measurement (e.g., ranging), but these are not perfect, so the depth values ​​contain errors and inaccuracies. Field-of-view shifting relies on depth information, and incomplete depth information leads to errors or inaccuracies in the composite image. The further the composite viewpoint is from the original camera viewpoint, the more severe the distortion in the composite target field-of-view image becomes.

[0059] Therefore, as the viewing pose moves further and further away from the capture pose, the quality of the composite image tends to deteriorate. If the viewing pose is far enough away from the capture pose, the image quality will degrade to an unacceptable degree, resulting in a poor user experience.

[0060] Figures 3 to 5 illustrate the problems associated with moving away from the capture pose. Figure 3 illustrates an example where, because the composite viewport is aligned closely with the capture camera's viewport, a specific image for the viewing pose viewport can be predicted from the capture camera using depth-image-based rendering, resulting in a high-quality image. In contrast, in the examples of Figures 4 and 5, the viewing pose and the capture pose differ only by the angular direction of a different viewport than the capture viewport. As illustrated, as a result of the angular change in the field of view, a large portion of the image (the right or left side of the image in this example) is not provided with adequate image data. Furthermore, while extrapolation information from the image data into the unknown range may provide some improved perception, as illustrated, it results in very significant degradation and distortion, leading to an unrealistic representation of the scene.

[0061] Viewing pose and capture pose differ only in the deviation of the field of view position and / or angle, and their effects are different. Positional changes, such as movement, tend to increase the occlusion removal range behind foreground objects, increasing the unreliability of field of view synthesis due to the uncertainty of 3D (depth / geometric shape) estimation. As a result of angle changes in the viewpoint that rotate and move away from the capture camera angle, for example (as illustrated in Figures 4 and 5), a situation arises where image data is unavailable over a large range of the new viewport.

[0062] As a result of the above problems, an insufficient immersive effect occurs because the entire field of view of the display (e.g., typically 110 degrees) is filled, and head rotation does not lead to new content. Also, spatial content is often lost, and navigation can be more difficult when the image is blurry or otherwise of low quality. Several different approaches have been proposed to address these problems, but they tend to be suboptimal, in particular, as they undesirably restrict user movement or result in undesirable user effects.

[0063] Figure 6 illustrates a video rendering apparatus / system / device that provides performance and approaches capable of achieving a more desirable user experience in many scenarios. Specifically, this apparatus may be the video rendering device 101 shown in Figure 1.

[0064] This video rendering device includes a first receiver 601 configured to receive captured video data of a real-world scene. In this example, the video data is provided by a video server 103.

[0065] The video data is captured video data of a real-world scene, typically three-dimensional video data generated from scene captures by multiple cameras offset from each other. The video data may be, for example, multiple video streams from different cameras, or video data for one or more capture locations with depth information. It will be understood that many different approaches are known for capturing video data of a real-world scene, generating (three-dimensional) video data representing that capture, and communicating / distributing the video data, and that any suitable approach may be used without diminishing the value of the present invention.

[0066] In many embodiments, 3D video data includes images of multiple fields of view and therefore includes multiple (simultaneous) images of a scene from different viewpoints. In many embodiments, 3D video data has the form of images and depth map representations, in which case the images / frames are provided with associated depth maps. 3D image data, in particular, includes depth representations added to multiple fields of view, and for each frame, includes at least two images from different viewpoints, at least one of which has an associated depth map. If the received data is, for example, a multi-field of view data representation without an explicit depth map, the depth map can be generated using an appropriate depth estimation algorithm, specifically a disparity estimation-based approach using different images of the multi-field of view representations.

[0067] In this specific example, the first receiver 601 receives MVD (Multiple Field of View and Depth) video data that describes a 3D scene using a series of multiple simultaneous images and depth maps, which will hereafter be referred to as source images and source depth maps. It will be understood that for a video stream, a series of such 3D images are provided in time.

[0068] The received video data is linked to a capture pose region. A capture pose region is typically a region of the scene adjacent to the capture pose, and is typically the region containing the capture pose. The capture pose region is a range of intervals for one, more, or all of the parameters that represent the capture pose and / or the view pose. For example, if the pose is represented by a two-dimensional position, the capture pose region is represented by the range of the two corresponding positions, i.e., as a two-dimensional range. In other embodiments, the pose is typically represented by six parameters, such as three position parameters and three orientation parameters, in which case the capture pose region is given by the limits in these six parameters, i.e., by the complete 6DoF representation of the pose.

[0069] In some examples, a capture pose region is a single capture pose corresponding to a single pose corresponding to a viewport (field of view position and orientation) for the provided captured video data. A capture pose region can be a set of poses that indicate / contain one or more poses in which the scene was captured.

[0070] In some embodiments, the capture pause region is provided directly from the video data source, specifically contained within the received video data stream. In some embodiments, it is provided specifically as metadata for the video data stream. In the example in Figure 2, the video data communicated to the video rendering device 101 is provided based on a row of cameras 205 positioned within the capture pause region 205.

[0071] In some embodiments, the video rendering device is configured to use the capture pause region as it is received directly. In other embodiments, the video rendering device may be configured to modify the capture pause region, or it may generate the capture pause region itself.

[0072] For example, in some embodiments, the received data only contains video data corresponding to a given capture pose, and does not include instructions for the capture pose itself, instructions for any enlarged area, or instructions on how appropriate the image data is to view a composite for poses other than the given capture pose. In such cases, the receiver 601 will, for example, generate a capture pose region based on the received capture pose. For example, it assumes that since the provided video data is linked to a reference pose, the video data is rendered directly relative to this reference pose without any field-of-view shifting or projection. Then, all poses are measured in relation to this reference pose, and the capture pose region is determined as the reference pose, or, for example, as a predetermined region centered on the reference pose. When the user moves, the viewing pose is then represented / measured in relation to this reference pose.

[0073] In some embodiments, the capture pose region is considered to simply correspond to a single pose, such as that of the received video data. In other embodiments, the receiver 401 generates an extended capture pose region by, for example, evaluating quality degradation as a function of the difference from or distance to the capture pose. For example, for various test poses that deviate from the capture pose region by different amounts, the first receiver 601 evaluates what large a proportion of the corresponding viewport is covered by the image data, and what large a proportion corresponds to, for example, deoccluded areas / objects or areas / objects for which no data is provided, due to the viewport extending into parts of the scene not covered by the capture camera. The capture pose region is determined, for example, as a 6-dimensional region where the proportion of the corresponding viewport not covered by the image data is below a given threshold. It will be understood that many other approaches are possible to evaluate the quality level or degradation as a function of the deviation between the capture pose and the viewing pose, and any appropriate operation may be used.

[0074] As another example, the first receiver 601 modifies the capture pose region to include all poses whose distance to the nearest capture pose is less than a given threshold, for example, to the nearest camera pose if multiple camera poses are provided, or to the nearest pose in the received capture pose region from which the video image is provided. The distance is determined according to any suitable distance measure that includes, as far as possible, consideration of both positional distance and angular (orientational) distance.

[0075] In other embodiments, other approaches are used to determine the capture pose region, and it will be understood that the specific approach to determining the capture pose region, which reflects a set of poses from which images can be generated with adequate quality, will depend on the requirements and preferences of that particular embodiment.

[0076] The video rendering apparatus in Figure 6 further includes a second receiver 603 configured to receive a viewing pose for the viewer (and especially in a three-dimensional scene). The viewing pose represents the position and / or orientation in which the viewer views the scene and specifically provides the pose from which the field of view of the scene should be generated.

[0077] Many different approaches are known for determining and providing viewing poses, and it will be understood that any appropriate approach will be used. For example, the second receiver 603 is configured to receive pose data from a VR headset or eye tracker worn by the user. In some embodiments, a relative viewing pose is determined (e.g., a change from an initial pose is determined), which may relate to a reference pose, such as a camera pose or the center of a capture pose area.

[0078] The first and second receivers 601 and 603 are implemented in any suitable manner and receive data from any suitable source, including local memory, network connections, wireless connections, data media, etc.

[0079] These receivers are implemented as one or more integrated circuits, such as application-specific integrated circuits (ASICs). In some embodiments, these receivers are implemented as one or more programmed processing units, such as firmware or software running on a suitable processor, such as a central processing unit, a digital signal processing unit, or a microcontroller. In such embodiments, it will be understood that the processing units include onboard or external memory, clock drive circuits, interface circuits, user interface circuits, and so on. These circuits are further implemented as integrated circuits and / or discrete electronic circuits as part of the processing units.

[0080] The first and second receivers 601 and 603 are coupled to a field-of-view synthesis or projection circuit, i.e., a renderer 605, configured to generate field-of-view frames / images from the received 3D video data, in which case the field-of-view images are generated to represent the field of view of the 3D scene from the viewing pose. Thus, the renderer 605 generates a video stream of field-of-view images / frames for the 3D scene from the received video data and the viewing pose. Below, the operation of the renderer 605 is described with reference to the generation of a single image. However, it will be understood that in many embodiments, the image is part of a series of images, specifically, frames of a video sequence. In fact, the approach described applies to multiple frames / images of the output video sequence, often all of them.

[0081] In many cases, it will be understood that a stereo video sequence is generated, which includes a video sequence for the right eye and a video sequence for the left eye. Therefore, when an image is presented to the user via an AR / VR headset, for example, it will appear as if the user is viewing a 3D scene from their viewing position.

[0082] The renderer 605 is typically configured to perform field-of-view shifting or projection of the received video image based on depth information. This typically includes techniques such as pixel shifting (changing the position of pixels to reflect appropriate imbalances corresponding to parallax changes), occlusion removal (typically based on filling from other images), and combinations of pixels from different images, as is known to those skilled in the art.

[0083] Many algorithms and approaches are known for image synthesis, and it will be understood that one of the appropriate approaches will be used by the renderer 605.

[0084] The image composite device then generates field-of-view images / videos for the scene. Furthermore, as the user moves around within the scene, the field of view of the scene is continuously updated to reflect the changes in the viewing pose as the viewing pose dynamically changes. For static scenes, the same source field-of-view image is used to generate the output field-of-view image, but for video applications, different source images are used to generate different field-of-view images; for example, a new set of source images and depth are received for each output image. Thus, the processing is frame-based.

[0085] Renderer 605 is configured to generate views of the scene from different angles in response to lateral movement of the viewer's pose. When the viewer's pose changes to a different direction / orientation, renderer 605 is configured to generate views of 3D scene objects from different angles. Therefore, as the viewer's pose changes, objects in the scene may be perceived as static and having a fixed orientation within the scene. The viewer can effectively move and see objects from different directions.

[0086] The field-of-view synthesis circuit 205 is implemented in any suitable manner, including one or more integrated circuits, such as an application-specific integrated circuit (ASIC). In some embodiments, these receivers are implemented as one or more programmed processing units, such as firmware or software running on a suitable processor, such as a central processing unit, a digital signal processing unit, or a microcontroller. In such embodiments, it will be understood that the processing units include onboard or external memory, clock drive circuits, interface circuits, user interface circuits, and so on. These circuits are further implemented as integrated circuits and / or discrete electronic circuits as part of the processing unit.

[0087] As mentioned above, the problem with field-of-view synthesis is that the quality degrades as the viewing pose from which the field of view is synthesized becomes increasingly different from the capture pose of the video data of the scene from which it is provided. In fact, if the viewing pose is too far removed from the capture pose region, the resulting image will be unacceptable due to significant artifacts and errors.

[0088] The video rendering device also includes store 615 for storing 3D mesh models of at least a portion of a real-world scene.

[0089] A mesh model provides a three-dimensional description of at least a portion of a scene. A mesh model consists of a set of vertices interconnected by edges that generate faces. A mesh model provides a three-dimensional representation of elements in a scene, for example, by a number of triangular or rectangular faces. Typically, a mesh is described, for example, by the three-dimensional positions of vertices.

[0090] In many embodiments, the mesh model further includes texture data and texture information, because the mesh is provided to indicate the texture for the faces of the mesh. In many embodiments, the 3D mesh model includes at least one pixel map having pixel values ​​linked to the vertices of the 3D mesh of the 3D mesh model.

[0091] A mesh model of a real-world scene provides an accurate yet practical representation of the scene's three-dimensional information, which is used in video rendering devices to provide improved image data for viewing poses that differ only by a large angle from the capture pose region.

[0092] In many embodiments, the mesh model provides a static representation of the scene, while in many embodiments, the video signal provides a dynamic (typically real-time) representation of the scene.

[0093] For example, the scene is a football pitch, i.e., a stadium, and the model is generated to represent the permanent parts of the scene, such as the pitch, goals, lines, and stands. The provided video data is a capture of a specific match and includes dynamic elements such as players, coaches, and spectators.

[0094] The renderer 605 includes a first circuit 607 configured to determine image data for at least a portion of the output image by projecting the received video data onto the viewing pose. The first circuit 607 is thus configured to generate image data for the viewpoint of the current viewing pose from the received video data. The first circuit 607 applies any appropriate view-shifting and reprojection process to generate image data for the viewport of the viewing pose, specifically generating a complete or partial intermediate image corresponding to the current viewport (which is the viewport for the current viewing pose). The projection / view-shifting is from the capture pose of the video data, specifically a projection from the capture pose of one or more capture cameras onto the current viewing pose. Any appropriate approach, including techniques such as parallax shifting and occlusion removal, is used as described above.

[0095] The renderer 605 further includes a second circuit 609 configured to determine a second image data for an output viewport for at least a first region in response to a three-dimensional mesh model. The second circuit 609 is thus configured to generate image data for the viewport for the current viewing pose from the stored mesh model, typically taking texture information into consideration. The second circuit 609 applies any suitable approach to generating image data from the mesh model for a given viewing pose, including using techniques to map vertices to image positions in the output image according to the viewer's pose, and filling in ranges based on vertex positions and textures, etc. Specifically, the second circuit 609 generates a second intermediate image corresponding to the viewport for the current viewing pose. This second intermediate image is a partial image and contains image data for only one or more regions of the viewport.

[0096] Many different approaches, algorithms, and techniques are known for synthesizing image data from 3D data, including captured image data and data from 3D mesh models, and it will be understood that any suitable approach and algorithm can be used without diminishing the value of the present invention.

[0097] An example of a suitable field-of-view synthesis algorithm is, for example, “A review on image-based rendering” Yuan HANG,Guo-Ping ANG Virtual Reality & Intelligent Hardware,Volume 1,Issue 1,February 2019,Pages 39-54 https: / / doi.org / 10.3724 / SP.J.2096-5796.2018.0004 or “A Review of Image-Based Rendering Techniques” Shum; Kang Proceedings of SPIE - The International Society for Optical Engineering 4067:2-13, May 2000 DOI:10.1117 / 12.386541 Alternatively, for example, a Wikipedia article on 3D rendering. https: / / en.wikipedia.org / wiki / 3D_rendering It can be found in [location].

[0098] Renderer 605 thus generates image data for the current viewpoint in two distinct modes: one based on the received video data and another based on the stored mesh model.

[0099] The renderer 605 further includes a third circuit 611 configured to generate an output image that includes a first image and a second image. Specifically, with respect to at least a first region, the output image is generated to include a second image generated from a mesh model, and with respect to at least a portion of the output image outside the first region, the output image is generated to include a first image generated from a video signal.

[0100] In many scenarios, the output image is generated to include a first image file for all ranges where the resulting image quality is considered sufficiently high, and a second image file for ranges where the image quality is not considered sufficiently high.

[0101] The renderer 605 includes a fourth circuit 613 configured to determine one or more regions of the output image in which a second image data should be used, i.e., where the image data generated from the mesh model rather than from the video data should be included in the output image. The fourth circuit 613 is configured to determine a first such region in response to the deviation of the viewing pose from the capture pose region. Thus, the renderer 605 is configured to determine a region of the output image in which video-based image data is replaced by model-based image data, the region of which depends on the viewing pose and how much it differs from the capture pose region.

[0102] In some embodiments, the fourth circuit 613 is configured to determine a first region based on the difference between the viewing pose and the capture pose region. For example, if the distance between them is less than a given threshold (according to an appropriate distance measure), no region is defined, i.e., the entire output image is generated from the received video data. However, if this distance is greater than the threshold, the fourth circuit 613 determines a region that is likely to be of insufficient quality and controls the second circuit 609 to use second image data for this region. This region is determined, for example, based on the direction of change (typically in six DoF spaces).

[0103] For example, a video rendering device is configured to model a scene using a graphics package, and the graphics model is rendered to the viewport after a composite image is generated by the capture, so that this data is replaced in one or more regions by a model generated when the difference between the viewing pose and the capture pose region is too large.

[0104] As a concrete example, the fourth circuit 613 is configured to take into account the horizontal angular orientation of the viewing pose (reflecting the viewer rotating their head). As long as the viewing pose reflects a horizontal angular rotation that is less than a given threshold angle, the output image corresponding to the viewport of the viewing pose is generated based exclusively on the video data. However, if the viewing pose exhibits an angular rotation exceeding this threshold, the fourth circuit 613 determines that there is a left or right region of the image that will be occupied by the second image data. Whether this region is on the left or right side of the output image depends on the direction of rotation indicated by the viewing pose (i.e., whether the viewer rotates their head to the left or to the right) and the size of the region, which depends on how large the angular rotation is. Figures 7 and 8 illustrate an example of how this approach improves the images in Figures 4 and 5.

[0105] If the viewing pose moves too far from the capture pose area, the image quality of the composited field of view will degrade. In this case, the user experience is typically significantly improved, instead of low-quality or blurry data, such as data generated by evaluating a static graphics model of the scene. This provides the viewer with improved spatial content regarding their presence in the scene.

[0106] It should be noted that in typical practical systems, it is desirable to use a capture camera with a limited field of view, because this allows for the capture of more distant objects at higher resolution relative to the given sensor resolution. For example, achieving the same resolution using a 180-degree wide-angle lens would require a sensor with extremely high resolution, which is not always practical. This is because such sensors are more expensive in terms of camera and processing hardware, and they place higher resource demands on processing and communication.

[0107] As described above, in some embodiments, the video rendering device determines whether a region should be included in which model-based image data is used, specifically based on the distance between the viewing pose and the capture pose region. In some embodiments, the determination of the region based on the deviation between the viewing pose and the capture pose region is based on considering the effect of the deviation on the quality of the image data that can be synthesized for the viewing pose using the video data.

[0108] In some embodiments, the first circuit 607 generates an intermediate image based on the projection of received video data from a suitable capture pose onto a viewing pose.

[0109] The fourth circuit 613 then proceeds to evaluate the resulting intermediate image and, in particular, determine quality measures for different parts / ranges / regions of the image. The quality measures are determined, for example, based on the algorithm or process used to generate the image data. For example, image data that can be generated by disparity shift is assigned a high quality value, and this value is further graded depending on how large the shift is (for example, in the case of a remote background, the disparity shift is zero, so it is not sensitive to errors and noise in disparity estimation). Image data generated by extrapolation to occluded ranges from other image data is assigned a lower quality value, and this value is further graded depending on how much data extrapolation is required, the degree of texture variation in adjacent ranges, etc.

[0110] Next, the fourth circuit 613 evaluates the determined quality measures to determine one or more areas where the quality does not meet the quality criteria. A simple criterion is simply determining an area as a range where the quality criterion is lower than a threshold. More complex criteria include, for example, requirements for the minimum size or shape of the area.

[0111] The second circuit 609 then proceeds to generate an output image as a combination of video-based (composite) image data from the intermediate image and model-based image data. For example, the output image is generated by overwriting the image data of the intermediate video-based image with model-based image data in areas determined by the fourth circuit 613 to have insufficient image quality.

[0112] It will generally be understood that multiple different approaches are used to evaluate quality.

[0113] For example, depth quality is determined for different reasons, and the areas where model data is used are determined based on depth quality, specifically, areas of images generated using depth data that is considered to have a quality below a certain threshold.

[0114] To explicitly determine depth data, a reprojection error can be calculated (on the encoder or decoder side). This means that a field of view from image data, particularly a multi-field-of-view dataset, is reprojected (using depth) from a set of multiple fields of view to other known fields of view. Next, a color difference measure (per pixel or averaged over a region) can be used to indicate quality. Occlusion / deocclusion, though undesirable, affects this error calculation. This is avoided only by accumulating the error in the metric when the absolute difference between the pixel depth and the warped depth is below a threshold. Such a process is used, for example, to identify depth data that is not considered sufficiently reliable. When generating new images for any desired viewpoint, regions generated as a result of using such unreliable depth data are identified and overwritten by image data generated from the model.

[0115] In some cases, a small overall warp error is not sufficient indication of rendering quality for any new viewpoint. For example, when any new viewpoint is close to the original capture viewpoint, such as near the center of the viewing area, rendering quality will typically be relatively high, even if the depth quality of the depth data used is relatively low. Thus, the area is determined by considering depth quality and identifying the area resulting from low-quality depth data, but it also depends on other parameters, such as how large the shift is (and specifically, the distance between the viewpoint from which the image is generated and the capture pose area defined for that image data).

[0116] Another way to determine rendering quality for an arbitrary viewpoint is to compare the image feature statistics of the composite image for that viewpoint with the image feature statistics of one or more reference images. A reasonable statistic is, for example, curvature. Curvature can be calculated directly for one of the color channels or during addition using a local filter window. Alternatively, edge / contour detection can be used first, after which the curvature statistic can be calculated. The statistics can be calculated for the entire region of a given area in the composite field of view. This region can then be warped to one or more reference fields of view and compared with the statistics found in the region there. Because a (larger) area is used, the evaluation relies less on strict pixel correspondence. Instead of physically meaningful features such as curvature, deep neural networks can be used to calculate field-invariant quality features based on multiple reference fields of view. By applying such an approach and evaluating the region, it becomes possible to determine low-quality regions.

[0117] In some cases, so-called "unreferenced" metric is used to evaluate the quality of a synthesized field of view without any reference point. A neural network that predicts image quality is typically trained.

[0118] Such quality speed is determined without explicitly determining the deviation or difference between the viewing pose and the capture pose area (i.e., such determination is indirect in quality measurements that reflect whether the viewing pose deviates from the capture pose area).

[0119] As described above, video rendering devices store the mesh model of a scene, and typically also store a pixel map that has pixel values ​​linked to the vertices of the 3D mesh model. Specifically, a pixel map is a map that shows visual properties (intensity, color, texture) using mappings that link the mesh to parts of the pixel map that reflect local visual properties. Specifically, a pixel map can be a texture map, and the scene model can be a mesh with a texture model and representation added.

[0120] In some embodiments, the server 103 is configured to transmit model information to a video rendering device, and the first receiver 601 is configured to receive model data from the server 103. In some embodiments, the model data is combined with video data to form a single data stream, and the first receiver 601 is configured to store the data locally upon receipt. In some embodiments, the model data is received independently of the video data, for example, at a different time and / or from a different source.

[0121] In some embodiments, the video rendering device is configured to generate models locally, specifically to generate models from received video data. The video rendering device specifically includes a model generator 617 configured to generate a three-dimensional mesh model in response to captured video data.

[0122] The model generator 617 is configured to generate a model by combining and adapting certain predetermined information (e.g., a goal), such as the expectation that the scene is a room containing certain objects, and by combining and adapting these parameters. For example, the texture and dimensions of the room are determined based on the received video data, and the positions of certain objects in the room are determined based on the video data.

[0123] In some embodiments, a (simple) graphics model is inferred from the received multi-viewpoint video. For example, flat surfaces such as floors, ceilings, and walls can be detected and converted into graphics. Accompanying textures can optionally be extracted from the video data. Such inference does not need to be derived frame by frame, but can be accumulated and improved over time. When presented / rendered to the viewer, such relatively simple visual elements, while lacking detail, do not evoke much interest because they are not compared to any other image or images that are distorted, thus providing a better experience. They keep the viewer immersed and navigable (VR) without causing confusion.

[0124] In some embodiments, the model generator is configured to use object detection techniques to recognize objects or people present in the scene, which are then represented by existing graphical models or avatars. The pose of the object or body can be optionally determined and applied to the graphical representation.

[0125] Various techniques and approaches are known for detecting the characteristics of objects and scenes, and it will be understood that any suitable approach may be used without diminishing the value of the present invention.

[0126] In some embodiments, the mesh model is provided from a remote source, which is specifically server 103. In such cases, server 103 uses, for example, one of the approaches described above.

[0127] In some embodiments, the mesh model is pre-generated and represents a static part of the scene, as described above. For example, prior to capturing an event (such as a football match), a dedicated capture of the static part of a second common network element 707 is performed. For example, the camera is moved around the scene to provide images for developing a more accurate mesh model. Model development is further based on input from a dedicated 3D scanner and / or manual adaptation of the model. Such an approach is more cumbersome but provides a more accurate model. It is particularly useful in the case of events where the same model can be reused for many users and / or events. For example, a lot of effort is put into developing an accurate model of a football stadium, which can be reused for millions of viewers and many matches / events.

[0128] In some embodiments, the renderer 605 is configured to adapt the processing and / or data of the video database in response to the model processing and / or data. Alternatively, or in addition to that, the renderer 605 is configured to adapt the model processing and / or data in response to the processing and / or data of the video database.

[0129] For example, a mesh model defines the components of the goal, such as the goalposts and the crossbar. Video data contains data for the portion of the goal visible from the current viewing pose, which is complemented by the mesh model providing data for the rest of the goal. However, the generated image data is adapted so that different data match more closely. For example, part of the crossbar is generated from video data, and part of the crossbar is generated from the mesh model. In such an example, the data is adapted to provide a better interface between these parts. For example, the data is adapted so that the crossbar forms a linear object in the generated output image. This is done, for example, by shifting the image data for the crossbar generated from one source so that it matches and has the same orientation as image data from another source for the crossbar. Renderer 605 is configured to adapt model-based image data to match received video-based image data, to match received video-based image data to model-based image data, or to adapt them so that they match each other.

[0130] In some embodiments, this adaptation is directly based on the generated image data, while in other embodiments, it is directly based on mesh model data using an appropriate approach. Similarly, in some embodiments, the video rendering device is configured to adapt the mesh model in response to the generated video-based image data. For example, instead of adapting the model-based image data to match the video-based image data, the video rendering device may modify the model, for example, by moving some vertices until model-based image data that matches the video-based image data is consequently generated.

[0131] Specifically, in some embodiments, the renderer 605 is configured to adapt the generated model-based image data in response to the captured video data. For example, the colors from the model-based image may deviate from the actual captured colors. This may be due to (dynamic) conditions such as lighting or shading conditions or limitations in the accuracy of the model. Therefore, the renderer 605 modifies the colors to (closer to) match the colors of the captured data.

[0132] As an example of adapting a model-based image, the color distribution may be sampled across the entire image range for both intermediate images, i.e., the video-based intermediate image and the model-based intermediate image. Consequently, a single color offset that minimizes the difference in the color distribution is adapted to the model-based image. An improvement is to adapt multiple color offsets linked to components or clusters in the color distribution. Another improvement is to perform both sampling of the distribution and adapting the offset to specific spatial visual elements (e.g., surfaces).

[0133] In some embodiments, the renderer 605 is configured to adapt the generated video-based image data in response to a three-dimensional mesh model.

[0134] For example, the colors of the generated video-based image are modified to more closely match those recorded by the mesh model, or the video-based image is rotated to more closely match the lines resulting from the mesh model.

[0135] In some embodiments, the renderer 605 is configured to adapt the generated video-based image data in response to the generated model-based image data.

[0136] For example, the orientation of linear image structures in model-based image data can be used to correct distortions of the same type of structure in video-based image data. Specifically, this can be done using filtering operations that utilize knowledge about the orientation and position of lines detected in model-based images.

[0137] In some embodiments, the renderer 605 is configured to adapt the generated model-based image data in response to the generated video-based image data.

[0138] For example, the previously provided example of applying model-based image color can also be used to directly modify stored colors (e.g., texture maps) for the model, thereby enabling corrections to be applied for future images / frames.

[0139] In some embodiments, the renderer 605 is configured to adapt a 3D mesh model in response to the generated video-based image data.

[0140] For example, the position of the light source used to illuminate the model can be modified to match the lighting conditions in a stadium (but perhaps without using knowledge of the light source's position, as it may not be available). As another example, the vertex positions are adapted to produce a resulting model-based intermediate image that matches the video-based image data. For example, different model-based images are generated for slightly shifted positions of vertices near the transition, and the image that best matches the video-based image (e.g., aligning the straight lines crossing the edges most closely) is selected. The vertex positions in the mesh model are then modified to match the positions for the selected image.

[0141] In some embodiments, the second circuit 609 is configured to vary the level of detail for the first region in response to the deviation of the viewing pose to the capture pose region. In particular, this level of detail is reduced as the difference between the viewing pose and the capture pose region increases. The level of detail is reflected, for example, by the number of objects, and the features of the model are included in the generated image data.

[0142] In some embodiments, intermediate images are gradually blended with each other.

[0143] In some embodiments, the first receiver 601 is configured to receive further captured video data of the scene for a second capture pause region. For example, as illustrated in Figure 9, the scene is captured by two different camera rigs 901, 903 located at different positions.

[0144] In such embodiments, the video rendering device applies a similar approach to both capture pose regions, and in particular, the first circuit 607 is configured to determine third image data for the viewport output image of the current viewing pose based on video data for the second capture pose. The output image is then generated taking into account the first image data and the second image data. For example, the image data is selected based on which of the two—those derived from the first capture pose and those derived from the second capture pose—allows for the best possible synthesis.

[0145] In some embodiments, the second circuit 609 simply selects one of the sources on an image-by-image basis (or for a group of images). However, in other embodiments, this selection is made individually for different regions, or even for each individual pixel.

[0146] For example, the output image is generated from video data from the nearest capture pose region, unless this would result in deocclusion. For these regions, the image data is instead generated from video data from the furthest capture pose region, unless this would result in deocclusion for the pixels in that region.

[0147] In this approach, the fourth circuit 613 is further configured to generate a first region of the output image, i.e., a region where the output image is dense based on a mesh model, depending on the consideration of viewing poses for both the first and second capture pose regions.

[0148] As an example of low complexity, mesh model-based data can be used for all ranges where the current viewing pose is deocclusion with respect to both capture pose regions.

[0149] In some embodiments, scene capture may be from two or more distinct regions, and video data linked to two different capture pose regions may be provided. For a given viewing pose, the video rendering device considers deviations or differences to multiple different capture pose regions in order to determine the range of images that can or should be generated based on mesh model data.

[0150] The following may be offered, namely, A first receiver (601) configured to receive captured video data of a real-world scene, wherein the video data is linked to a capture pause region, and the first receiver (601) A store (615) configured to store at least a portion of the 3D mesh models of a real-world scene, A second receiver (603) configured to receive viewing pauses, A device is provided comprising a renderer (605) configured to generate an output image for a viewport in relation to a viewing pose, wherein the renderer (605) A first circuit (607) is configured to generate first image data for a viewport for at least a portion of the output image by projecting captured video data onto a viewing pose, A second circuit (609) configured to determine a second image data for an output viewport for at least a first region of the output image in response to a 3D mesh model, A third circuit (611) configured to generate an output image, including at least a portion of the first image data and the second image data for the first region. It is equipped with.

[0151] This device, A fourth circuit (613) configured to determine the first region in response to an image quality measure for the first image data for the first region, A fourth circuit (613) configured to determine an intermediate image containing a first image data, and to determine a first region as a region where the quality of the image data in the intermediate image does not meet the quality criteria, and / or The system may include a fourth circuit (613) configured to determine a first region in response to a quality measure for the first image data.

[0152] This device and / or the fourth circuit may not determine the deviation and / or difference of the viewing pose relative to the capture pose area.

[0153] This approach provides a particularly engaging user experience in many embodiments. As an example, consider a football match captured by a camera rig at the center line and a second camera rig closer to the goal. The viewer assumes a viewing pose near the center line and is presented with a high-resolution image of the match. The user then decides to virtually move closer to the goal, and upon arriving at this destination, is provided with high-quality video of the match based on the camera rig positioned near the goal. However, in contrast to the conventional approach of teleportation between multiple locations, the user is provided with an experience of continuous change in location from the center line to the goal (for example, by emulating the user physically walking between these locations). However, since there may not be enough video data to accurately render the field of view from a location between the center line and the goal, video data is rendered from model data for at least a portion of the image. This provides an improved and more immersive experience compared to the conventional experience of simply teleporting the user from one location to another in many scenarios.

[0154] The approach described generates images for the viewing pose / viewport in this manner. The images are generated from two fundamentally different types of data, and are adaptively generated to include, specifically, regions generated from these different types of data: one region generated from captured video data of a real-world scene, and another region generated from 3D mesh model data for the real-world scene.

[0155] Specifically, this approach addresses the problem that, in many scenarios, the capture of real-world scenes is often incomplete. This approach makes it possible to generate improved output images / viewpoints of the scene and / or reduce the need for video capture of the real-world scene.

[0156] In contrast to the conventional approach where images for scene regions where captured video data is unavailable are generated by extrapolating available data, the approach described uses two fundamentally different representations of the scene and combines them in generating a single image. The first type is captured video data, and the second type is a 3D mesh model. Thus, both captured video data and 3D mesh model data are used. In particular, the mesh model data is used to complement the captured video data so that portions of the generated image where the captured video data provides no information can still be presented.

[0157] This approach adaptively combines two fundamentally different types of scene representations to provide improved image quality, and in particular, enables the generation of image data for scenes where the captured video data contains no information at all.

[0158] For example, the approach described allows for the generation of an image for a given viewpoint that includes a portion of a scene for which no captured video data exists at all, in which case the generated image will include features and even objects from the scene for which no captured data exists.

[0159] The approach described offers many advantageous effects.

[0160] In particular, it is possible to generate images that provide an improved field of view of real-world scene features for a wider range of viewing poses, and for a given capture, the scenario can be achieved. For example, for a given viewing pose, it becomes possible to display parts of the scene that would not be possible otherwise, including the presentation of objects that the captured video does not contain any data for. This approach actually facilitates capture, which includes allowing fewer cameras to be used for capture while enabling a large portion (potentially all) of the scene to be seen in some form.

[0161] This approach also reduces the data rate required to transmit video data for a given scene. The capture is scaled down to a smaller portion of the scene because it is considered acceptable to replace parts of the scene with model data (for example, the playing area of ​​a football pitch is captured in real time by a video camera, while the top of the stadium is represented by static 3D mesh model data). Because video data is typically dynamic and real time, it actually tends to require a much, much higher data rate. The data rate required to represent, for example, the top of a stadium with 3D mesh model data is actually much lower compared to when it needs to be captured by a video camera and represented by video data.

[0162] This approach enables a significantly improved user experience, typically including increased freedom. The technical benefit is a reduction in the limitations on motion caused by incomplete video data capture (compared to, for example, D1).

[0163] This approach also, in many cases, simplifies implementation and / or reduces complexity, and / or alleviates the computational burden. For example, it achieves reduced encoding / decoding of video captures and simplified rendering (rendering based on 3D mesh models is typically less complex and more computationally intensive than rendering captured video).

[0164] The present invention can be implemented in any suitable form, including hardware, software, firmware, or any combination thereof. The present invention may optionally be implemented at least partially as computer software operating on one or more data processors and / or digital signal processors. Elements and components of embodiments of the present invention can be implemented physically, functionally, and logically in any suitable way. In fact, its functionality can be implemented as a single unit, as multiple units, or as part of other functional units. Therefore, the present invention can be implemented in a single unit, or physically and functionally distributed among different units, circuits, and processors.

[0165] In this application, whenever any of the terms “in response,” “based on,” “correspondingly,” and “as a function” are used, it should be understood that they refer to the term “in response / based on / correspondingly / as a function.” Any of these terms should be understood as a disclosure of any of the other terms, and the use of a single term alone should be understood as an abbreviated concept that includes the other options / terms.

[0166] Although the present invention has been described in relation to several embodiments, it is not intended to be limited to any particular form described herein. Rather, the scope of the present invention is limited only by the appended claims. Furthermore, while some features may appear to be described in relation to a particular embodiment, those skilled in the art will recognize that various features of the described embodiments can be combined in accordance with the present invention. The terms “having” and “equipped” in the claims do not preclude the existence of other elements or steps.

[0167] Furthermore, even if individually listed, multiple means, elements, circuits, or steps of a method may be carried out by a single circuit, unit, or processor, etc. Additionally, individual features may be included in multiple different claims, but these may be combined advantageously, and their inclusion in multiple different claims does not mean that the combination of features is impossible and / or unfavorable. Also, the inclusion of a feature in one category of claims does not mean that it is limited to that category; rather, it indicates that the feature is equally applicable to other claim categories as needed. Furthermore, the order of features in a claim does not imply a specific order in which the features must be carried out, and in particular, the order of individual steps in a method claim does not mean that those steps must be performed in that order. Rather, those steps can be performed in any suitable order. Additionally, a singular reference does not exclude the plural. Therefore, references to "one," "first," "second," etc., do not exclude the plural. Reference numerals in a claim are provided merely as examples for clarity and should not be construed in any way as limiting the scope of the claim.

[0168] Generally, examples of apparatus and methods are shown by the embodiments described below.

[0169] Embodiments: Claim 1. A first receiver (601) configured to receive captured video data of a real-world scene, wherein the video data is linked to a capture pause region, A store (615) configured to store at least a portion of the 3D mesh models of a real-world scene, A second receiver (603) configured to receive viewing pauses, A renderer (605) configured to generate output images for the viewport in relation to the viewing pose, The device is equipped with a renderer (605), A first circuit (607) is configured to generate first image data for a viewport for at least a portion of the output image by projecting captured video data onto a viewing pose, A second circuit (609) configured to determine a second image data for an output viewport for at least a first region of the output image in response to a 3D mesh model, A third circuit (611) is configured to generate an output image, including at least a portion of the first image data and the second image data for the first region. A fourth circuit (613) configured to determine a first region in response to the deviation of the viewing pose relative to the capture pose region, and A device equipped with the following features.

[0170] Claim 2. The renderer (605) is, An intermediate image containing the first image data is determined, The apparatus according to claim 1, configured to determine a first region as a region where the quality of the image data of the intermediate image does not meet the quality standard.

[0171] Claim 3. The apparatus according to claim 1 or 2, wherein a third circuit (609) is configured to determine a first region in response to the difference between a viewing pose region and a capture pose region.

[0172] Claim 4. The apparatus according to claim 3, wherein the difference is a difference in angle.

[0173] Claim 5. The apparatus according to any one of claims 1 to 4, wherein the renderer (605) is configured to adapt second image data in response to captured video data.

[0174] Claim 6. The apparatus according to any one of claims 1 to 5, wherein the renderer (605) is configured to adapt the first image data in response to three-dimensional mesh data.

[0175] Claim 7. The apparatus according to any one of claims 1 to 6, wherein the renderer (605) is configured to adapt second image data in response to first image data.

[0176] Claim 8. The apparatus according to any one of claims 1 to 7, wherein the renderer (605) is configured to adapt first image data in response to second image data.

[0177] Claim 9. The apparatus according to any one of claims 1 to 8, wherein the renderer (605) is configured to adapt a three-dimensional mesh model in response to a first image data.

[0178] Claim 10. The apparatus according to any one of claims 1 to 9, further comprising a model generator (617) for generating a three-dimensional mesh model in response to captured video data.

[0179] Claim 11. The apparatus according to any one of claims 1 to 10, wherein the first receiver (601) is configured to receive video data from a remote source (103) and to further receive a three-dimensional mesh model from the remote source (103).

[0180] Claim 12. The apparatus according to any one of claims 1 to 11, wherein the second circuit (609) is configured to vary the level of detail for the first region in response to a deviation of the viewing pose relative to the capture pose region.

[0181] Claim 13. The first receiver (601) is further configured to receive a second captured video data for a real-world scene, the second video data being linked to a second captured pause region. The first circuit (607) is further configured to determine third image data for at least a portion of the output image by projecting the captured second video data onto the viewing pose, The apparatus according to any one of claims 1 to 12, wherein the third circuit is configured to determine the first region in response to a deviation of the viewing pose relative to the second capture pose region.

[0182] Claim 14. A step of receiving captured video data for a real-world scene, wherein the video data is linked to a capture pose region, Steps include: storing a 3D mesh model of at least a portion of the real-world scene, Steps to receive viewing pause, The steps include generating an output image for the viewport based on the viewing pose, A method comprising the steps of generating an output image, The steps include generating a first image data for a viewport for at least a portion of the output image by projecting the captured video data onto the viewing pose, The steps include determining a second image data for an output viewport for at least a first region of the output image in response to a 3D mesh model, A step of generating an output image, which includes at least a portion of the first image data and the second image data for the first region. A step of determining a first region in response to the deviation of the viewing pose from the capture pose region, and A method having.

Claims

1. A first receiver that receives captured video data providing a dynamic representation of a real-world scene, wherein the video data is linked to a capture pause region, and the first receiver A store that stores a 3D mesh model that provides a static representation of at least a part of the aforementioned real-world scene, A second receiver that receives the viewing pause, A renderer that generates an output image for the viewport for the aforementioned viewing pose, A device comprising, the renderer, A first circuit generates first image data for a viewport for at least a portion of the output image by shifting the field of view of the captured video data from the capture pose to the viewing pose of the captured video data, A second circuit that generates a second image data for the viewport for at least a first region of the output image from the three-dimensional mesh model, A third circuit for generating the output image, including at least a portion of the first image data and the second image data for the first region, A fourth circuit that determines the first region according to the deviation of the viewing pose from the capture pose region, A device equipped with the following features.

2. The apparatus according to claim 1, wherein the renderer determines the first region as a region in which the quality of the first image data generated by the first circuit does not meet the quality standard.

3. The apparatus according to claim 1 or 2, wherein the third circuit determines the first region according to the difference between the viewing pose region and the capture pose region.

4. The apparatus according to claim 3, wherein the difference is a difference in angle.

5. The apparatus according to claim 1 or 2, wherein the renderer adapts the second image data according to the captured video data.

6. The apparatus according to claim 1 or 2, wherein the renderer adapts the first image data according to the three-dimensional mesh model.

7. The apparatus according to claim 1 or 2, wherein the renderer adapts the second image data according to the first image data.

8. The apparatus according to claim 1 or 2, wherein the renderer adapts the first image data according to the second image data.

9. The apparatus according to claim 1 or 2, wherein the renderer adapts the three-dimensional mesh model according to the first image data.

10. The apparatus according to claim 1 or 2, further comprising a model generator for generating the three-dimensional mesh model in accordance with the captured video data.

11. The apparatus according to claim 1 or 2, wherein the first receiver receives the video data from a remote source and further receives the three-dimensional mesh model from the remote source.

12. The apparatus according to claim 1 or 2, wherein the second circuit varies the level of detail for the first region in accordance with the deviation of the viewing pose relative to the capture pose region.

13. The first receiver further receives second video data captured for the real-world scene, the second video data being linked to a second capture pause region, The first circuit further determines third image data for at least a portion of the output image by projecting the captured second video data onto the viewing pose, The apparatus according to claim 1 or 2, wherein the third circuit determines the first region according to the deviation of the viewing pose from the second capture pose region.

14. A step of receiving captured video data that provides a dynamic representation of a real-world scene, wherein the video data is linked to a captured pose region. The steps include storing a three-dimensional mesh model that provides a static representation of at least a part of the aforementioned real-world scene, Steps to receive viewing pause, The step of generating an output image for the viewport for the aforementioned viewing pose. A method comprising the steps of generating the output image, A step of generating first image data for the viewport for at least a portion of the output image by shifting the field of view of the captured video data from the capture pose to the viewing pose of the captured video data, The steps include generating a second image data for the viewport for at least a first region of the output image from the three-dimensional mesh model, The steps of generating the output image include, The steps include determining the first region in accordance with the deviation of the viewing pose from the capture pose region and A method having

15. A computer program comprising computer program code, wherein the computer program code, when the computer program is run on a computer, performs all the steps of the method according to claim 14.

Images