Method and system for generating an image

Abstract

A method for generating an image of a subject from a viewpoint of a virtual camera is provided. The method includes obtaining a plurality of source images of the subject, and pose data of each of one or more cameras that capture the source images. Virtual camera pose data indicating the pose of the virtual camera relative to the subject is also obtained. Each source image is distorted based on a difference in the pose of the corresponding source camera and the pose of the virtual camera. A weight for each distorted image is determined based on a similarity between the pose of the corresponding source camera and the pose of the virtual camera. The distorted images are then blended together according to the weights to form an image of the subject from the viewpoint of the virtual camera. A system for implementing the method is also provided.

Description

Technical Field

[0001] The present disclosure relates to methods and systems for generating images of a subject from a viewpoint of a virtual camera. The present disclosure also relates to generating such images for display on a head mounted display (HMD). Background Art

[0002] In this day and age, video content is becoming increasingly immersive. This is due in part to the proliferation of HMDs, through which viewers can view this content. HMDs enable viewers to view content that is partially (in the case of augmented reality) or completely (in the case of virtual reality) superimposed on their view of the world. In some cases, virtual content can be displayed to the viewer in a way that respects the physical environment in which it is located (so-called mixed reality).

[0003] In some applications, a three-dimensional reconstruction of a real-life event may be presented to the viewer. For example, the real-life event may correspond to a recording of a performer, such as playing a piece of music, presenting content, performing in a fictional series, etc. An example of such content is the "Joshua Bell VR Experience" available on PS4 (via PSVR).

[0004] One known technique for presenting a performer in virtual reality involves capturing a stereoscopic video of the performer and projecting that video as a texture onto a mesh in a 3D reconstruction of the environment. An example of this technique is described in PCT / GB2018 / 052671. This technique is often adequate where small head movements of the viewer are expected; however, for larger head movements, the limitations of the technique may become more apparent.

[0005] Another known technique for presenting a performer in virtual reality (VR) involves arranging multiple cameras around the performer using, for example, a rig. The video captured by the cameras is used to create a complete geometric model of the performer, which is then appropriately displayed to the viewer based on the viewer's orientation relative to the model. It should be appreciated that capturing VR content in this manner is a time-consuming and expensive process. Furthermore, VR content generated by such a capture process often looks lifeless if soft edges (such as hair) and complex surface materials are not accurately represented.

[0006] Light field rendering is another technique that can be used to represent objects in 3D. However, light field rendering tends to be computationally expensive and is usually limited to representing static objects.

[0007] Generally, there are problems in the art regarding rendering objects (static and / or dynamic) in 3D without requiring excessive amounts of data or computational processing. It will be appreciated that if such rendering is performed on a video game console, the available computing power and resources may be somewhat limited. Furthermore, if the images are to be displayed on an HMD, any delay between the movement of the viewer's head and the rendering of the object from the new viewpoint may be uncomfortable for the viewer. The present invention seeks to address or at least alleviate some of these problems. Summary of the invention

[0008] According to a first aspect disclosed herein, a method for generating an image of a subject displayed on a head mounted display (HMD) is provided. The method comprises: obtaining a plurality of source images of a subject in a scene, each source image being captured from a different respective viewpoint; obtaining camera pose data indicating a pose of a camera relative to the subject for each source image; obtaining virtual camera pose data indicating a pose of a virtual camera relative to the subject; generating respective distorted images for at least some of the source images, each distorted image being generated based on a difference between a pose of the camera capturing the source image and a pose of the virtual camera; and determining a weight associated with each distorted image based on a similarity between a pose of the camera capturing the distorted image and a pose of the virtual camera, and combining the distorted images according to the determined weights so as to generate an image of the subject from the viewpoint of the virtual camera, wherein determining a weight associated with each distorted image comprises: determining an intersection point between a plane defined by at least three cameras and a line connecting the virtual camera to the subject, and determining the weight associated with the distorted image based on a distance between the camera capturing the distorted image and the intersection point.

[0009] According to a second aspect disclosed herein, a system for generating an image of an object is provided. The system comprises: a display element operable to display an image of an object to a viewer; a posture detector operable to detect the posture of the viewer's head; an image processor operable to acquire a plurality of source images of the object, and camera posture data indicating the posture of a camera that captured each source image; wherein the image processor is configured to receive input from the posture detector, and in response to the input, distort at least some of the source images based on the difference between the posture of one or more cameras associated with the source images and the reconstructed current posture of the viewer's head relative to the object; and an image combiner configured to distort at least some of the source images based on the difference between the posture of the camera associated with each distorted image and the posture of the viewer's head relative to the reconstructed image. and determining a weight associated with the distorted image based on a similarity between the two images; and wherein the image combiner is configured to combine the distorted images together according to the associated weights to form an image of the object for display at the display element, wherein the posture of the viewer's head corresponds to a posture of a virtual camera for viewing the object, and wherein the posture of one or more of the cameras that capture the source images corresponds to a camera array plane; wherein the image combiner is configured to determine an intersection point between a vector connecting the virtual camera to the object and the camera array plane; and wherein the image combiner is configured to determine the weight associated with the distorted image based on a distance between the camera that captured the distorted image and the intersection point. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] To aid in understanding the present disclosure and to show how embodiments may be implemented, reference is made to the accompanying drawings by way of example, in which:

[0011] Figure 1 An example of a head mounted display device is schematically shown;

[0012] Figure 2 An example of a method for reconstructing an image of an object from a virtual camera's viewpoint is shown;

[0013] Figure 3 An example of a subject and an arrangement of one or more cameras for capturing a source image of the subject are shown;

[0014] Figure 4 shows an example of a stereo source image captured by one or more cameras located at different positions relative to a subject;

[0015] Figure 5 An example of a depth map generated for each corresponding stereo source image is shown;

[0016] Fig. 6Aand 6B An example of a subdivision process for generating a mesh for a given source image and corresponding depth map is shown;

[0017] Figure 7 shows an example of a trimming step in which portions of a generated mesh may be trimmed from the mesh;

[0018] Fig. 8A and 8B schematically illustrates an example of a portion of a mesh that is reprojected based on a pose difference between poses of a corresponding source camera and a virtual camera;

[0019] Fig. 9 An example of a parallax mapping technique for mapping regions of a source image to regions of a corresponding reprojection mesh is shown;

[0020] Fig. 10A and 10B An example of a method for determining a weight associated with each distorted image is schematically shown;

[0021] Fig.11 shows an example of a distorted image generated according to the present disclosure and a final image obtained by blending these images together; and

[0022] Fig.12 An example of a system for generating an image of an object from the perspective of a virtual camera is shown. DETAILED DESCRIPTION

[0023] Reference now Figure 1 , the user 10 wears the HMD 20 (as an example of a general head mounted device). In this example, the HMD includes a frame 40 composed of a rear band and a top band, and a display portion 50.

[0024] Figure 1 The HMD completely (or at least substantially completely) blocks the user's view of the surrounding environment. The user can only see a pair of images displayed in the HMD.

[0025] The HMD has associated earphone audio transducers or earphones 60 that fit over the user's left and right ears 70. The earphones 60 play back audio signals provided from an external source, which may be the same video signal source that provides the video signal for display to the user's eyes.

[0026] The combination of the fact that the user can only see content displayed by the HMD, and can only hear content provided through the headphones, subject to the noise blocking or active cancellation characteristics of the headphones and associated electronics, means that the HMD can be considered a so-called "full immersion" HMD. Note, however, that in some embodiments the HMD is not a full immersion HMD, and may provide the user with at least some facilities for seeing and / or hearing the user's surroundings. This may be done by providing some degree of transparency or partial transparency in the display arrangement, and / or projecting an external view through the display of the HMD (captured using, for example, a camera mounted on the HMD), and / or by allowing transmission of ambient sound through the headphones and / or by providing a microphone to generate input sound signals (for transmission to the headphones) based on the ambient sound.

[0027] In use, a forward-facing camera (not shown) may capture images of the front of the HMD. An antenna (not shown) may provide communication facilities, or may simply be arranged as a directional antenna to allow detection of the direction of nearby Bluetooth transmitters.

[0028] In operation, a video signal is provided by the HMD for display. This may be provided by an external video signal source 80 such as a video game console or a data processing device such as a personal computer, in which case the signal may be sent to the HMD via a wired or wireless connection. Examples of suitable wireless connections include Connection. Audio signals for the headset 60 may be transmitted via the same connection. Similarly, any control signals transmitted from the HMD to the video (audio) signal source may be transmitted via the same connection.

[0029] In addition, a power source (including one or more batteries and / or connectable to a power outlet) can be connected to the HMD via a cable. Note that the power source and video signal source 80 can be separate units, or can be implemented as the same physical unit. There can be separate cables for power and video (or even for audio) signal supply, or these cables can be combined on a single cable for transmission (for example, using separate conductors, such as in a USB cable, or in a manner similar to "Power over Ethernet", in which data is transmitted as a balanced signal and power is transmitted as direct current through the same physical set of wires). Video and / or audio signals can be transmitted via, for example, optical cables. In other embodiments, at least part of the functions associated with generating images and / or audio signals for presentation to the user can be performed by circuits and / or processing that constitute part of the HMD itself. The power source can be provided as part of the HMD itself.

[0030] therefore, Figure 1The arrangement provides an example of a head-mounted display system that includes a frame mounted on a viewer's head, the frame defining one or two eye display locations (which, in use, are located in front of the viewer's corresponding eyes) and a display element mounted relative to each eye display location, the display element providing a video displayed virtual image of a video signal from a video signal source to the viewer's eyes.

[0031] Figure 1 Only one example of an HMD is shown. Other formats are possible: for example, the HMD may use a frame more similar to that associated with traditional glasses, i.e., substantially horizontal legs extending from the display portion back to behind the tops of the user's ears, perhaps curling up behind the ears. In other (not fully immersive) examples, the user's view of the external environment may not actually be completely obstructed; the displayed image may be arranged to be superimposed (from the user's perspective) on the external environment.

[0032] Figure 1 The HMD described in the specification can be used to display immersive video to a viewer. In this article, the term immersive video is used to refer to video content suitable for viewing in virtual, augmented and / or mixed reality. Typically, the video will correspond at least in part to a video of a real scene, including one or more real objects. Immersive video may be different from game content because images of objects in the real world have been used to present three-dimensional images of these objects to the viewer. However, immersive video still allows the viewer to interact with objects represented in the virtual, augmented or mixed reality environment.

[0033] As previously mentioned, the immersive video may correspond to a video of a performance. Examples of immersive video content include, for example, "Joshua Bell VR Experience", "Tom Grennan VR", etc., available on PS4. For contextual purposes, the embodiments described herein will be described primarily with respect to the display of immersive video content in VR.

[0034] As mentioned previously, there are many techniques for generating immersive video content. These techniques include, for example, capturing a scene as volumetric video; capturing stereoscopic video of a scene and projecting it onto a mesh in a 3D reconstruction of the scene; light field rendering, etc. However, typically these techniques are time consuming, expensive, and computationally intensive in terms of the resources required to render the video.

[0035] If a video game console is to be used to render the immersive video, the available computing power may be somewhat limited. Furthermore, if the video display at the HMD is not updated in sync with the viewer's head movements, any delays in the video rendering may cause discomfort or even nausea to the viewer. Therefore, there is a need in the art for a rendering process capable of generating immersive video content that is less susceptible to the drawbacks associated with the above-mentioned techniques.

[0036] Now, refer to Figure 2 A method of generating an image of a subject for display on an HMD according to the present invention is described.In this document, a "subject" corresponds to an object to be displayed as part of an immersive video.

[0037] In a first step S201, a plurality of source images of a subject in a scene are acquired. The subject may correspond to a static object or a dynamic object to be represented as part of an immersive video. Each source image is captured from a different viewpoint. That is, the pose of the camera of each source image relative to the subject is different from the poses of the cameras of other source images. Therefore, each source image corresponds to a different view of the subject. The term pose is used herein to refer to the position and / or orientation of the camera relative to the subject.

[0038] Each source image may correspond to a stereo image captured by a stereo camera. In some examples, multiple stereo cameras may be used to capture stereo images (e.g., one stereo camera per source image). That is, an array of stereo cameras may be arranged around the subject, and the source images may correspond to stereo images captured by the stereo cameras in the array. In additional or alternative examples, a single stereo camera may be used, wherein the stereo camera is moved to different positions around the object to obtain multiple source images of the subject from different respective viewpoints. The stereo images may correspond to color images, such as RGB or YUV images. In some examples, the source images may be compressed prior to the processing described herein.

[0039] In some examples, each stereo camera includes two image sensors, where each image sensor has one or more lenses associated therewith for focusing light onto the corresponding sensor. The images captured by the image sensors may correspond to a left image and a right image, respectively. The left image and the right image may be displayed at a left and right display element of the HMD, respectively (subsequent processing as described below).

[0040] In additional or alternative examples, at least some of the cameras may include monoscopic cameras (i.e., including one image sensor and one or more lenses for focusing light onto the sensor). These sensors may be used in conjunction with additional depth sensors (such as time of flight, structured light sensors), as described below.

[0041] In some examples, each source image may correspond to an image of a subject captured in front of a green curtain. Portions of the source image corresponding to the green curtain may be removed, thereby separating the subject from its environment. Using a green curtain enables high-quality soft edges of the subject to be captured, which can then be represented in 3D.

[0042] Figure 3 An example of a subject and the location of a stereo camera used to capture source images of the subject are shown. Figure 3 In , the subject corresponds to a stuffed lion toy 300 placed on and in front of a green curtain. It should be understood that this is just one example of a subject, and other subjects (such as performers) may also be captured in the same manner. Figure 3 , the subject and camera are shown in a top-down view.

[0043] exist Figure 3 As can be seen in FIG, the posture of the stereo camera relative to the subject corresponds to an array of stereo cameras arranged in an x-shape. That is, the stereo camera pair 302A, 302B is arranged on the left and right sides of the subject, respectively; another stereo camera 302C is located in the center of the subject behind the first pair of stereo cameras 302A, 302B; and the second pair of stereo cameras 302D, 302E is located behind the centrally positioned stereo camera, also on the left and right sides of the subject, respectively. In some examples, the x-shape can be simulated by moving the individual stereo cameras to respective positions forming the x-shape.

[0044] In additional or alternative examples, the stereo cameras may be arranged at different vertical heights. For example, the x-shape may also (or alternatively) correspond to a vertical arrangement of the cameras. This means that a first pair of stereo cameras may be located above a lower pair of stereo cameras, with one stereo camera occupying a central position (vertically) between the two pairs of stereo cameras. Such a camera arrangement may be said to be coplanar in the xy plane (z corresponds to depth). This will be referred to later. Fig. 10A and 10B An example of a stereo camera array arranged in this manner is described.

[0045] It should be understood that in other examples, the cameras need not be coplanar, with one or more cameras offset in the z direction. In some cases, a non-coplanar arrangement of cameras can improve the accuracy of photogrammetry used to calculate depth maps.

[0046] It should be understood that the relative separation of the stereo cameras will determine the amount of source image data that can be used to reconstruct the subject in 3D. By separating the cameras further so that the occluded areas come into view, better coverage of the subject can be obtained. The greater the coverage, the more subjects can be represented in 3D, and therefore the greater the freedom of movement of the viewer when viewing the reconstruction. However, if the cameras are too far apart, this may result in less image data being shared between the source images, which in turn may cause difficulties when trying to blend the source images together. Of course, the accuracy of the representation of the subject in 3D can be improved by capturing more images from different viewpoints. However, capturing more images will generate greater costs in terms of memory and processing power required to generate the 3D reconstruction, which is a well-known high cost, such as volumetric video. Therefore, the number of viewpoints can be an empirical trade-off between computing resources and the quality of the three-dimensional reconstruction.

[0047] The inventor discovered that Figure 3 The x-arrangement shown is sufficient for immersive video purposes, where relatively limited movement of the viewer relative to the subject is expected. This movement may be limited due to the nature of content being typically consumed, for example, from a stationary position (such as sitting) or from the physical environment in which the content is consumed (e.g., there may not be enough physical space for a viewer to attempt to walk behind a subject represented in VR). Thus, in immersive video, the subject may only need to be presented for a limited range of views. Additionally, by Figure 3 Source images captured by the arrangement shown will include overlapping areas, resulting in fewer "holes" when the source images are combined together.

[0048] It should be understood that arranging the stereo cameras in an x-shape is only one example of how the stereo cameras may be arranged relative to the subject. In other examples, alternative camera capture layouts may be used. Ultimately, the layout used and the number of images captured may depend on the subject being captured and the hardware that will be used to process those images. More generally, Figure 3 The x-shape shown in can be considered as an example of evenly distributing viewpoints within the viewer's intended motion region.

[0049] Figure 4 Examples of stereo images 402A, 402B, 402C, 402D, 402E captured by each of the stereo cameras are shown. Figure 4 In , the positioning of the stereo images corresponds to the positioning of the stereo cameras relative to the subject. Figure 4 The stereo images shown in are captured by an array of stereo cameras arranged in the x-shape described above.

[0050] Back to Figure 2In step S201, camera pose data indicating a pose of a camera relative to a subject in a scene is obtained for each source image. The camera pose data defines a position and / or orientation of the camera relative to the subject for each source image. The camera pose data may be defined as metadata stored in association with a source image.

[0051] The camera pose data may be acquired during capture of the source image. For example, the distance of the subject from the camera may be measured using, for example, a time-of-flight sensor, and changes in the orientation of the camera may be tracked using, for example, an accelerometer or gyroscope associated with the camera. In some examples, the relative pose of the camera may not change during the capture process (e.g., where an array of multiple cameras is used), and known photogrammetry techniques may be used to determine the pose of the camera relative to the subject. For example, a test subject (such as a chessboard) is used to determine the exterior of the camera.

[0052] In a second step S202, virtual camera pose data indicating a pose of the virtual camera relative to the subject is obtained. The virtual camera data may correspond to a pose of the HMD through which the reconstruction of the subject may be viewed. The pose of the HMD determines how the viewer is positioned and oriented relative to the reconstruction of the subject, and therefore corresponds to a virtual camera through which the subject may be viewed. It should be understood that for the pose of the virtual camera pose relative to the subject to be determined, it is not actually necessary to display the reconstruction of the subject at the HMD. The pose of the virtual camera may be determined at rendering time.

[0053] The virtual camera pose data may be obtained from a pose detector incorporated into the HMD. For example, the HMD may include an accelerometer and / or a gyroscope for tracking changes in the viewer's head pose. Alternatively, or in addition, one or more external cameras may be used to track the pose of the viewer's head, for example, by constellation tracking. The position of the subject in the reconstructed scene may be known, and used to determine the pose of the HMD of the viewer's head relative to the (virtual) pose of the subject in the reconstructed scene.

[0054] Back to Figure 2 In a third step S203, at least some of the source images are distorted. Each source image is distorted based on the difference between the pose of the camera that captured the source image and the reconstructed pose of the virtual camera relative to the subject. In other words, the degree of distortion of a given source image depends on the degree to which the view of the virtual camera corresponds to the view of the camera that captured the source image.

[0055] Generating distorted images

[0056] Now refer to Figures 4 to 11 Example methods for generating distorted images are described.

[0057] In some examples, generating the distorted image includes acquiring depth data associated with each source image. As previously described, each source image may correspond to a stereoscopic image of the subject, and thus the depth data of a given source image may be acquired by performing stereo matching on the left and right images constituting the source image. The depth data acquired for a given source image may be used to generate a depth map (i.e., a depth image) of the subject in the image. If the subject is positioned in front of a green curtain, pixels in the source image corresponding to the background may be removed before generating depth maps from these images. Optionally, areas in the depth map that exceed a certain depth threshold may be rejected as background areas. Each depth map may correspond to a depth image captured from the same viewpoint as the camera that captured the corresponding source image. In some examples, the depth map may be smoothed before a mesh is generated from the depth map.

[0058] Figure 5 Shown according to Figure 3 and 4 Examples of depth maps 502A, 502B, 502C, 502D, 502E generated from captured stereo images. Figure 5 In , each depth map is displayed at a position corresponding to the position of the stereo camera from which the stereo images used to generate the depth map were captured. Figure 5 There is a one-to-one correspondence between the depth map and the source image.

[0059] In an alternative example, the depth data for each source image may be captured separately from the capture of the source images. For example, for each source image, a LIDAR scan may be performed on the subject from a corresponding viewpoint. In such an example, the source images need not be stereoscopic. Other known techniques for acquiring depth data for corresponding viewpoints may be used.

[0060] The source images may be defined in terms of RGB pixels and, optionally, an alpha value representing the transparency of each pixel. For each source image, a transparent mask may be generated using the region of the image corresponding to the green curtain and the corresponding depth map (i.e., large regions of uniform depth may be identified as belonging to the background region). The distance from the edge of the transparent mask may be stored in a signed distance field (SDF) to allow the edge of the transparent mask to be easily expanded or contracted as needed. In examples where the source images include stereoscopic images, transparent masks may be generated for the left and right images that make up each stereoscopic image.

[0061] In some examples, the depth map of the subject may be slightly extended to exceed the alpha contour and fill any holes (i.e., areas where depth data is missing). The depth map for a given source image may be stored as multiple mipmaps, with each successive mipmap corresponding to a lower resolution depth map. The mipmap level may be predetermined based on image quality, i.e., the mipmap level may be a parameter that may be adjusted based on how noisy the data is. A depth texture map may be generated by storing the closest (closest to the camera) depth of, for example, 4 pixels of the above texture layer.

[0062] Generating the distorted image may also include generating a polygonal mesh (e.g., a triangular mesh) of the subject for each source image based on the source image and the depth data obtained for the source image. Each polygonal mesh may be generated by applying a two-dimensional mesh to the respective source image and determining a depth variation across each cell of the two-dimensional mesh for the source image. A subdivision density of the mesh may be determined based on the depth variation of each cell of the two-dimensional mesh applied to the corresponding source image. Fig. 6A and 6B An example of such subdivision processing is shown.

[0063] exist Fig. 6A In FIG. 6 , a two-dimensional grid 604 having triangular patches is shown aligned with a depth map 602A associated with a given source image. Figure 6B In FIG. 6 , the same grid 604 is shown aligned with a corresponding source image 602B, which in this case corresponds to one of the color images that make up the stereoscopic image. Fig. 6A and 6B In the , parts of the source image and the corresponding depth map are perfectly aligned. The x and y coordinates of the patch are set to be equidistant when viewed on the screen, and the z coordinate is determined by sampling the appropriate texture level of the corresponding depth map. The tessellation density of each triangle patch is determined by the depth variation of the patch vertices. This ensures that as much of the surface shape as possible is captured in the geometry.

[0064] After generating a polygonal mesh for a source image, at least some portions of the mesh may be pruned. The pruned may be performed based on determining that at least some portions of the mesh are outside a threshold boundary corresponding to an outline of a subject.

[0065] An example of this type of pruning is Figure 7 As shown, the top image 702 corresponds to the signed distance field representing the object, and a threshold boundary 704 is superimposed. The threshold boundary 704 can be defined based on the SDF, that is, the boundary between the "inside" and "outside" of the shape of the object defined by the SDF. Figure 76, the bottom-most image 706 corresponds to the source image with the triangular mesh superimposed thereon as shown in FIG6. Triangles may be removed from the mesh by defining a threshold distance to the border 704, and triangles above the threshold distance may be removed from the mesh. In one example, any triangle located more than 4 pixels outside the border 704 may be removed. In the bottom-most image 706, region 708 corresponds to the region from which any triangles of the mesh contained therein are removed.

[0066] After generating a polygonal mesh for each source image, a distorted image can be generated by reprojecting at least some portions of each mesh to be perpendicular to the direction the virtual camera is facing. That is, for each source image, a distorted image can be generated by reprojecting at least some portions of the corresponding mesh to face the virtual camera. Fig. 8A and 8B An example of this technique is shown.

[0067] exist Fig. 8A In FIG. 8 , a top view of a subdivided surface of a camera 802 and a mesh 804A representing a subject is shown. The position and orientation of the camera 802 relative to the subject corresponds to the position and orientation of the camera relative to the subject in one of the source images. Figure 8B In , the vertices of the mesh are shown as having been reprojected according to the position of the virtual camera (shown at position "x") corresponding to the position of the rendering camera. Figure 8B , the reprojected portion of the mesh is shown as reprojected portion 804B. Figure 8B As shown, the vertices of these portions have been reprojected so as to be parallel to the view plane (and perpendicular to the view direction of the virtual camera). For example, such reprojection may involve realigning at least some triangles in the polygonal mesh so that in screen space, the x and y values ​​of the triangle are preserved, but the z value is set to the value of the viewpoint closest to the three vertices, or, alternatively, to the median or average of the three vertices. Based on the pose of the camera of each source image and the pose of the virtual camera, the process of reprojecting portions of the mesh to face the virtual camera may be repeated for each mesh.

[0068] After reprojecting at least some portions of each mesh, a texture may be generated for application to the meshes. That is, generating a distorted image from a source image may include generating a texture for application to each reprojected mesh. The texture for a given mesh may be generated by parallax mapping the corresponding source image onto the mesh. A parallax shader may be used to perform this mapping. When mapped to the view of a virtual camera, the parallax shader corrects for changes in perspective of the source image. Each distorted image (for a given source image) may correspond to a rendering of a corresponding reprojected mesh to which a texture has been applied.

[0069] Fig. 9An example of a technique is shown in which color information obtained from a source image can be applied as a texture to a corresponding mesh, at least some portions of which have been reprojected (as described above). A mesh here is "corresponding" in the sense that it is generated from a source image. That is, each mesh can be generated using a different source image.

[0070] exist Fig. 9 On the left hand side of FIG. 9 , a portion of a depth map 902 and a corresponding portion of a source image (undistorted) 904A are shown. In this portion, an area 906A is shown, which corresponds to a texture area to which disparity mapping is to be applied. Fig. 9 The area 906 shown in FIG. 9 may correspond to a portion of the source image to be mapped onto a polygon (eg, a triangle) of the corresponding mesh. Fig. 9 The right hand side of shows the same texture region 906B within the distorted portion of the source image 904B after disparity mapping.

[0071] exist Fig. 9 In the middle area of ​​, the surface of the object 908 is shown relative to the source camera 910 that acquired the corresponding source image and depth map of the object. Fig. 9 The surface of the illustrated subject 908 is represented as a polygonal proxy geometry surface generated from the depth map.

[0072] The pose of the source camera corresponds to the pose of the camera that captured the source image in question. Fig. 9 In FIG. 1 , the pose of the virtual camera is also shown relative to the subject (indicated at position “x”). The pose of the virtual camera corresponds to the pose of the viewer relative to the subject.

[0073] exist Fig. 9 , a first ray 912 is shown starting from a polygon edge parallel to the VR image (corresponding to a reprojected polygon of the mesh) and projected to the surface edge of the depth map using ray marching. The ray is then traced back to find its position on the source texture. This process is repeated for multiple rays along polygon edges parallel to the VR image (as shown by the second ray 914). From this, it can be determined how to warp a given portion of the source image (i.e., the source texture) into the image space of the VR viewpoint. Once the appropriate warp is determined for a given portion of the source texture, that portion of the source texture can be applied to the corresponding polygon of the mesh.

[0074] Combining distorted images

[0075] Back to Figure 2, in a fourth step S204, a weight is determined for each distorted source image. The weight of a given source image is determined based on the similarity between the pose of the camera that captured the distorted image and the pose of the virtual camera (i.e., the degree of correspondence in the real and virtual camera views). This means that the greater the similarity between the poses of the source camera and the virtual camera, the greater the contribution of the distorted image in the final mixed image. In some examples, the weight of a given distorted image can be determined based on the proximity of the corresponding source camera to the virtual camera and the subject.

[0076] Fig. 10A An example of a method by which weights associated with a distorted image may be determined is shown. Fig. 10A In FIG, a virtual camera 1002 is shown at a given pose relative to a subject 1004. The line connecting the virtual camera to the subject is indicated by a dashed arrow 1006, which provides an indication of the distance of the subject in the z direction. The pose of the source camera relative to the subject is indicated at pose 1008. Fig. 10A and 10B , the source cameras are arranged in a coplanar arrangement.

[0077] Fig. 10A A diagram corresponding to a significant overhead view of the subject, source camera, and virtual camera. Fig. 10B The same source camera is shown in a landscape view. That is, Fig. 10B , the relative horizontal and vertical positions of the source cameras are shown (but not their relative distances from the subject).

[0078] The weights associated with the distorted image can be determined by computing the intersection of the line connecting the virtual camera to the subject and the camera array plane. Fig. 10A and 10B In , the intersection of the three rightmost cameras is marked as "x". The camera array plane can be defined as the plane connecting at least three adjacent cameras. For example, in Fig. 10A and Fig. 10B , the xy plane is shown partitioned into four slices centered about the central camera, each slice corresponding to a camera array plane defined by a different subset of three cameras.

[0079] The weight associated with a given distorted image may be determined based on the distance between the corresponding source camera and the intersection point (labeled "x") in a given camera array plane. For example, for the three rightmost cameras, the smaller the distance between the point labeled "x" and the position of that camera, the larger the weight associated with each camera may be. It should be understood that, optionally, contributions from increasingly distant cameras may also be used.

[0080] In some examples, determining the weight associated with each distorted image can include determining the signed barycentric coordinates of the intersection between a ray from the viewer to the subject and each triangle constituting the camera array plane. In a typical non-degenerate use case, two or three of the barycentric coordinates will be positive for one of the triangles and can be used as weights for matching views.

[0081] In general, the weight of a given distorted image can be determined based on the proximity of the corresponding source camera to the virtual camera and the subject. Fig. 10A In the example shown, for example, a lower weight may be assigned to a source image captured by the top left camera than to a source image captured by the top right camera. This is because the view of the top left camera corresponds less to the view of the virtual camera.

[0082] In some examples, the direction in which each source camera is pointing may be known, and this information may be compared to the vector connecting the virtual camera to the subject to determine how closely each source camera is aligned with the virtual camera. The positions of the source cameras and the virtual camera may also be used to determine how closely the source camera is aligned with the virtual camera. Likewise, the closer a given source camera is aligned with the virtual camera, the greater the weight assigned to the distorted image generated by that source camera.

[0083] It should be understood that although Fig. 10B The cameras shown in are arranged coplanarly, but other arrangements may be used and there may be different (and in fact multiple) camera array planes, where intersection points are determined for the cameras defining that plane or sub-plane, thereby determining the weights associated with the distorted images acquired from those cameras.

[0084] The above techniques can be extended to any arrangement of cameras where there is a central camera and N remote cameras are arranged symmetrically or asymmetrically around the central camera. For example, there can be up to N camera array planes (where N+1 equals the total number of cameras, with the "1" coming from the central camera), and the intersection of the ray connecting the virtual camera and the subject with one or more of those planes can be determined. This intersection can then be used to determine the weights associated with the images acquired from each camera using any of the above techniques.

[0085] More generally, any arrangement of cameras can be used where the triangulated irregular network coverage of the plane is determined using the origin of these cameras projected onto the xy plane. The final hybrid image can then be generated using the weighted contributions of at least three cameras that form a triangle that includes the intersection "x" with the virtual camera.

[0086] Back to Figure 2In a fifth step S205, the distorted images are combined according to the determined weights to generate an image of the subject from the viewpoint of the virtual camera. The generated image may correspond to a stereoscopic image to be output for display at the HMD. Optionally, fully transparent pixels of each distorted image may be discarded before mixing the distorted images.

[0087] In some examples, a set of distorted images may be generated for each set of left and right images captured by a stereo camera. The distorted images generated for the left images may be blended together to form an image for display at a left-eye display element of the HMD. Similarly, the distorted images generated for the right images may be blended together to form an image for display at a right-eye display element of the HMD.

[0088] Fig.11 Examples of distorted images 1102A, 1102B, 1102C, 1102D, and 1102E generated according to the above method are shown. Fig.11 The relative arrangement of the distorted images in corresponds to the relative arrangement of the stereo camera (or cameras) that captured the corresponding source images. Fig.11 , all distorted images correspond to a set of distorted images generated from the left (or right) image of the previously described stereoscopic image.

[0089] like Fig.11 As shown, there are varying degrees of holes (i.e., missing color data) in the distorted image. The degree of holes associated with a given distorted image generally depends on how closely the associated source image corresponds to the viewpoint of the virtual camera that is to represent the subject. Fig.11 In , the viewpoint of the virtual camera corresponds to the left side toward the subject, so the distorted images (e.g., images 1102A, 1102C, 1102D) acquired from the source camera at the corresponding position have fewer holes. In any case, once the distorted images are combined, there will be fewer holes in the composite image.

[0090] An example of an image generated by mixing the distorted images together according to the determined weights is Fig.11 The final image 1104 is shown in FIG. Fig.11 As shown, there are still some gaps in the final image 1104, which correspond to areas of the subject that cannot be filled by any distorted image. For example, a simple threshold-based filter can be used to fill these areas. Therefore, the final image actually displayed may correspond to a hybrid distorted image for which at least some hole filling has been performed.

[0091] In some examples, the texture space may be limited and only one image from the stereo pair (for each stereo pair) is used in the final blend. When these images are reprojected, the quality of the final image is often sufficient, especially if this tradeoff is only used for the outer cameras (e.g., all left cameras or all right cameras). This may be appropriate, for example, for situations where the final image is not displayed as a stereo image, but as a single image. Alternatively, or in addition, a stereo image is subsequently generated from the single image generated in this way.

[0092] The inventors have found that when only one set of images from the stereo pair is used to generate the final image, specular highlights and reflections are often still represented correctly. This is because the depth of the reflections is stored in the depth map, not in the reflecting surface. However, when both images from each stereo pair are used to generate the final image, the results are often improved.

[0093] Once the final image of the subject is generated, it can be displayed as part of the immersive video. The environment in which the subject is displayed can be generated separately from the subject itself. For example, the subject can be superimposed in front of a computer-generated environment. In some cases, it may be that a LIDAR scan of the environment (e.g., a studio or set) is performed and this is used to generate a reconstruction of that environment. This may be appropriate, for example, when the immersive video corresponds to a live performance or game taking place in a real-life environment. The subject can then be appropriately positioned in the reconstructed environment and the representation of the subject adjusted according to the above methods.

[0094] Although the above embodiments have been described with respect to generating images for display at an HMD, it should be appreciated that the same techniques may likewise be used to render objects in 3D for display on any type of display screen.

[0095] A computer readable medium may also be provided having computer executable instructions adapted to cause a computer to perform the above method. In some examples, the rendering method may be performed at, for example, a game console, and the final image may be displayed at a display of an HMD in communication with the game console. Alternatively, or in addition, at least some of the rendering may be performed at, for example, a server, and the final image may be provided to a display of an HMD in communication with the server, for example, via an Internet connection or via a game console in communication with the server. Alternatively, or in addition, if the HMD includes appropriate processing means, the HMD may perform at least part of the rendering method itself.

[0096] In some examples, rendering may be shared between two or more of a server, a game console, or an HMD, with the final image relayed from the game console to the HMD if necessary. Ultimately, any combination of servers, game consoles, and HMDs may be used to perform rendering, depending on the capabilities of their respective hardware and the nature of the subject being rendered.

[0097] Now refer to Fig.12 A display system for implementing the above method is described.

[0098] The display system 1200 includes a display element 1202 that is operable to display an image of an object to a viewer. The object may be a static (e.g., a stuffed toy) or a dynamic (e.g., a performer) object. The object corresponds to the subject described above. The display element may correspond to the image of the object described above. Figure 1 A display of the HMD.

[0099] The system also includes a pose detector 1204 that is operable to detect a pose of the viewer's head. The pose detector 1204 may include, for example, an accelerometer and / or a gyroscope incorporated into the HMD. Alternatively, or in addition, the pose detector 1204 may include one or more cameras, and may detect the pose of the viewer's head based on the pose of the viewer's head in images captured by the one or more cameras. In some examples, the HMD has multiple light sources attached thereto, and the pose of the viewer's head may be determined based on how these light sources appear in images captured by the cameras (e.g., using constellation tracking).

[0100] As previously described, the pose of the viewer's head may correspond to the pose of the virtual camera that will be used to view the object from different respective viewpoints. Thus, the pose detector 1204 enables the determination of the pose of the virtual camera. It is generally believed that when a viewer watches an immersive video, the pose of the virtual camera (and the viewer's head) will be different from the pose of the source camera that captured the source image of the object.

[0101] The system also includes an image processor 1206 that is operable to acquire a plurality of source images of the object and camera pose data indicating the pose of the camera that captured each source image. As previously described, each source image may be captured by an array of multiple cameras (e.g., stereo cameras) or by moving a single camera to a different position relative to the object. Each source image is associated with a corresponding source camera to which the camera pose data corresponds. Figures 3 to 11 The source image is captured in any manner described.

[0102] The image processor 1206 is configured to receive input from the pose detector 1204 and, in response thereto, distort at least some of the source images based on a difference between a pose of one or more cameras associated with the source images and a current pose of the viewer's head relative to the object. The pose of the viewer's head relative to the object corresponds to a reconstructed pose of the virtual camera relative to the object.

[0103] Specifically, the image processor 1206 is configured to distort each source image based on a difference in the pose of the source camera associated with the image and the pose of the virtual camera through which the object can be viewed. Figures 3 to 11 Any method related to performing the distortion of the source image.

[0104] The system also includes an image combiner 1208 configured to determine a weight associated with each distorted image based on a similarity between a pose of a camera associated with the image and a pose of a viewer's head relative to the object (or a reconstruction thereof). The image combiner 1208 is further configured to combine the distorted images together according to the associated weights to form an image of the subject for display at the display element. That is, the image combiner 1208 can be configured to blend the distorted images together to generate a final image of the object from the viewer's viewpoint (which in turn corresponds to the viewpoint of a virtual camera through which the object can be viewed). The final blended image can be superimposed with a real or virtual environment that can be viewed by the viewer.

[0105] In some examples, the image processor 1206 is further configured to obtain depth data associated with each source image and generate a polygonal mesh of the subject for each source image based on the source image and the depth data associated with the image. The depth data may be obtained in any manner described above. The image processor 1206 is configured to generate a distorted image from each source image by reprojecting at least some portion of the mesh generated for the source image so as to be perpendicular to the direction in which the viewer's head is facing (corresponding to the viewer's gaze direction). Fig. 8A and 8B This reprojection of the mesh is described.

[0106] The image processor 1206 may be configured to generate a texture for each reprojected mesh to be applied to the mesh. The image processor 1206 may include a parallax shader operable to parallax map portions of a source image to corresponding portions of a reprojected mesh generated from the source image. As previously described, the parallax mapping process may include: Fig. 9 shown.

[0107] The pose of the viewer's head corresponds to the pose of the virtual camera used to view the object, and the pose of the one or more cameras capturing the source image corresponds to the camera array plane. The image combiner 1208 can be configured to determine the weight of each distorted image by determining the intersection between the vector connecting the virtual camera to the object and the camera array plane. The image combiner 1208 can be configured to determine the weight of each distorted image based on the distance of the camera associated with the image and the intersection on the camera array plane. As previously described, the processing is as follows Fig. 10A and 10B shown.

[0108] In some examples, image processor 1206 may be implemented on a video game console that communicates with a head mounted display (HMD) that includes display element 1202. Thus, the video game console may be configured to generate and blend the distorted image and send the result to the HMD for display therein. In some examples, for example, the distorted image may be generated at the game console and the blending performed at the HMD. Next generation consoles (e.g., PS5 TM ) may be adapted to distort each source image according to the method described above.

[0109] Alternatively, or in addition, the image processor 1206 may be implemented at the HMD itself, in a manner similar to the video game consoles described above.

[0110] Alternatively, or in addition, the image processor 1206 may be implemented, for example, on a server with which the video game console and / or HMD communicates. Thus, the distorted image may be generated at the server and sent from the server to the HMD (possibly via the game console). It will be appreciated that the viewer may make several small movements of their head, so it is preferred that the distorted image is generated on the game console rather than on the server. This will limit any delay introduced by having to transmit the head pose data to the server before rendering the object from the corresponding viewpoint.

[0111] It should also be understood that the methods described herein may be performed on conventional hardware appropriately adapted by software instructions or by inclusion or replacement of dedicated hardware (as applicable). Thus, the required adaptation of existing portions of conventional equivalent devices may be implemented in the form of a computer program product comprising processor-implementable instructions stored on a non-transitory machine-readable medium such as a floppy disk, optical disk, hard disk, PROM, RAM, flash memory, or any combination of these or other storage media, or implemented in hardware as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array) or other configurable circuit suitable for adapting conventional equivalent devices. Additionally, such a computer program may be sent via a data signal on a network such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks.

Claims

1. A method for generating an image of a subject from a viewpoint of a virtual camera, the method comprising: Acquire multiple source images of a subject in a scene, each source image being captured from a different respective viewpoint; acquiring, for each source image, camera pose data indicating a pose of the camera relative to the subject; Acquiring virtual camera pose data indicating a pose of the virtual camera relative to the subject; generating respective distorted images for at least some of the source images, each distorted image being generated based on a difference between a pose of the camera that captured the source image and a pose of the virtual camera; as well as determining a weight associated with each distorted image based on a similarity between a pose of the camera that captured the distorted images and a pose of the virtual camera, and combining the distorted images according to the determined weights to generate an image of the subject from the viewpoint of the virtual camera, Wherein determining a weight associated with each distorted image comprises: determining an intersection point between a plane defined by at least three cameras and a line connecting the virtual camera to the subject, and determining the weight associated with the distorted image based on a distance between the camera that captured the distorted image and the intersection point.

2. The method according to claim 1, wherein generating the distorted image comprises: Obtaining depth data associated with each source image; generating a polygonal mesh of the subject for each source image based on the source images and depth data associated with the images; as well as For each mesh, at least some portion of the mesh is reprojected to be perpendicular to the direction the virtual camera is facing.

3. The method according to claim 2, wherein generating the distorted image further comprises: generating, for each reprojected mesh, a texture for application to the mesh, the texture being generated by parallax mapping a corresponding source image onto the mesh; and The generating of the distorted image comprises: rendering a 2D image of each re-projected mesh to which the texture has been applied.

4. The method of claim 1 , wherein each source image corresponds to an image of the subject captured in front of a green screen; and in, Generating the distorted image further includes: For each source image, pixels corresponding to the object are separated from pixels corresponding to the background.

5. The method according to claim 2 or 3, wherein generating the polygonal mesh comprises: Applying a two-dimensional grid to each source image; for each source image and the acquired depth data for that image, determining a depth variation for each cell throughout the two-dimensional grid; as well as A corresponding polygonal mesh is generated for each source image, wherein a subdivision density of the mesh is determined based on the depth variation throughout each cell of the two-dimensional mesh applied to the source image.

6. The method of claim 1 , wherein each source image comprises a stereo image captured by a stereo camera; and in, At least five source images are acquired, the poses associated with the source images corresponding to an array of stereo cameras arranged vertically or horizontally in an x-shape relative to the subject.

7. The method of claim 1 , wherein the virtual camera pose data corresponds to a pose of a reconstructed HMD used to view the subject; and in, Generating the image of the subject includes generating a stereoscopic image and outputting the generated stereoscopic image at the HMD.

8. A computer readable medium having computer executable instructions adapted to cause a computer system to perform the method of any one of claims 1 to 7.

9. A display system comprising: a display element operable to display an image of the object to a viewer; a gesture detector operable to detect a gesture of the viewer's head; an image processor operable to acquire a plurality of source images of the object, and camera pose data indicative of a pose of a camera that captured each source image; wherein the image processor is configured to receive input from the pose detector and, in response to the input, distort at least some of the source images based on a difference between a pose of one or more cameras associated with the source images and a reconstructed current pose of the viewer's head relative to the object; and an image combiner configured to determine a weight associated with each distorted image based on a similarity between a pose of the camera associated with the distorted image and the pose of the viewer's head relative to the reconstruction; and wherein the image combiner is configured to combine the distorted images together according to the associated weights to form an image of the object for display at the display element, wherein the pose of the viewer's head corresponds to a pose of a virtual camera used to view the object, and wherein the pose of one or more of the cameras capturing the source images corresponds to a camera array plane; wherein the image combiner is configured to determine an intersection point between a vector connecting the virtual camera to the object and the camera array plane; and Wherein the image combiner is configured to determine a weight associated with the distorted image based on a distance between the camera that captured the distorted image and the intersection point.

10. The display system of claim 9, wherein the image processor is further configured to obtain depth data associated with each source image, and generate a polygonal mesh of the subject for each source image based on the source image and the depth data associated with the image; and in, The image processor is configured to generate a distorted image from each source image by reprojecting at least some portions of the grid generated for the source image so as to be perpendicular to the direction in which the viewer's head is facing.

11. The display system of claim 10, wherein the image processor is configured to generate, for each reprojected mesh, a texture for application to the mesh; and in, The image processor includes a parallax shader operable to parallax map portions of the source image to corresponding portions of a reprojection mesh.

12. The display system according to claim 9, comprising: Head-mounted displays; wherein the display element corresponds to the display element of the head mounted display, and wherein the gesture detector is operable to detect a position and / or orientation of the head mounted display.

13. The display system according to claim 12, comprising: A gaming console in communication with the head mounted display, wherein the gaming console includes the image processor and the gaming console is operable to send the combined distorted image to the head mounted display for display at the display element.

Images