Generate an image based on a rendered image

Abstract

The present disclosure relates to a computing device for generating images configured to be captured by a camera, the computing device being configured to determine an image to be rendered associated with at least one camera pose and render the image, receive information about a current camera pose, and generate an image associated with the current camera pose based on the rendered image associated with the at least one camera pose.

Description

[Technical field]

[0001] The present disclosure relates to a computer device, system and method for generating an image based on a rendered image. [Background technology]

[0002] The rendering of two- or three-dimensional models or scenes using computer programs allows augmented reality to be applied in various sectors, including the film, broadcast and live event industries. For example, in film production, instead of using a green screen to provide a background for a scene, virtual content can be rendered and displayed on a large screen while an actor performs in front of the screen and a camera films the actor and the rendered content. This technique allows the rendered virtual content to be composited in real-time with the live action content. In broadcast and live events, virtual content can be rendered in real-time and displayed on user devices such as mobile phones and head-mounted displays (HMDs), providing users with a personalized augmented reality experience. Depending on the application, the virtual content can be mixed with the real content being displayed.

[0003] Rendering virtual content in real time can require significant computational resources, depending on the rate at which viewpoints need to be rendered, the complexity of the content, and the number of users wanting to access the content. A key aspect in creating a convincing virtual environment for filmmaking, broadcast, live events, or other augmented reality applications is that the viewpoint of the rendered frame or image matches the viewpoint of the camera capturing the frame, or the viewpoint of the user device projecting and displaying the rendered frame. Summary of the Invention [Means for solving the problem]

[0004] According to one aspect of the present disclosure, a computing device is provided for generating images configured to be captured by a camera, which may be suitable for compositing virtual and live action content. The computing device is configured to determine an image to be rendered associated with at least one camera pose and render the image, receive information regarding a current camera pose, and generate an image associated with the current camera pose based on the rendered image associated with the at least one camera pose.

[0005] By generating images based on already rendered images while accounting for differences in image viewpoints or camera poses, the computing device reduces the computational power required by not having to render speculative images in the hope of matching the actual pose of the camera, and also reduces latency as fewer messages are sent over the network to display images for the relevant viewpoints.

[0006] The viewpoint of the image associated with the current camera pose may be different from the viewpoint of the rendered image of at least one camera pose. This means that the generated image has a different viewpoint than the rendered image. Therefore, images do not need to be rendered for each viewpoint or camera pose, and instead can be generated with less computational power. Also, latency can be reduced, since generating an image as disclosed herein is much faster than rendering an image involving various components.

[0007] The computing device may be configured to generate an image associated with a current camera pose based on depth and / or geometry information of a rendered image associated with at least one camera pose. Alternatively or additionally, the image may be generated based on any suitable geometric scene representation of the rendered image that forms the basis of the generated image.

[0008] In one example, the at least one camera pose may be a predicted at least one camera pose. This means that the computer device is configured to predict at least one camera pose and render an image associated with the camera pose. The computer device can then generate an image corresponding to the actual camera pose or the current camera pose based on the rendered predicted camera pose. It should be understood that this is useful when the camera moves and does not estimate the predicted camera pose, but estimates a pose close to the predicted camera pose.

[0009] In one example, at least one camera pose may be a previous camera pose. This means that the computer device is configured to determine a previous camera pose, which may be the last known camera pose. Then, based on the rendered image of the previous camera pose, a new image corresponding to the actual camera pose is generated. It should be understood that this is useful when there is an incremental change in the camera pose.

[0010] The computing device may be configured to generate an image associated with the current camera pose by deprojecting content of the image associated with the at least one camera pose from the two-dimensional screen space to its position in the three-dimensional world space using an inverse of the view projection matrix of the at least one camera pose, and applying the view projection matrix of the current camera pose to the position of the content in the three-dimensional world space such that the image of the current camera pose can be projected into the two-dimensional screen space, such that the generated image has perspective and parallax effects that match or correspond to the current camera pose.

[0011] The image associated with the current camera pose can be generated by pixel reprojection or mesh reprojection, as described herein.

[0012] In one example, the computing device may be further configured to divide an image associated with at least one camera pose into a mesh, warp and move the mesh based on a depth map of the image, perform backprojection by using an inverse of a view projection matrix of the at least one camera pose, and apply a view projection matrix of a current camera pose so that the image can be projected into a two-dimensional screen space.

[0013] In one example, the at least one camera pose may include a plurality of camera poses, and the computer device is configured to render a plurality of images, each associated with a corresponding camera pose, based on the received information on the current camera pose, select a camera pose from the plurality of camera poses that is closest in matching the current camera pose, and generate an image associated with the current camera pose based on the selected camera pose and at least one further image associated with the plurality of camera poses. This means that a plurality of camera poses are predicted and an image is generated based on two rendered images, one of which is closest to the actual camera pose. It also means that a plurality of known camera poses are considered and an image is generated based on two previously rendered images, one of which is closest to the actual camera pose. Also, an image can be generated based on an image rendered associated with the predicted camera pose and a rendered image associated with the previous camera pose.

[0014] In the multiple camera pose example described above, the computing device may be further configured to generate an image associated with the current camera pose by back-projecting the content of the image associated with the selected camera pose from the two-dimensional screen space to its position in the three-dimensional world space using the inverse of the view projection matrix of the selected camera pose, and then applying the view projection matrix of the current camera pose to the position of the content in the three-dimensional world space so that the image of the current camera pose can be projected into the two-dimensional screen space.

[0015] The image associated with the current camera pose can be generated by pixel reprojection or mesh reprojection, as described herein.

[0016] In one example, the computing device may be further configured to divide an image associated with the selected camera pose into a mesh, warp and move the mesh based on a depth map of the image, perform a backprojection by using an inverse of a view projection matrix of the selected camera pose, and apply a view projection matrix of the current camera pose so that the image can be projected into a two-dimensional screen space.

[0017] The computing device may be configured to perform inpainting (inpainting) to fill in deoccluded regions in a generated image associated with a current camera pose, where the inpainting is based on images associated with at least one camera pose, where the at least one camera pose may be a predicted camera pose or a known camera pose, and the inpainting may be based on rendered images used to generate the image, or on other adjacent rendered images, or a combination thereof.

[0018] The inpainting may be based on surface normal information of objects in an image associated with at least one camera pose.

[0019] The inpainting may further be based on albedo and ambient occlusion information of an image associated with at least one camera pose.

[0020] According to another aspect of the present disclosure, a method for generating an image is provided. The method includes determining a rendered image associated with at least one camera pose and rendering the image. The method further includes receiving information regarding a current camera pose and generating the image associated with the current camera pose based on the image associated with the at least one camera pose. A viewpoint of the image associated with the current camera pose may be different from a viewpoint of an image of the at least one camera pose.

[0021] The method may further include generating an image associated with the current camera pose based on depth and / or geometry information of an image associated with at least one camera pose, or other geometric representation of the scene.

[0022] The method may also include back-projecting content of an image associated with the at least one camera pose from the 2D screen space to a position in the 3D world space using an inverse of a view projection matrix for the at least one camera pose, and applying the view projection matrix for the current camera pose to the position of the content in the 3D world space to generate an image associated with the current camera pose, such that the image for the current camera pose can be projected into the 2D screen space.

[0023] The image associated with the current camera pose may be generated by pixel reprojection or mesh reprojection, as described herein.

[0024] In one example, the method may include dividing an image associated with at least one camera pose into a mesh, warping and moving the mesh based on a depth map of the image, backprojecting by using an inverse of a view projection matrix of the at least one camera pose, and applying a view projection matrix of the current camera pose so that the image can be projected into a two-dimensional screen space.

[0025] In one example, the at least one camera pose includes a plurality of camera poses, and the method may further include rendering a plurality of images, each associated with a corresponding camera pose, based on the received information regarding the current camera pose, and the method may further include selecting a camera pose from the plurality of camera poses that is closest to the current camera pose, and the method may include generating an image associated with the current camera pose based on the selected camera pose and the at least one further image associated with the plurality of camera poses.

[0026] Generating an image associated with the current camera pose may include back-projecting content of the image associated with the selected predicted camera pose from the 2D screen space to its position in the 3D world space using an inverse of the view projection matrix of the selected predicted camera pose, and applying the view projection matrix of the current camera pose to the position of the content in the 3D world space so that the image for the current camera pose can be reprojected into the 2D screen space.

[0027] In one example, the method may include dividing an image associated with a selected predicted camera pose into a mesh, warping and translating the mesh based on a depth map of the image, performing a backprojection using an inverse of a view projection matrix of the selected predicted camera pose, and applying a view projection matrix of a current camera pose so that the image can be projected into a two-dimensional screen space.

[0028] The method further includes performing inpainting to fill the deoccluded regions in the generated image associated with the current camera pose, the inpainting being based on the image associated with the at least one camera pose. The inpainting may be based on surface normal information, albedo, and / or ambient occlusion information of the image associated with the at least one camera pose, as described herein.

[0029] According to another aspect of the present disclosure, there is provided a system including a computing device for generating an image, a display for displaying the rendered image, and a camera configured to capture the rendered image displayed on the display, The computing device may be any computing device disclosed herein.

[0030] According to another aspect of the disclosure, a system is provided that includes a computer device for generating an image, a display for displaying the rendered image, and a camera configured to capture the rendered image displayed on the display. The computer device is configured to determine a plurality of camera poses and render an image for each camera pose, the camera includes at least one sensor for detecting a current camera pose, and the camera is configured to transmit its current pose to the computer device, the computer device is further configured to select a determined camera pose of the plurality of camera poses that is closest in matching the current camera pose, and generate an image associated with the current camera pose based on at least one image of the plurality of determined camera poses, and the display is configured to display the generated image and capture the generated image with the camera in the current pose. The computer device may be further configured to perform any of the methods described herein.

[0031] According to another aspect of the present disclosure, a computing device is provided for generating an image for display on a first device, the computing device being configured to: determine a pose of a first device having a display; identify at least one second device having a display, the at least one second device having a different pose than the first device; render an image associated with the pose of the at least one second device; and generate the image associated with the pose of the first device based on the image associated with the pose of the at least one second device.

[0032] This means that instead of rendering content for multiple devices in real-time, content can be rendered for some devices and for the remaining devices content can be generated based on the rendered content. This becomes particularly useful in scenarios where devices display the same content / environment from different perspectives at the same time, for example the content could be an augmented reality of a concert or show.

[0033] The computing device may be configured to generate an image associated with a pose of the first device by using depth information and / or geometry information of an image associated with a pose of the at least one second device, or other suitable geometric scene representation.

[0034] The computing device may be configured to generate an image associated with a pose of the first device by back-projecting content of an image associated with the at least one second device pose from the two-dimensional screen space to a position in the three-dimensional world space using an inverse of the view projection matrix of the at least one second device, and then applying the view projection matrix of the first device to the position of the content in the three-dimensional world space so that the image of the first device can be projected into the two-dimensional screen space.

[0035] The image associated with the first device pose may be generated by pixel reprojection or mesh reprojection, as described herein. In one example, the computing device may be further configured to divide the image associated with the at least one second device pose into a mesh, warp and move the mesh based on a depth map of the image, perform backprojection by using an inverse of a view projection matrix of the at least one second device pose, and apply the view projection matrix of the first device pose so that the image can be projected into a two-dimensional screen space.

[0036] In one example, the at least one second device includes a plurality of second devices with different poses, and the computing device is configured to render an image associated with each second device such that the plurality of images are rendered, determine a pose for each of the plurality of second devices, select a second device that is closest in matching the pose of the first device, and generate an image associated with the first device based on the image of the selected second device and at least one further image associated with another second device. This can be useful when there are a plurality of second devices each rendering content from a different perspective.

[0037] The computing device may be configured to generate an image associated with a selected second device pose by back-projecting content of the image associated with the selected second device pose from the two-dimensional screen space to its position in the three-dimensional world space using an inverse of the second device's view projection matrix, and then applying the first device's view projection matrix to the position of the content in the three-dimensional world space so that the first device's image can be projected into the two-dimensional screen space.

[0038] The image associated with the pose of the first device may be generated by pixel reprojection or mesh reprojection, as described herein.

[0039] In one example, the computing device may be further configured to divide an image associated with the selected second device pose into a mesh, warp and move the mesh based on a depth map of the image, perform backprojection by using an inverse of a view projection matrix of the selected second device pose, and apply the view projection matrix of the first device pose so that the image can be projected into a two-dimensional screen space.

[0040] The computing device may be further configured to perform inpainting information for filling the unoccluded regions in a generated image associated with the first device, where the inpainting may be based on an image associated with at least one second device.

[0041] The inpainting may be based on surface normal information of objects in an image associated with at least one second device, where the inpainting may be based on rendered images used to generate the image, or may be based on other adjacent rendered images of other second devices, or a combination thereof.

[0042] The inpainting may be based on albedo and ambient occlusion information of an image associated with at least one second device.

[0043] According to another aspect of the present disclosure, a method of generating an image for display on a first device is provided, the method including determining a pose of a first device having a display and identifying at least one second device having a display, the at least one second device having a different pose than the first device. The method further includes rendering an image associated with the pose of the at least one second device and generating an image associated with the pose of the first device based on the image associated with the pose of the at least one second device.

[0044] The method may further include generating an image associated with a pose of the first device by using depth information and / or geometry information of an image associated with a pose of the at least one second device, or other geometric representation of the scene.

[0045] The method may include generating an image associated with a pose of the first device by back-projecting content of an image associated with the at least one second device pose from the 2D screen space to a position in the 3D world space using an inverse of the view projection matrix of the at least one second device, and applying the view projection matrix of the first device to the position of the content in the 3D world space so that the image of the first device can be reprojected into the 2D screen space.

[0046] The image associated with the pose of the first device may be generated by pixel reprojection or mesh reprojection, as described herein.

[0047] In one example, the method may include dividing an image associated with at least one second device pose into a mesh, warping and translating the mesh based on a depth map of the image, performing a backprojection by using an inverse of a view projection matrix of the at least one second device pose, and applying a view projection matrix of the first device pose so that the image can be projected into a two-dimensional screen space.

[0048] In one example, the at least one second device includes a plurality of second devices having different poses, and the method includes rendering an image associated with each second device such that a plurality of images are rendered, determining a pose for each of the plurality of second devices, selecting a second device that is closest in matching the pose of the first device, and generating an image associated with the first device based on the image of the selected second device and at least one further image associated with another second device. In this example, the method may include generating the image associated with the pose of the first device by back-projecting content of the image associated with the pose of the selected second device from the 2D screen space to its position in the 3D world space using an inverse of the view projection matrix of the second device, and applying the view projection matrix of the first device to the position of the content in the 3D world space so that the image of the first device can be projected into the 2D screen space.

[0049] The image associated with the first device pose may be generated by pixel reprojection or mesh reprojection, as described herein. In one example, a method may include dividing an image associated with a selected second device pose into a mesh, warping and translating the mesh based on a depth map of the image, performing a backprojection by using an inverse of a view projection matrix of the selected second device pose, and applying the view projection matrix of the first device pose so that the image can be projected into a two-dimensional screen space.

[0050] The method may further include inpainting information for filling the disoccluded regions in the generated image associated with the first device, the inpainting being based on the image associated with the at least one second device. The inpainting may be based on surface normal information of objects in the image associated with the at least one second device. The surface normal information may be extracted from the image associated with the at least one second device and applied to the generated image. The inpainting may further include albedo and ambient occlusion information of the image associated with the at least one second device. The albedo and ambient occlusion information may be extracted from the image associated with the at least one second device and applied to the generated image.

[0051] According to another aspect of the present disclosure, a computing device for generating an image is provided, the computing device configured to receive a first image depicting an object from a viewpoint, generate a new image based on depth information and / or geometry information of the first image or other geometric representation of the scene, the new image depicting the object from a viewpoint different from the first image, and perform inpainting to fill deoccluded regions in the new image, the inpainting being based on surface normal information of the first image and / or at least the other image different from the first image.

[0052] Surface normal information may be extracted from the first image and / or at least the other image and applied to the new image. The inpainting may further include albedo and ambient occlusion information of the first image and / or at least the other image. Albedo and ambient occlusion information may also be extracted from the first image and / or at least the other image. The first image and at least one other image may be of different perspectives and therefore may include different or complementary information used to generate the new image.

[0053] According to another aspect of the present disclosure, there is provided a method of generating an image, the method including receiving a first image depicting an object from a viewpoint, generating a new image based on depth and / or geometry information of the first image or other scene geometric representation, the new image depicting the object from a viewpoint different from that of the first image, and performing inpainting to fill deoccluded regions in the new image, the inpainting being based on surface normal information of the first image and / or at least the other image different from the first image.

[0054] According to another aspect of the present disclosure, a method of generating an image is provided, the method including receiving a first image depicting an object from one viewpoint, dividing the image associated with a selected second device pose into a mesh, warping and moving the mesh based on a depth map of the image, performing backprojection by using an inverse of a view projection matrix of the selected second device pose to generate a new image depicting the object from another viewpoint, and applying a view projection matrix of the first device pose so that the image can be projected into a two-dimensional screen space. Any inpainting and / or filling methods described herein may be further applied to the new image.

[0055] According to another aspect of the present disclosure, a computer-readable medium is provided having stored thereon instructions that, when executed by a processor or processing unit, cause the processor to implement a method or methods described herein.

[0056] A computer program which, when executed on a computer, causes the computer to carry out the methods as described herein.

[0057] It is expressly intended that within the scope of this application, the various aspects, embodiments, examples, and alternatives presented in the preceding paragraphs, claims, and / or in the following description and drawings, in particular their individual features, may be employed independently or in any combination. That is, the features of all examples and / or any examples may be combined in any manner and / or combination, so long as such features are not inconsistent. Applicants reserve the right to modify the originally filed claims or to submit new claims accordingly, including the right to amend the originally filed claims to depend on or incorporate features of any other claims, even if not originally so claimed. [Brief description of the drawings]

[0058] Various features of the disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate features of the disclosure. [Figure 1] FIG. 1 is a schematic diagram of a system for generating virtual content composited with live action. [Figure 2a] FIG. 1 is a schematic diagram of a computing device for generating an image configured to be captured by a camera. [Figure 2b] 1A-1D are schematic diagrams of various backprojection and reprojection methods; [Diagram 3] 4 is a flow chart of a method for generating an image configured to be captured by a camera. [Figure 4] FIG. 1 is a schematic diagram of a computing device that generates images for display on a user device. [Diagram 5] 1 is a flowchart of a method for generating an image for display on a user device. [Figure 6] FIG. 1 is a schematic diagram of a computer device for generating an image. [Figure 7] 1 is a flow chart of a method for generating an image. [Figure 8] 1 is a schematic diagram of a computing device that can be used to implement aspects of one or more of the methods described herein. [Figure 9] FIG. 9 is a schematic diagram of a system 900 for generating virtual content composited with live action. [Figure 10] FIG. 1 is a schematic diagram of a computing device that generates images for display on a user device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0059] As mentioned in the background section, rendering images in real time can require significant computational power and resources, especially when creating virtual environments that need to match the user or camera's viewpoint to be convincing.

[0060] For example, in the film industry, the background of a scene may be rendered in real-time rather than applied in post-production, but it is important that the viewpoint of the background matches the viewpoint of the camera, as described in detail herein. An example of a system 100 for creating virtual content composited with live action is shown in FIG. 1. The system comprises a network of components that can communicate with each other via wired and / or wireless connections using TCP / IP messaging protocols. The components of the system 100 include a compositing server 101 that acts as a central hub of the system, a camera 102 configured to communicate with the main compositing server 101, and a tracker 103 configured to track the pose and characteristics of the camera. The tracker 103 may be attached to the camera 102. The components of the system 100 further include a dedicated server 104 configured to receive the tracked pose and characteristics of the camera, record the pose and characteristics, and send it to the compositing server 101. The system also comprises a rendering server 105 configured to render frames of the virtual content and send the rendered frames to the compositing server 101. The system 100 further includes a display 106, for example an LED display, for projecting or displaying the rendered frames.

[0061] In use, the camera may move as it records the rendered content to be displayed on the display. The pose and identity of the camera 102 is tracked using the tracker 103. More specifically, the tracker 103 determines the location, orientation, and / or identity of the camera as tracking information and transmits this tracking information to the dedicated server 104. The dedicated server 104 transmits the tracking information to the compositing server 101, which determines a frame of the virtual content to render that corresponds to the pose of the camera based on the tracking information. The compositing server 101 then instructs the rendering server 105 to render the determined frame and project this frame on the display 106. Once the frame is projected on the display 106, the camera 102 can capture or film not only the display 106 but also actors and filming equipment positioned in front of the display 106.

[0062] It is important that the rendered frames correspond to the camera's viewpoint, so that the virtual content can be combined with actors and props on the real set to create composite images that place real objects within the virtual environment with correct perspective and parallax effects.

[0063] A challenge with this system is that when the camera pose changes, the projected frames need to be updated to match the new viewpoint of the camera. The steps of the tracker 103 sending updated tracking information to the dedicated server 104, which then updates the compositing server 101, which then calculates which frame to render and instructs the rendering server 105 accordingly, and then projecting the rendered frame on the display 106, have inherent latency. By the time the rendered frame is displayed on the display 106, the camera may have already changed its pose and the displayed frame may not correspond to the camera's current pose.

[0064] To overcome this, it can be envisaged that the rendering server 105 can render multiple frames corresponding to predicted camera poses, such that at least one rendered frame matches the actual camera pose. However, rendering multiple frames requires huge computational power and places unnecessary strain on the system, with no guaranteed match between the final selected frame and the final camera pose.

[0065] In another example, when a user device's display shows rendered frames to provide a virtual environment, the viewpoint of each displayed frame needs to match the viewpoint of the user device for the virtual content to be convincing. This application can also require a significant amount of computational resources, especially when the user device moves and changes its viewpoint, and new frames need to be rendered to account for the change in viewpoint. Also, when there is an audience, with each member simultaneously viewing the augmented reality content on their personal user device (e.g., mobile phone, head mounted display (HMD), etc.), frames need to be rendered for each user device with a different viewpoint, requiring a significant amount of computational resources. These types of applications can also be prone to latency between the time the user device sends its pose to the rendering server and the time it receives the rendered frames for projection or display.

[0066] Particular examples of the present disclosure may attempt to reduce the computational resources required to render an image and / or reduce the latency experienced by a system rendering the image, while providing convincing perspective and parallax effects for the image when displayed on a display.

[0067] In the following description, for purposes of explanation, numerous specific details of particular examples are set forth. Reference in the specification to an "example" or similar terminology means that a particular feature, structure, or characteristic described in connection with the example is included in at least one of the examples, but not necessarily in other examples. It should also be understood that features of one example may be applied to other examples, where possible, even if not explicitly described.

[0068] For purposes of this description, the terms frame, viewpoint, and image should be understood to have the same meaning and be used interchangeably. Additionally, a frame is understood to be a two-dimensional or three-dimensional representation of a three-dimensional scene or model. Additionally, a frame should be understood to be associated with information describing the geometry and pose of the frame of a camera or device, which for purposes of this description also includes the location and / or orientation, and optionally the specific information, of the camera used to render or capture the frame.

[0069] For the purposes of this description, a camera should be understood as a camera configured to record or capture video displayed on a screen, the camera may be fixed, configured to change orientation, and / or the camera may be configured to move and change location. In other examples, the camera may refer to a synthetic camera model or object in computer graphics.

[0070] A generated image as used herein is understood as an image that is created or synthesized based on a rendered image or on information in a rendered image, which may for example be a geometric representation of the scene, such as geometric information and / or depth information. A rendered image is understood as a two-dimensional image that is created or rendered from a two-dimensional or three-dimensional model using a computer or a computer program.

[0071] Figure 2a shows a computing device 201 for generating an image configured to be suitable for capture by a camera 102, which may form part of the system shown in Figure 1. Here, an image configured to be suitable for capture by a camera may mean an image having a viewpoint that corresponds to a pose of a camera configured to capture the image.

[0072] The computing device 201 is configured to determine an image to be rendered associated with at least one camera pose, and thereafter render said image, receive information about a current camera pose, and generate an image associated with the current camera pose based on the image associated with the at least one camera pose.

[0073] By generating images based on already rendered images while accounting for viewpoint differences, the computing device reduces the computational power required by not having to render speculative images in the hope of matching the actual pose of the camera, and also reduces latency as fewer messages are sent over the network to display the images for the relevant viewpoints.

[0074] The computing device may be a server, multiple servers, a compositing server, a rendering server, a dedicated server, a computer, or other hardware having computing capabilities. The computing device may include input and output interfaces for communicating with other components, such as the camera 102, the tracker 103, and / or a display 106 configured to display the content.

[0075] The computing device may receive information about the current and previous camera poses from the tracking device to track the camera poses and, if necessary, track camera specific information.

[0076] As mentioned above, the term “camera pose” or “camera pose” as used herein is understood as the location and / or orientation of the camera, and optionally the intrinsic information of the camera. “Intrinsic information” as used herein is understood as the intrinsic parameters of the camera. Examples of intrinsic information may include the focal length, lens distortion, resolution, pixel dimensions, and / or optical length of the camera. The camera is configured to capture or record the rendered content displayed on the display 106. The camera pose can be tracked using, for example, an accelerometer, a gyroscope, a GPS, and / or a tracking device such as simultaneous localization and mapping (SLAM), or other suitable techniques. Recent history and / or current camera pose can be used to predict future camera poses.

[0077] As mentioned above, it should be understood that the viewpoint of the image associated with the current camera pose may be different from the viewpoint of the image of the at least one camera pose. Also, in one example, the at least one camera pose may be at least a predicted camera pose and / or at least one previous or estimated camera pose, e.g., the last known camera pose. To further explain, the computing device 201 may predict at least one future camera pose, determine an image to be rendered associated with at least one predicted camera pose, and then render said image, receive information about the current camera pose, and generate an image associated with the current camera pose based on the image associated with the at least one predicted camera pose. Alternatively or additionally, the computing device may determine an image to be rendered associated with at least one known or previous camera pose, which may be, e.g., the last known camera pose, render the image if not yet rendered, receive information about the current camera pose, and generate an image associated with the current camera pose based on the image associated with the at least one previous camera pose.

[0078] The computing device 201 may further be configured to generate an image associated with the current camera pose based on depth and / or geometry information of an image associated with at least one camera pose. The depth information indicates the distance of a pixel from the camera, and in deferred rendering, the depth information is included in a depth buffer or a G-buffer. The geometry information indicates edges and / or vertices of objects in the image. In some examples, the computing device is configured to generate an image associated with the current camera pose based on a geometric representation of an arbitrary scene, for example, a mesh including depth information as described above, or information about vertices and faces.

[0079] In one example, the computing device 201 can be configured to generate an image associated with a current camera pose by deprojecting content of an image associated with at least one camera pose from a two-dimensional (2D) screen space to a location in a three-dimensional (3D) world space using an inverse of a view projection matrix of the at least one camera pose, and then applying the view projection matrix of the current camera pose to the location of the content in the 3D world space so that the image of the current camera pose can be reprojected into the 2D screen space. This can be further described as inverting the view projection matrix of the at least one camera pose and multiplying the matrix by the x, y and z coordinates of each pixel in the 2D screen space (where the z coordinate corresponds to or can be determined by depth information, which may be included in a depth buffer, G-buffer, depth map, or other scene geometry representation, as described above). This returns each pixel to its original x, y, z location in the 3D world space. Each original x,y,z location in the 3D world space is multiplied by the view projection matrix of the actual (or current) camera pose so that the image of the actual pose can be reprojected into the 2D screen space with the correct viewpoint. The view projection matrix is ​​a well-known concept in the field of rendering 3D models into 2D, and can be understood as the multiplication or composition of the view matrix with the projection matrix.

[0080] Image reprojection as described above can be applied directly to each pixel, referred to herein as pixel reprojection, or can be achieved by reprojecting a geometric representation of the 3D scene to 2D and determining the final pixel location, or pixel color, via interpolation, referred to herein as mesh projection. Mesh projection can be understood as reprojecting a mesh or vertex, rather than a pixel, as in pixel reprojection. Furthermore, when reprojecting a mesh or vertex, the relationship or connection between the pixels is taken into account or included in the reprojection, so that non-occluded regions in the generated image can be filled based on an interpolation of known pixel values. This is achieved by dividing a 2D image associated with at least one camera pose into a grid or mesh, and then warping and translating the image based on a depth map, backprojecting by using an inverse view projection matrix associated with at least one camera pose, and then applying a view projection matrix of the current camera pose to render the transformed image, and thus the 2D image of the camera's current viewpoint.

[0081] Further explanation of pixel reprojection and vertex or mesh reprojection will now be described with reference to Fig. 2b. Fig. 2b illustrates pixel reprojection where pixels of rendered image 251 are displayed in 2D space and when reprojected into 2D space in generated image 252, the pixels move leaving pixels 253 that need to be filled with a color, for example using inpainting (inpainting). Fig. 2b further illustrates mesh reprojection where pixels of rendered image 261 are shown in 2D space. The vertices associated with these pixels are backprojected and reprojected into 2D space to generate image 262, with the relationship between the vertices being maintained through the mesh or vertices. By using the mesh or vertices, colors can be interpolated for pixels that are located within the triangles shown in image 262 and need to be filled, thus eliminating or reducing the need for inpainting.

[0082] In another example, the computing device 201 is configured to determine an image to be rendered, each image associated with a corresponding camera pose, such that there are multiple camera poses, and based on the received information regarding the current camera pose, the computing device is further configured to select a camera pose from among the multiple camera poses that is closest to the current camera pose. The computing device 201 then generates an image associated with the current camera pose based on the selected camera pose and at least one further image associated with the multiple camera poses. Similar to above, the image can be generated based on depth information, geometry information, and / or other scene geometry representation of the image associated with the selected camera pose.

[0083] Further, the selected camera pose can be used for backprojection and reprojection, as described above. Additionally or alternatively, at least one further image can be used for inpainting, as described below. With respect to backprojection, the computing device 201 can be configured to generate an image associated with the current camera pose by backprojecting the content of the image associated with the selected camera pose from the 2D screen space to a position in the 3D world space using the inverse of the view projection matrix of the selected camera pose, and applying the view projection matrix of the current camera pose to the position of the content in the 3D world space, thereby allowing the image of the current camera pose to be reprojected into the 2D screen space. As above, this can be further described as inverting the view projection matrix of the selected camera pose and multiplying the matrix by the x, y, z coordinates of each pixel in the 2D screen space (where the z coordinates correspond to or can be determined by the depth information, which may be included in a depth buffer, G-buffer, depth map, or other scene geometry representation, as described above). This returns each pixel to its original x, y, z location in the 3D world space. Then, each original x, y, z location in the 3D world space is multiplied by the view projection matrix of the actual (or current) camera pose, allowing the image of the actual pose to be reprojected into 2D screen space with the correct viewpoint.

[0084] As described in the previous paragraph, the backprojection of the image associated with the selected camera pose and the reprojection of the image associated with the actual camera pose can be performed using either pixel reprojection or mesh reprojection, as described herein.

[0085] It should be understood that when the at least one camera pose includes multiple camera poses as described above, the multiple camera poses may be predicted camera poses and / or previous / known camera poses. To further explain, the computing device 201 may predict multiple future camera poses, determine an image to be rendered for each camera pose, and render said images, e.g., receive information about the current camera pose from a tracking device. The computing device may then select a predicted camera pose from the multiple camera poses that is closest to the current camera pose, and generate an image associated with the current camera pose based on an image associated with the selected predicted camera pose. Alternatively or additionally, the computing device may determine an image to be rendered associated with at least one known or previous camera pose, e.g., the last known camera pose, and render the image if not yet rendered. The computing device may then receive information about the current camera pose, select a previous known camera pose from the multiple camera poses that is closest to the current camera pose, and generate an image associated with the current camera pose based on an image associated with the selected known camera pose.

[0086] It should be understood that the generated images need not be based on images associated with a predicted or previous pose that is closest to the actual pose of the camera 102. The generated images can be based on images associated with a predicted or previous pose that is just close to the actual pose of the camera 102.

[0087] Disocclusion may appear in the generated image because the generated image has a different viewpoint than the rendered image on which it is based. Disocclusion is an area that appears when the foreground content of the image moves more than the background and disoccludes an area of ​​the background. In this case, disocclusion is an area that was occluded by an object in the rendered image but is visible in the generated image. Disocclusion can be avoided or mitigated by using mesh reprojection as described herein. In mesh reprojection, the final pixel location, or color of the pixel, is determined via interpolation. Additionally or alternatively, disocclusion can be filled in real-time using an inpainting technique. In one example, the computing device 201 is configured to perform inpainting to fill disoccluded areas in the generated image associated with the current camera pose, the inpainting being based on an image associated with at least one camera pose. This means that the inpainting may be based on the same image used for backprojection, or on other rendered images, or on another number of images corresponding to the camera pose. These other images may be associated with or correspond to previous and / or predicted poses of the camera 102. The inpainting may be based on surface normal information. The surface normal information may include normals associated with surfaces of objects in the images. The surface normal information may be extracted from the associated rendered images and applied to the generated image to fill in the disocclusions.

[0088] Furthermore, albedo and / or ambient occlusion information of an image associated with at least one camera pose can be used to fill disoccluded regions in the generated image. This means that albedo and / or ambient occlusion information of an image underlying the generated image is extracted and applied to the generated image to fill the disocclusion. As with surface normal information, albedo and / or ambient occlusion information may be extracted from other rendered images different from the rendered image underlying the generated image.

[0089] Surface normal information, albedo, and / or ambient occlusion information form part of the G-buffer in a technique known as deferred rendering, so in one example, inpainting is based on the G-buffer of the rendered image that is the basis of the generated image, and / or other rendered images.

[0090] By using G-buffer, albedo, normal, and / or ambient occlusion information, as described above, deocclusion can be filled in real-time and images can be blended to account for subtle changes in lighting between the images. By using albedo, normal, and / or ambient occlusion information, the exact effect that the lighting of the scene has on each pixel can be determined and taken into account, resulting in a more seamless blend between images.

[0091] In one example, additional or alternative inpainting techniques can be applied, for example, screen crack filling methods can be applied to fill small disocclusions a few pixels wide by using valid neighboring pixels to guess the most likely pixel value.

[0092] It should be understood that any inpainting technique or combination thereof may be applied or used to the generated image. Furthermore, any inpainting technique or combination thereof may be applied or used regardless of whether the generated image is generated via pixel reprojection or mesh reprojection as described herein.

[0093] The computing device 201 described above may form part of a system for generating images. Such a system comprises a display for displaying rendered images and a camera configured to capture the rendered images displayed on the display, similar to the diagram of FIG. 2a. In one example, the computing device 201 may be configured to determine a plurality of camera poses and render an image for each camera pose. The camera comprises at least one sensor for detecting a current camera pose, and the camera is configured to transmit its current pose to the computing device. The computing device is further configured to select a determined camera pose of the plurality of camera poses that is closest in matching the current camera pose, and generate an image associated with the current camera pose based on at least one image of the plurality of determined camera poses. The display 106 is configured to display the generated image and capture the generated image with the camera in its current pose. The computing device 201 of the system may comprise any of the features or any combination of features described above in relation to FIG. 2a.

[0094] Advantages of the computing device 201 are that it requires fewer computational resources, reduces latency in projecting images onto a display in real time, and provides convincing perspective and parallax effects, which are particularly useful in applications where live action is compositing with virtual content in real time.

[0095] As used herein, "real-time" may be understood as rendering and / or generating images at a frame rate that corresponds to the frame rate at which frames / images are displayed by the display (e.g., tens of frames per second).

[0096] A method 300 for generating an image configured to be captured by a camera is described with reference to Figure 3. Any alternatives, combinations or variations described with reference to Figure 2a may also be applied to the method 300 of Figure 3. The method 300 may be performed by a computing device as described in relation to Figure 2a.

[0097] 3 discloses various operations of a method 300. The method 300 includes determining an image to be rendered associated with at least one camera pose and rendering the image (301). The method 300 further includes receiving information about a current camera pose (302) and generating an image associated with the current camera pose based on the image associated with the at least one camera pose (303). Operation 302 may be performed prior to operation 301 or may be performed simultaneously with operation 301. A viewpoint of the image associated with the current camera pose may be different from a viewpoint of an image of the at least one camera pose.

[0098] The method 300 may further include generating an image associated with the current camera pose based on depth and / or geometry information of an image associated with at least one camera pose, or other geometric representation of the scene.

[0099] Method 300 may also include back-projecting content of an image associated with the at least one camera pose from the 2D screen space to a position in the 3D world space using an inverse of the view projection matrix of the at least one camera pose, and applying the view projection matrix of the current camera pose to the position of the content in the 3D world space to generate an image associated with the current camera pose, such that the image for the current camera pose can be reprojected into the 2D screen space.

[0100] As described in the previous paragraph, the backprojection of an image associated with at least one camera pose and the reprojection of an image associated with the current camera pose can be performed using either pixel reprojection or mesh reprojection as described herein.

[0101] In one example, the at least one camera pose includes a plurality of camera poses, and the method 300 may further include rendering a plurality of images, each associated with a corresponding camera pose, based on the received information regarding the current camera pose, and may further include selecting a camera pose from the plurality of camera poses that is closest to the current camera pose, and the method may include generating an image associated with the current camera pose based on the selected camera pose and the at least one further image associated with the plurality of camera poses.

[0102] Generating an image associated with the current camera pose may include back-projecting content of the image associated with the selected predicted camera pose from the 2D screen space to a position in the 3D world space using an inverse of the view projection matrix of the selected predicted camera pose, and applying the view projection matrix of the current camera pose to the position of the content in the 3D world space so that the image for the current camera pose can be reprojected into the 2D screen space.

[0103] As described in the previous paragraph, the backprojection of the image associated with the selected predicted camera pose and the reprojection of the image associated with the current camera pose can be performed using either pixel reprojection or mesh reprojection as described herein.

[0104] The method may further include performing inpainting to fill the deoccluded regions in the generated images associated with the current camera pose, the inpainting being based on the images associated with the at least one camera pose, the inpainting may be based on surface normal information, albedo, and / or ambient occlusion information of the images associated with the at least one camera pose, as described in relation to FIG.

[0105] Referring now to FIG. 4, a computing device 401 is disclosed. The computing device 401 is for generating images to be displayed on a user device. The user device may be referred to as a first device. The computing device 401 may be the same as the computing device 201 described above, and / or it may be a server, multiple servers, a compositing server, a rendering server, a dedicated server, a computer, or other hardware having computing capabilities. The computing device may include input and output interfaces for communicating with other components, such as, for example, user devices 402a-402h. The user devices may be mobile phones, tablets, head-mounted displays, augmented reality glasses, or similar computing devices having displays configured to display rendered content.

[0106] The computing device 401 is configured to determine a pose of a first device 402a having a display and to identify at least one second device 402b having a display, the at least one second device having a different pose than the first device. The computing device 401 is further configured to render an image associated with the pose of the at least one second device 402b and generate an image associated with the pose of the first device 402a based on the image associated with the pose of the at least one second device 402b. A viewpoint of the image of the first device 402a may be different from a viewpoint of the image of the at least one second device 402b.

[0107] The pose of a device, such as a user device, is understood as the location and / or orientation of the device. In some examples, the pose of the device should be understood in particular as the location and / or orientation of the device's display. The pose of the device can be tracked using an accelerometer, a gyroscope, a GPS, and / or simultaneous localization and mapping (SLAM), or other suitable techniques. The devices 402a-402h or associated SLAM components transmit these poses to the computing device 401 so that the computing device can receive information regarding the current device pose or a previous device pose.

[0108] The computing device 401 may further be configured to generate an image associated with a pose of the first device 402a by using depth and / or geometry information of an image associated with a pose of at least one second device 402b. As in the examples of Figures 2 and 3, the depth information indicates the distance of pixels from the user device and, in deferred rendering, is included in a depth buffer or G-buffer. The geometry information indicates edges and / or vertices of objects in the image. In some examples, the computing device 401 is configured to generate an image associated with a current camera pose based on a geometric representation of an arbitrary scene, for example, a mesh including depth information as described above or information about vertices and faces.

[0109] The computing device 401 may be configured to generate images in a manner similar to the computing device 201. More specifically, the computing device 401 may be configured to generate images associated with the pose of the first device 402a by back-projecting the content of the image associated with the pose of the at least one second device 402b from the 2D screen space to a position in the 3D world space using an inverse of the view projection matrix of the at least one second device, and then applying the view projection matrix of the first device 402a to the position of the content in the 3D world space so that the image of the first device 402a can be reprojected into the 2D screen space. This may be further described as inverting the view projection matrix of the at least one second device 402b and multiplying the matrix with the x, y, z coordinates of each pixel in the 2D screen space (where the z coordinate corresponds to and is determined by the depth information, which may be included in a depth buffer, G-buffer, depth map, or other scene geometry representation, as described above). This returns each pixel to its original x, y, z location in 3D world space. Each original x, y, z location in 3D world space is then multiplied by the view projection matrix of the first device 402a so that the image of the first device 402a can be reprojected into 2D screen space with the correct viewpoint. Inpainting can then be applied, as described below.

[0110] As described in the previous paragraph, the backprojection of the image associated with the pose of at least one second device and the reprojection of the image associated with the pose of the first device can be performed using either pixel reprojection or mesh reprojection, as described herein.

[0111] If the at least one second device comprises a plurality of second devices 402b-402h, the computing device 401 may be configured to render an image of each second device such that a plurality of images are rendered, determine a pose of each second device, and select the second device 402b having a pose closest to the pose of the first device 402a. The computing device may then be configured to generate an image associated with the first device 402a based on the image of the selected second device 402b and at least one further image associated with another second device. It should be understood that the selected second device 402b is not necessarily the device closest to the pose of the first device 402a, but instead, the selected second device may be a device that closely matches the pose of the first device 402a.

[0112] The computing device 401 may be configured to apply the same backprojection and reprojection steps described above to generate the image, after which inpainting can be applied, as described below.

[0113] Since the generated image of the first device 402a has a different viewpoint than the rendered image of the second device 402b on which it is based, deocclusions may appear in the generated image. Deocclusions can be avoided or mitigated using mesh reprojection as described herein, where the final pixel location or pixel color is determined via interpolation. Additionally or alternatively, deocclusions can be filled in real-time using an inpainting technique, or a combination of inpainting techniques. In one example, the computing device 401 is configured to perform inpainting to fill the deocclusion regions in the generated image associated with the first device using information from an image associated with at least one second device. This means that the inpainting may be based on the same image of the second device 402b used for backprojection, or on other rendered images or on some other images corresponding to other second devices 402b-402h.

[0114] The inpainting may be based on surface normal information of the at least one second device image, in particular it may involve extracting surface normal information from the at least one second device image and applying it to the generated image.

[0115] Additionally, albedo and / or ambient occlusion information of an image associated with at least one second device may be used as inpainting, where the albedo and / or ambient occlusion information of an image on which the generated image is based is extracted and applied to the generated image to fill in the disocclusion. As with surface normal information, the albedo and / or ambient occlusion information may be extracted from another rendered image different from the rendered image on which the generated image is based.

[0116] As mentioned above, surface normal information, albedo, and / or ambient occlusion information form part of the G-buffer in a technique referred to as deferred rendering. Thus, in one example, inpainting is based on the G-buffer of either the rendered image on which the generated image is based, and / or other rendered images.

[0117] By using G-buffer, albedo, normal, and / or ambient occlusion information, as described above, deocclusion can be filled in real-time and images can be blended to account for subtle changes in lighting between the images. By using albedo, normal, and ambient occlusion information, the exact effect that the lighting of the scene has on each pixel can be determined and taken into account, resulting in a more seamless blend between images.

[0118] In one example, the computing device 401 is configured to apply additional or alternative inpainting techniques, for example, a screen crack filling method can be applied to fill small disocclusions a few pixels wide by using valid neighboring pixels to guess the most likely pixel value.

[0119] The above example is particularly useful when there are multiple devices or user devices, as shown in Figures 4 and 10. Here, there may be multiple user devices 402a-402h simultaneously displaying rendered content of a performance or show on a stage 403 in real time. Rather than rendering images in real time for each user device (which would require a huge amount of computational power and tax the network), images can be generated by a computing device based on rendered images of adjacent viewpoints (requiring fewer computational resources), as described above.

[0120] A method 500 for generating an image for display on a first device will now be described with reference to Figure 5. All alternatives, combinations or variations described with reference to Figure 4 may also be applied to the method 500 of Figure 5. The method 500 may be performed by a computing device as described in relation to Figure 4.

[0121] 5 discloses various operations of a method 500. The method 500 includes determining a pose of a first device having a display 501 and identifying at least one second device having a display 502, the at least one second device having a different pose than the first device. The method 500 further includes rendering (503) an image associated with the pose of the at least one second device and generating (504) an image associated with the pose of the first device based on the image associated with the pose of the at least one second device.

[0122] The method 500 may further include generating an image associated with a pose of the first device by using depth and / or geometry information of an image associated with a pose of the at least one second device, or other geometric representation of the scene.

[0123] In one example, method 500 may include generating an image associated with a pose of the first device by back-projecting content of an image associated with the at least one second device pose from the 2D screen space to a position in the 3D world space using an inverse of a view projection matrix of the at least one second device, and applying a view projection matrix of the first device to the position of the content in the 3D world space such that the image of the first device can be reprojected into the 2D screen space.

[0124] As described in the previous paragraph, the backprojection of the image associated with the pose of at least one second device and the reprojection of the image associated with the pose of the first device can be performed using either pixel reprojection or mesh reprojection as described herein.

[0125] In one example, the at least one second device includes a plurality of second devices with different poses, and the method 500 includes rendering an image associated with each second device such that a plurality of images are rendered, determining a pose for each of the plurality of second devices, selecting a second device that is closest in matching the pose of the first device, and generating an image associated with the first device based on the image of the selected second device and at least one further image associated with the other second device. In this example, the method 500 may include generating an image associated with the pose of the first device by back-projecting content of an image associated with the pose of the selected second device from the 2D screen space to its position in the 3D world space using an inverse of a view projection matrix of the second device, and applying the view projection matrix of the first device to the position of the content in the 3D world space to enable the image of the first device to be reprojected into the 2D screen space. Also, where the image reprojection can be performed using pixel reprojection or mesh reprojection as described herein.

[0126] The method 500 may further include inpainting information for filling the disoccluded regions in the generated image associated with the first device, the inpainting being based on the image associated with the at least one second device. The inpainting may be based on surface normal information of objects in the image associated with the at least one second device. The surface normal information may be extracted from the image associated with the at least one second device and applied to the generated image. The inpainting may further include albedo and ambient occlusion information of the image associated with the at least one second device. The albedo and ambient occlusion information may be extracted from the image associated with the at least one second device and applied to the generated image.

[0127] Another example will now be described with reference to Fig. 6. Fig. 6 shows a computing device 601 for generating an image. The computing device may be a server, a plurality of servers, a compositing server, a rendering server, a dedicated server, a computer, or other hardware with computing capabilities. The computing device 601 is configured to receive a first image depicting an object from a viewpoint, and generate a new image based on depth information and / or geometry information of the first image, the new image depicting the object from a different viewpoint than the first image, and perform inpainting to fill the unoccluded regions in the new image. The inpainting is based on surface normal information of the object in the first image.

[0128] The computer device can generate a new image based on the depth and / or geometry information of the first image according to any example or combination of examples described herein. Additionally, the computer device can perform inpainting according to any example or combination of examples described herein.

[0129] A method 700 for generating an image will now be described with reference to Fig. 7. Any of the alternatives, combinations or variations described with reference to Fig. 6 may also be applied to the method 700 of Fig. 7. The method 700 may be performed by a computing device as described in relation to Fig. 6.

[0130] The method 700 includes receiving a first image depicting an object from a viewpoint (701), generating a new image based on depth and / or geometry information of the first image, the new image depicting the object from a different viewpoint than the first image, and performing inpainting to fill de-occluded regions in the new image (703), the inpainting being based on surface normal information of the object in the first image.

[0131] As illustrated in the above examples, the methods of the present disclosure may be performed by a computing device. These methods may be executed upon receipt of suitable computer-readable instructions that may be embodied in a computer program executed on the computing device. Such a computing device 800 is illustrated in FIG. 8, which may include a processing unit 810 and a memory 811. The computing device 800 may further include an input / output (I / O) interface 812. The processing unit 810 may include any suitable processor and / or combination of processors. For example, the processing unit 810 may include one or more of a central processing unit (CPU) and a graphical processing unit (GPU). The memory 811 may include volatile memory and / or non-volatile / persistent memory. The memory 811 may be used to store various data, such as, for example, an operating system, instructions executed by the processing unit (e.g., in the form of software executed by the processing unit), configuration information associated with the computing device 800, and / or data associated with images to be rendered and / or generated. The memory may include instructions executable by the processing unit 810 such that the computing device is operable to perform any of the methods described herein, in particular method 300, method 500, and / or method 700.

[0132] At least the processing unit 810 can be connected to an input / output (I / O) interface 812. The I / O interface 812 facilitates communication with one or more other devices, network nodes, or modules in the network. For example, the I / O interface 812 is operable to transmit and / or receive communications to and from other devices in the network. In some examples, the I / O interface 812 is operable to transmit and / or receive communications over an air interface. For example, the I / O interface 812 can include a transmitter and / or receiver for transmitting and / or receiving wireless communications (e.g., radio frequency signals). In some examples, the I / O interface 812 can include a transceiver configured to receive and transmit wireless communications (e.g., radio frequency signals). In some examples, the I / O interface 812 is additionally or alternatively operable to communicate over one or more wired connections.

[0133] Optionally, device 800 may further include a display. For example, device 800 may include a display for displaying information to a user of device 800. The information may be rendered content / images or generated images created according to examples described herein. The display may comprise any suitable electronic display. The display may be coupled to at least processing unit 810. Processing unit 810 may generate display signals that are sent to the display to cause the display to display information.

[0134] It will be understood that the embodiments of the present disclosure can be realized in the form of hardware, software, or a combination of hardware and software. Such software can be stored in the form of volatile or non-volatile storage, such as, for example, a storage device such as a ROM, whether erasable or rewritable, or in the form of memory, such as a RAM, memory chips, devices, integrated circuits, or on an optically or magnetically readable medium, such as a CD, DVD, magnetic disk, magnetic tape, etc. It will be understood that the storage device and storage medium are machine-readable storage embodiments suitable for storing a program or programs that, when executed, implement the embodiments of the present disclosure. Thus, the embodiments provide a program including code for implementing the system or method claimed in any of the preceding claims, and a machine-readable storage for storing such a program. Furthermore, the embodiments of the present disclosure can be transmitted electronically via any medium, such as a communication signal carried via a wired or wireless connection, and the embodiments encompass it as appropriate.

[0135] Examples of how the above-described computer devices, systems and methods may be implemented are described with reference to FIGS.

[0136] FIG. 9 illustrates a system 900 including a network of components that can communicate with each other via wired and / or wireless connections using TCP / IP messaging protocol. The components of the system 900 include a compositing server 901 that serves as a central hub of the system, a camera 902 configured to communicate with the main compositing server 901, and a tracker 903 configured to track the pose and, optionally, the unique information of the camera 902. The tracker 903 may be a gyroscope, accelerometer, and / or GPS attached to the camera 902. It should be understood that instead of a tracker attached to the camera, a simultaneous localization and mapping (SLAM) technique or other tracking technique can be used to track the pose and unique information of the camera. The components of the system 900 further include a dedicated server 904 configured to receive the tracked camera pose and, optionally, the unique information as tracking information. The dedicated server 904 is further configured to record the tracking information and transmit it to the compositing server 101. The system also includes a rendering server 905 configured to render an image of the virtual content and transmit the rendered image to the compositing server 101. It should be understood that the compositing server 901, the dedicated server 904 and the rendering server 905 can be implemented on a single server or can be distributed across multiple servers. The system 900 further comprises a display 906, e.g., an LED display, for projecting or displaying the rendered image.

[0137] In use, the pose and optionally the specific information of the camera 902 are tracked using the tracker 903. This means that the compositing server 901 (or one of the other servers) is provided with a recent history of the pose and optionally the specific information of the camera 902. Based on this, future camera poses can be predicted. In one example, multiple or a series of future camera poses can be predicted along a predicted motion trajectory. The rendering server 905 then renders an image for each predicted camera pose. The number of predicted poses can be determined based on the available computing power while maintaining real-time rendering speed.

[0138] In one example, only a single camera pose may be predicted.

[0139] The rendered image or images may contain virtual content, each of which is associated with its camera pose and, if necessary, identity information, and information describing the geometry and depth of the image. For example, in deferred rendering, the information describing the depth of an image may be contained in a G-buffer, G-buffers, or more specifically, a depth buffer.

[0140] When an image or multiple images are rendered, the tracker 903 determines the current or actual pose of the camera 902. If multiple images are rendered, the compositing server 901 (or one of the other servers) determines which of the predicted poses is closest to the actual pose of the camera 902 and selects the image associated with the selected predicted pose as forming the basis for a new composite or generated image that corresponds to the actual pose or viewpoint of the camera 902. If a single image is rendered, this image becomes the selected image.

[0141] The newly generated image is created using the depth and / or geometry information of the selected image or other scene geometry representation. The depth information may be a depth buffer, such as used in deferred rendering, or a depth map, such as used in other rendering techniques. Generating an image using the depth and / or geometry information of the selected image is described in more detail herein.

[0142] The content of the selected image is back-projected from 2D screen space to its location in 3D world space using the inverse of the view projection matrix of the selected camera pose. The view projection matrix of the actual camera pose is then applied to the content's location in 3D world space so that the image of the actual camera pose can be reprojected into 2D screen space. This can be further described as inverting the view projection matrix of the camera pose associated with the selected image, and multiplying that matrix by the x, y, z coordinates of each pixel in the 2D screen space (where the z coordinate corresponds to or is determined by depth information, which may be contained in a depth buffer, G-buffer, depth map, or other scene geometry representation as described above). This returns each pixel to its original x, y, z location in 3D world space. Each original x, y, z location in 3D world space is then multiplied by the view projection matrix of the actual camera pose so that the image of the actual pose can be reprojected into 2D screen space with the appropriate viewpoint.

[0143] As described in the previous paragraph, the backprojection of the image associated with the selected camera pose and the reprojection of the image associated with the actual camera pose can be performed using either pixel reprojection or mesh reprojection as described herein.

[0144] Because the newly generated images are from the perspective of the current or actual camera pose, the perspective and parallax effects will look convincing and realistic when displayed on the display 906. As described above, generating images based on already rendered images requires less computational power than rendering a new image. Additionally, latency is reduced since generating a new image is much faster than rendering a new image.

[0145] The newly generated image may be created by a compositing server 901, a rendering server 905, a dedicated server 904, or any of the computing devices described herein.

[0146] It should be understood that when multiple camera poses are predicted, the generated image need not be based on an image associated with a predicted pose that is closest to the actual pose of the camera 902. The generated image can be based on an image associated with a predicted pose that is not closest to the actual pose of the camera 902, but is instead close to it. Alternatively, the generated image need not be based on an image associated with a predicted pose, but may be based on an image associated with the last known pose of the camera 902, or one of the last known poses. In this example, the compositing server 901 uses a history of camera poses when it does not predict a camera pose, but instead generates an image for the actual camera pose. This is particularly useful when an image is rendered and the camera pose has changed slightly.

[0147] Since the generated image has a different perspective than the rendered image on which it is based, disocclusions may appear in the generated image. Disocclusions can be avoided or mitigated using mesh reprojection as described herein, and the final pixel location, or pixel color, is determined via interpolation. Additionally or alternatively, disocclusions can be filled in real-time using inpainting techniques. The inpainting technique may be to use a screen crack filling method, whereby small disocclusions of a few pixels wide can be filled by using valid neighboring pixels to guess the most appropriate pixel value. Another inpainting technique that can be applied to the generated image is to use normals of the selected image and / or other rendered images. Normals are the surface normals of objects and can be described as normal information. Additionally, albedo and / or ambient occlusion information of the selected image and / or other rendered images (predicted or past) can be used as described with reference to other examples herein.

[0148] By using G-buffer, albedo, normal, and / or ambient occlusion information, deocclusion can be filled in real-time and images can be blended to account for subtle changes in lighting between images. By using albedo, normal, and ambient occlusion information, the exact effect that the scene's lighting has on each pixel can be determined and taken into account, resulting in a more seamless blend between images.

[0149] It should be understood that any combination of inpainting techniques may be applied or used in the generated image.

[0150] The generated image is then displayed on the display 906, and the camera 902 can capture or photograph the display 906, as well as the actors and equipment performing the scene positioned in front of the display 906. Because the viewpoint of the camera 902 matches the viewpoint of the generated image displayed on the screen, the generated image can be created faster than rendering a new image. Furthermore, as described herein, generating an image requires less computational power than rendering a new image.

[0151] Another example of how the computer devices, systems and methods described herein may be implemented is described with reference to FIG.

[0152] FIG. 10 shows a computing device 1001 for rendering and transmitting images to a user device 1002. The user device 1002 may be a mobile phone, a tablet, a head mounted display, augmented reality glasses, or a similar computing device having a display configured to display the rendered content. The user devices may simultaneously display the rendered content of the performance or show, so that the users of the user devices form part of the audience. The user devices may be spaced around a real or virtual stage 1003, as shown in FIG. 10, or may be located to one side of a stage, virtually or actually, in a more traditional sense. The user devices may be static or movable, relative to the stage or to each other. The stage 1003 may be of any shape.

[0153] In this example, the pose of the first user device 1002a is determined, and then the second user device 1002b is identified. The second user device 1002b may be the user device among the multiple user devices that has the closest pose to the pose of the first user device 1002a. Then, an image is rendered or has been rendered for the second user device 1002b. The image of the second user device 1002b is of a viewpoint corresponding to the pose of the second user device 1002b. To generate the image of the first user device 1002a, the image of the second user device 1002b can be used, similar to the example shown in FIG. 9. More specifically, the newly generated image is created using depth information and / or geometry information of the rendered image of the second user device 1002b, or other suitable scene geometry representation. The depth information may be a depth buffer used in deferred rendering, or a depth map used in other rendering techniques. Generating an image using depth and / or geometry information of the rendered image of the second user device 1002b will now be described in further detail.

[0154] The image content associated with the user device 1002b is back-projected from the 2D screen space to a location in the 3D world space using the inverse of the view projection matrix of the second user device 1002b. The view projection matrix of the first device 1002a is then applied to the location of the content in the 3D world space, and the image of the first user device 1002a can be reprojected into the 2D screen space. This can be further described as inverting the view projection matrix of the second user device 1002b and multiplying that matrix with the x, y, z coordinates of each pixel in the 2D screen space (where the z coordinate corresponds to or is determined by the depth information, which may be included in a depth buffer, G-buffer, depth map, or other scene geometry representation, as described above). This returns each pixel to its original x, y, z location in the 3D world space. The location of each original x, y, z location in the 3D world space is then multiplied with the view projection matrix of the first user device 1002a so that the image of the first user device 1002a can be reprojected into 2D screen space with the correct perspective.

[0155] As described in the previous paragraph, the backprojection of the image associated with the pose of the second user device and the reprojection of the image associated with the pose of the first user device can be performed using either pixel reprojection or mesh reprojection as described herein.

[0156] The same inpainting method as described with reference to Figure 9 or any of the other examples described herein can then be applied to the image of the first user device 1002a. In particular, normal, albedo, and / or ambient occlusion information can be extracted from the rendered image of the second user device 1002b and / or extracted from the rendered images of other user devices, such as other nearby user devices 1002h. The extracted normal, albedo, and / or ambient occlusion information can then be applied to the generated image of the first user device 1002a to fill in the disocclusions.

[0157] It should be understood that the example of Figure 9 can also be applied to the use case of Figure 10, where an image associated with the moving user device 1002a can be generated based on a rendered image of its final pose or a pose close to its final pose, and / or a predicted pose.

[0158] In the examples of the present disclosure, it should be understood that generating an image by backprojecting a rendered image and reprojecting a new image as described above does not have to be applied to the entire rendered image, but only to a portion of the rendered image. For example, the portion of the rendered image that is backprojected may have depth values ​​that place pixels far away from the camera or user device, so that small changes in viewpoint between the rendered image and the generated image do not noticeably affect the parallax. In such cases, pixels of the rendered image are reused in the generated image.

[0159] The examples disclosed herein allow for the generation of images or viewpoints based on previously rendered images, regardless of parallax changes. Discontinuities in the generated images can be avoided or reduced by using mesh reprojection as described herein, with final pixel locations, or pixel colors, determined via interpolation. Additionally or alternatively, the generated images can be improved in that they can be made more realistic by applying inpainting based on normals of the rendered image and / or other adjacent rendered images. By generating convincing images with correct perspective and parallax effects as described herein, the computational power required to play or display the scene on a display can be reduced.

[0160] It should be understood that features, integers, properties or groups described in connection with a particular aspect, embodiment or example of the present disclosure are applicable to other aspects, embodiments or examples described herein, unless inconsistent therewith. All of the features disclosed in this specification (including the accompanying claims, abstract and drawings), and / or all of the steps of a method or process similarly disclosed, may be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive. The present disclosure is not limited to the details of the foregoing examples.

Claims

1. A computer device for generating images configured to be captured by a camera, Determine an image to be rendered associated with at least one camera pose, and then render the said image. We receive information about the current camera pose, and then, A computer device configured to generate an image associated with the current camera pose based on an image rendered in association with at least one camera pose.

2. The computer device according to claim 1, wherein the viewpoint of the image associated with the current camera pose is different from the viewpoint of the rendered image of at least one camera pose.

3. The computer device according to claim 1, configured to generate an image associated with the current camera pose based on depth information and / or geometric information of an image to be rendered in association with at least one camera pose.

4. The computer device according to claim 1, wherein at least one camera pose is at least one predicted camera pose.

5. The computer device according to claim 1, wherein at least one camera pose is the previous camera pose.

6. The computer device uses the inverse of the view projection matrix of at least one camera pose to backproject the content of the image associated with at least one camera pose from two-dimensional screen space to its position in three-dimensional world space, and then, By applying the view projection matrix of the current camera pose to the position of the content in 3D world space, the image of the current camera pose can be projected into 2D screen space. The computer device according to claim 1, configured to generate an image associated with the current camera pose.

7. The computer device further divides the image associated with at least one camera pose into a mesh, The mesh is warped and moved based on the depth map of the aforementioned image, and a back projection is performed by using the inverse of the view projection matrix for at least one camera pose, and then, The computer device according to claim 6, configured to project an image into a two-dimensional screen space by applying a view projection matrix of the current camera pose.

8. At least one camera pose includes multiple camera poses. The computer device renders multiple images, each associated with a corresponding camera pose. Based on the received information regarding the current camera pose, the system selects the camera pose that is closest to the current camera pose from among several camera poses when matching the current camera pose. The computer device according to claim 1, configured to generate an image associated with the current camera pose based on a selected camera pose and at least one further image associated with a plurality of camera poses.

9. The computer device uses the inverse of the view projection matrix of the selected camera pose to backproject the content of the image associated with the selected camera pose from the two-dimensional screen space to its position in the three-dimensional world space, and then, By applying the view projection matrix of the current camera pose to the position of the content in 3D world space, the image of the current camera pose can be projected into 2D screen space. The computer device according to claim 8, configured to generate an image associated with the current camera pose.

10. The computer device further divides the image associated with the selected camera pose into a mesh, Based on the depth map of the aforementioned image, the mesh is warped and moved, and a back projection is performed by using the inverse of the view projection matrix of the selected camera pose, and then, The computer device according to claim 9, configured to project an image into a two-dimensional screen space by applying a view projection matrix of the current camera pose.

11. It is further configured to perform inpainting to fill in the unoccluded areas in the generated image associated with the current camera pose, The computer apparatus according to claim 1, wherein the inpainting is based on an image associated with at least one camera pose.

12. The computer apparatus according to claim 11, wherein the inpainting is based on surface normal information of an object in an image associated with at least one camera pose.

13. The computer apparatus according to claim 12, wherein the inpainting is further based on albedo and ambient occlusion information of an image associated with at least one camera pose.

14. A system comprising a computer device for generating an image, a display for displaying the rendered image, and a camera configured to capture the rendered image displayed on the display, The computer device is configured to determine multiple camera poses and render images for each camera pose. The camera is equipped with at least one sensor for detecting the current camera pose, and the camera is configured to transmit its current pose to a computer device. The computer device further, When matching the current camera pose, select the closest determined camera pose from among several camera poses. It is configured to generate an image associated with the current camera pose based on at least one image from a set of multiple determined poses. The system is configured such that the display shows the generated image and captures the generated image while the camera is in its current pose.

15. A computer device that generates an image for display on a first device, Computer equipment, Determine the pose of the first device having a display, Identify at least one second device having a display, and the at least one second device having a different pose from the first device. Render images associated with the pose of at least one second device, A computer device configured to generate an image associated with a pose of a first device based on an image associated with a pose of at least one second device.

16. The computer device according to claim 15, configured to generate an image associated with a pose of a first device by using depth information and / or geometric information of an image associated with a pose of at least one second device.

17. The computer device uses the inverse of the view projection matrix of at least one second device to backproject the content of the image associated with the pose of at least one second device from two-dimensional screen space to its position in three-dimensional world space, and, By applying the view projection matrix of the first device to the location of the content in three-dimensional world space, the image of the first device can be projected onto two-dimensional screen space. The computer device according to claim 15, configured to generate an image associated with a pose of a first device.

18. The computer device further divides the images associated with the pose of at least one second device into a mesh, The mesh is warped and moved based on the depth map of the aforementioned image, and backprojection is performed by using the inverse of the view projection matrix associated with the pose of at least one second device, and then, The computer device according to claim 17, configured to project an image into a two-dimensional screen space by applying a view projection matrix of the pose of a first device.

19. At least one second device includes multiple second devices with different poses, The computer device renders the image associated with each second device, so that multiple images are rendered. Determine the pose of each of the multiple second devices, select the closest second device when matching it to the pose of the first device, and then, The computer device according to claim 15, configured to generate an image associated with a first device based on an image of a selected second device and at least one further image associated with other second devices.

20. The computer device uses the inverse of the view projection matrix of the second device to backproject the image content associated with the selected pose of the second device from the two-dimensional screen space to its position in the three-dimensional world space, and then, The computer device according to claim 16, configured to generate an image associated with the pose of a first device by applying the view projection matrix of a first device to the position of content in three-dimensional world space so that an image of the first device is projected onto two-dimensional screen space.

21. The computer device further divides the image associated with the pose of the selected second device into a mesh, Based on the depth map of the aforementioned image, the mesh is warped and moved, and a back projection is performed by using the inverse of the view projection matrix associated with the pose of the selected second device, and then, The computer device according to claim 20, configured to project an image into a two-dimensional screen space by applying a view projection matrix of a first device.

22. Further configured to perform inpainting information to fill in the unoccluded regions in the generated image associated with the first device, The computer apparatus according to claim 15, wherein the inpainting is based on an image associated with at least one second device.

23. The computer apparatus according to claim 22, wherein the inpainting is based on surface normal information of an object in an image associated with at least one second device.

24. The computer apparatus according to claim 23, wherein the inpainting is further based on albedo and ambient occlusion information of an image associated with at least one second device.

25. A computer device for generating images, Computer equipment, A first image is received that depicts an object from a certain viewpoint. A new image is generated based on the depth information and / or geometric shape information of the first image, and the new image depicts the object from a different viewpoint than the first image. Perform inpainting to fill in the de-occluded areas in the new image. This inpainting is performed by a computer device configured to be based on surface normal information of a first image or another image.

Images