Determining a point of a three-dimensional representation of a scene

Abstract

Determining a point of a three-dimensional representation of a scene There is described a method of determining a point of a three-dimensional representation, the method comprising: determining a first point of the representation with a first capture device; identifying a second point of the representation; determining a capture device to be used for capturing the second point in dependence on the first capture device; and capturing the second point using the determined capture device.

Description

[0001] Determining a point of a three-dimensional representation of a scene

[0002] Field of the Disclosure

[0003] The present disclosure relates to methods, systems, and apparatuses for determining a point of a three- dimensional representation of a scene.

[0004] Background to the Disclosure

[0005] Three-dimensional representations of environments are used in many contexts, including for the generation of virtual reality videos, in which depth information for a plurality of points of the representation is used to generate different images for a left eye and a right eye of a user. Typically, substantial processing power is required to determine such a three-dimensional representation, and the file size of files associated with these representations is typically large so that substantial amounts of storage are needed to keep the files and substantial amounts of bandwidth are required to transfer the files.

[0006] Summary of the Disclosure

[0007] According to an aspect of the present disclosure, there is described a method of determining a point of a three-dimensional representation, the method comprising: determining a first point of the representation with a first capture device; identifying a second point of the representation; determining a capture device to be used for capturing the second point in dependence on the first capture device; and capturing the second point using the determined capture device.

[0008] Preferably, the method comprises determining the capture device so as to minimise a number of capture devices required to capture the points of the representation.

[0009] Preferably, determining a capture device to be used for capturing the second point in dependence on the first capture device comprises deleting one or more (e.g. other) points captured by one or more further capture devices.

[0010] Preferably, the method comprises determining the capture device based on a number of points already captured by one or more available capture devices and / or based on a number of points already captured by the first capture device.

[0011] Preferably, the method comprises: determining whether the first capture device is able to capture the second point; capturing the second point with the first capture device if the first device is able to capture the second point; and capturing the second point with a second capture device if the first device is not able to capture the second point.

[0012] Preferably, the method comprises: determining a first quality value associated with the first capture device capturing the second point; capturing the second point with the first capture device if the first quality value exceeds a threshold value; and capturing the second point with a second capture device if the first quality value does not exceed the threshold value.

[0013] According to another aspect of the present disclosure, there is described a method of determining a point of a three-dimensional representation, the method comprising: determining a first quality factor associated with a first capture device that is able to capture the point; determining a second quality factor associated with a second capture device that is able to capture the point; and determining whether to capture the point with the first capture device or the second capture device based on the first quality factor and the second quality factor.

[0014] Preferably, the point is a second point, and the method comprises: determining a first point with the first capture device; and determining whether to capture the second point with the first capture device or the second capture device based on whether the first quality factor exceeds a threshold value. Preferably, the threshold quality factor is associated with a maximum possible quality factor, the maximum possible quality factor being the quality factor associated with a maximum quality capture device.

[0015] Preferably, the threshold quality factor is at least 50%, at least 80%, and / or at least 90% of the maximum possible quality factor.

[0016] Preferably, the method comprises: in response to determining that the first quality factor is below the threshold quality factor, capturing the point using the second capture device.

[0017] Preferably, the method comprises determining the second capture device.

[0018] Preferably, the threshold quality factor depends on the first capture device. Preferably, one or more capture devices are associated with different threshold quality factors.

[0019] Preferably, the threshold quality factor depends on a distance of the point from a viewing zone and / or from a capture device.

[0020] Preferably, the quality value for each capture device is dependent on a distance of that capture device from the point.

[0021] Preferably, the quality value for each capture device is dependent on an angle of the capture device to a normal value of the point and / or an angle of the capture device to an angle of a surface that includes the point.

[0022] Preferably, determining the second capture device comprises: determining a set of capture devices that have a quality factor higher than the threshold quality factor; and determining the capture device in this set that has the highest quality factor.

[0023] Preferably, the method comprises determining the (e.g. second) capture device as the capture device that provides the maximum quality factor.

[0024] Preferably, the method comprises determining the (e.g. second) capture device based on a priority list associated with the capture devices.

[0025] According to another aspect of the present disclosure, there is described a method of determining a point of a three-dimensional representation, the method comprising: identifying a point of the representation; identifying a plurality of capture devices capable of capturing points of the representation; determining a capture device to be used for capturing the point in dependence on a priority list associated with the capture devices; and capturing the point using the determined capture device.

[0026] Preferably, the priority list is associated with a viewing zone. Preferably, a highest priority capture device is located towards (e.g. at) a centre of the viewing zone. Optionally, the capture device is within 20%, within 10%, and / or within 5% of the centre of the viewing zone (e.g. relative to a length between a centre of the viewing zone and a side of the viewing zone). Preferably, a plurality of second highest priority capture devices are located at the centres ef faces of the viewing zone.

[0027] Preferably, the method comprises identifying a plurality of priority lists. Preferably, each of the priority lists is associated with a different region of the scene.

[0028] Preferably, the priority list is determined based on a number of points that can be captured by each of the priority devices.

[0029] Preferably, the method comprises: in a first step, determining a number of points for which each capture device exceeds a / the threshold quality factor; and in a second step, determining the priority list based on said number of points.

[0030] Preferably, the priority list is based on one or more capabilities of the capture devices.

[0031] Preferably, the priority list is dependent on a user input. Preferably, the first point and the second point are adjacent points of the three-dimensional representation.

[0032] Preferably, the first point and the second point are both in a first region of the three-dimensional representation. Preferably, the separation between the first point and the second point is less than 10% of a maximum dimension of the representation, preferably less than 5%, less than 2%, and / or less than 1 %.

[0033] Preferably, each capture device is arranged to capture a plurality of sub-points associated with the point, the sub-points being associated with an angular spread, wherein the quality value for each capture device is dependent on an area bounded by the sub-points captured by that capture device.

[0034] Preferably, the three-dimensional representation is associated with a viewing zone, the viewing zone comprising a subset of the scene and / or the viewing zone enabling a user to move through a subset of the scene. Preferably, the user is able to move within the viewing zone with six degrees of freedom (6DoF).

[0035] Preferably, the viewing zone has a volume of less than 50% of the volume of the scene, less than 20% of the volume of the scene, and / or less than 10% of the volume of the scene.

[0036] Preferably, the viewing zone has, or is associated with, a volume, preferably a real-world volume, of less than five cubic metres (5m3), less than one cubic metre (1 m3), less than one-tenth of a cubic metre (0.1 m3) and / or less than one-hundredth of a cubic metre (0.01 m3).

[0037] Preferably, the three-dimensional representation comprises a point cloud.

[0038] Preferably, the method comprises storing the three-dimensional representation and / or outputting the three- dimensional representation. Preferably, the method comprises outputting the three-dimensional representation to a further computer device.

[0039] Preferably, the method comprises generating an image and / or a video based on the three-dimensional representation.

[0040] Preferably, the method comprises forming one or more two-dimensional representations of the scene based on the three-dimensional representation. Preferably, the method comprises comprising forming a two- dimensional representation for each eye of a viewer.

[0041] Preferably, the point is associated with one or more of: a location; an attribute; a transparency; a colour; and a size.

[0042] Preferably, the point is associated with an attribute for a right eye and an attribute for a left eye.

[0043] Preferably, the scene comprises one or more of: an extended reality (XR) scene; a virtual reality (VR) scene; an augmented reality (AR) scene; and a mixed reality (MR) scene.

[0044] According to another aspect of the present disclosure, there is described a system for carrying out the aforesaid method, the system comprising one or more of: a processor; a communication interface; and a display.

[0045] According to another aspect of the present disclosure, there is described an apparatus for determining a point of a three-dimensional representation, the apparatus comprising: means for (e.g. a processor for) determining a first point of the representation with a first capture device; means for (e.g. a processor for) identifying a second point of the representation; means for (e.g. a processor for) determining a capture device to be used for capturing the second point in dependence on the first capture device; and means for (e.g. a processor for) capturing the second point using the determined capture device.

[0046] According to another aspect of the present disclosure, there is described an apparatus for determining a point of a three-dimensional representation, the apparatus comprising: means for (e.g. a processor for) determining a first quality factor associated with a first capture device that is able to capture the point; means for (e.g. a processor for) determining a second quality factor associated with a second capture device that is able to capture the point; and means for (e.g. a processorfor) determining whetherto capture the point with the first capture device or the second capture device based on the first quality factor and the second quality factor.

[0047] According to another aspect of the present disclosure, there is described an apparatus for determining a point of a three-dimensional representation, the apparatus comprising: means for (e.g. a processor for) identifying a point of the representation; means for (e.g. a processor for) identifying a plurality of capture devices capable of capturing points of the representation; means for (e.g. a processor for) determining a capture device to be used for capturing the point in dependence on a priority list associated with the capture devices; and means for (e.g. a processor for) capturing the point using the determined capture device.

[0048] Any feature in one aspect of the disclosure may be applied to other aspects of the invention, in any appropriate combination. In particular, method aspects may be applied to apparatus aspects, and vice versa.

[0049] Furthermore, features implemented in hardware may be implemented in software, and vice versa. Any reference to software and hardware features herein should be construed accordingly.

[0050] Any apparatus feature as described herein may also be provided as a method feature, and vice versa. As used herein, means plus function features may be expressed alternatively in terms of their corresponding structure, such as a suitably programmed processor and associated memory.

[0051] It should also be appreciated that particular combinations of the various features described and defined in any aspects of the disclosure can be implemented and / or supplied and / or used independently.

[0052] The disclosure also provides a computer program and a computer program product comprising software code adapted, when executed on a data processing apparatus, to perform any of the methods described herein, including any or all of their component steps.

[0053] The disclosure also provides a computer program and a computer program product comprising software code which, when executed on a data processing apparatus, comprises any of the apparatus features described herein.

[0054] The disclosure also provides a computer program and a computer program product having an operating system which supports a computer program for carrying out any of the methods described herein and / or for embodying any of the apparatus features described herein.

[0055] The disclosure also provides a computer readable medium having stored thereon the computer program as aforesaid.

[0056] The disclosure also provides a signal carrying the computer program as aforesaid, and a method of transmitting such a signal.

[0057] The disclosure extends to methods and / or apparatus substantially as herein described with reference to the accompanying drawings.

[0058] The disclosure will now be described, by way of example, with reference to the accompanying drawings.

[0059] Description of the Drawings

[0060] Figure 1 shows a system for generating a sequence of images.

[0061] Figure 2 shows a computer device on which components of the system of Figure 1 may be implemented.

[0062] Figure 3 shows a method of determining a three-dimensional representation of a scene.

[0063] Figures 4a and 4b show method of determining a point based on a plurality of sub-points.

[0064] Figure 5 shows a scene comprising a viewing zone. Figures 6a and 6b show arrangements of capture devices for determining points of the three-dimensional representation.

[0065] Figure 7 shows a point that can be captured by a plurality of capture devices.

[0066] Figures 8a and 8b shows grids formed by the different capture devices.

[0067] Figure 9 shows a method of determining whether to combine a plurality of sub-points.

[0068] Figure 10 shows a method of determining a capture device to be used for capturing a point of the three- dimensional representation.

[0069] Figures 11 a-11 d show the capturing of different versions of a point by different capture devices.

[0070] Figures 12a and 12b show the capturing of different points by different capture devices.

[0071] Figure 13 shows a method of determining a capture device to be used for capturing a point based on a quality factor associated with the capture device.

[0072] Figure 14 shows a method of determining a capture device to be used for capturing a point based on a quality factor threshold.

[0073] Description of the Preferred Embodiments

[0074] Referring to Figure 1 , there is shown a system for generating a sequence of images. This system can be used to generate, and then display, a representation of an environment, which may comprise a VR environment (or an XR environment).

[0075] The system comprises an image generator 11 , an encoder 12, a transmitter 13, a network 14, a receiver 15, a decoder 16 and a display device 17.

[0076] These components may each be implemented on separate apparatuses. Equally, various combinations of these components may be implemented on a shared apparatus; for example, the image generator 11 , the encoder 12, and the transmitter 13 may all be part of a single image data generation device. Similarly, the receiver 15, the decoder 16, and the display device 17 may all be a part of a single image rendering device.

[0077] Typically, the system comprises at least one encoding computer device (e.g. a server of a content provider) and at least one rendering computer device (e.g. a VR headset).

[0078] Referring to Figure 2, each of the components, and in particular the image generator 11 , the encoder 12, the transmitter 13, the receiver 15, the decoder 16 and the display device 17 is typically implemented on a computer device 20, where, as described above, a plurality of these components may be implemented on a shared computer device.

[0079] Each computer device comprises one or more of: a processor 21 for executing instructions (e.g. so as to perform one or more of the steps of the various methods described below), a communication interface 22 for facilitating communication between computer devices (e.g. an ethernet interface, a Bluetooth® interface, or a universal serial bus (UBS) interface, a memory 23 and / or storage 24 for storing information and instructions (e.g. a random access memory (RAM), a read only memory (ROM), a hard drive disk (HDD) a solid state drive (SSD), and / or a flash memory, and a user interface 25 (e.g. a display, a mouse, and / or a keyboard) for enabling a user to interact with the computer device. These components may be coupled to one another by a bus 25 of the computer device.

[0080] The computer device 20 may comprise further (or fewer) components. In particular, the computer device (e.g. the display device 17) may comprise one or more sensors, such as an accelerometer, a GPS sensor, or a light sensor. These sensors typically enable the computer device to identify an environmental condition and / or an action of wearer of the display device. Turning back to Figure 1 , the image generator 11 is configured to generate a sequence of image data (e.g. a sequence of image frames) to enable the display device 17 to use this image data to display a plurality of images. The image data may comprise one or more digital objects and the image data may be generated or encoded in any format. For example, the image data may comprise point cloud data, where each point has a 3D position and one or more attributes. These attributes may, for example, include, a surface colour, a transparency value, an object size and a surface normal direction. Each attribute may have a value chosen from a continuous range or may have a value chosen from a discrete set.

[0081] The image data enables the later rendering of images. This image data may enable a direct rendering (e.g. the image data may directly represent an image). Equally, the image data may require further processing in order to enable rendering. For example, the image data may comprise three-dimensional point cloud data, where rendering a two-dimensional image using this data requires processing based on a viewpoint of this two-dimensional image.

[0082] The image data may comprise depth map data, where one or more pixels or objects in the image is associated with a depth that is specified by the depth map data. The depth map data may be provided as a depth map layer, separate from an image layer. In some contexts, such as MPEG Immersive Video (MIV), the image layer may instead be described as a texture layer. Similarly, in some contexts, the depth map layer may instead be described as a geometry layer.

[0083] The image data may include a predicted display window location. The predicted display window location may indicate a portion of an image that is likely to be displayed by the display device 17. The predicted display window location may be based on a viewing position (such as a virtual position and / or orientation of the user in a 3D environment) of the user, where this viewing position may be obtained from the display device. The predicted display window location may be defined using one or more coordinates. For example, the predicted display window location may be defined using the coordinates of a corner or center of a predicted display window, and may be defined using a size of the predicted display window. The predicted display window location may be encoded as part of metadata included with the frame.

[0084] The image data for each image (e.g. each frame) may include further information, which may be provided as a part of an image, e.g. as part of the point cloud data, or as separate layers. In particular, the image data may include audio information or haptic feedback information indicating audio or haptics which can accompany displayed visual data. An audio layer or haptic layer may accompany each image, and may be omitted for images where no accompanying audio or haptics are required.

[0085] Similarly, the image data may comprise interactivity information, where the image data may contain or indicate elements with which a user can interact. The interactivity information may, for example, define a behaviour of an element, where a user is able to interact with the element based on this behaviour. The behaviour typically defines a change in an element that occurs as a result of a user interaction where this change may comprise a change in the attributes of the element or in the rendering of the element. As an example, where an image contains a target element, the target element may be arranged to disappear when a user interacts with this element, or to provide feedback indicating that the user has interacted with the target. This interactivity data may be provided as part of, or separately to, the image data.

[0086] The image data may indicate, or may be combinable with, a state of the virtual environment, a position of a user, ora viewing direction of the user. Here, the position and viewing direction may be physical properties of the user in the real-world, or position and viewing direction may also be purely virtual, for example being controlled using a handheld controller. The image generator 11 may, for example, obtain information from the display device 17 that indicates the position, viewing direction, or motion of the user. Equally, the image generator may generate image data such that it can later be combined with this position, viewing direction, or motion, where the image generator may generate a full scene which is only partially viewed by a user depending on the position of that user. In some cases, the generated image may be independent of user position and viewing direction. This type of image generation typically requires significant computer resources such as a powerful GPU, and may be implemented in a cloud service, or on a local but powerful computer. For example, a cloud service (such as a Cloud Rendering Service (CRN)) may reduce the cost per-user and thereby make the image frame generation more accessible to a wider range of users. Here “rendering” refers at least to an initial stage of rendering to generate an image. Further rendering may occur at the display device 17 based on the generated image to produce a final image which is displayed.

[0087] The image generator 11 may, for example, comprise a rendering engine for initially rendering a virtual environment such as a game or a virtual meeting room.

[0088] The encoder 12 is configured to encode frames to be transmitted to the display device 17. The encoder may be implemented using executable software or may be implemented on specific hardware such as an ASIC. In some embodiments, the image generator 11 may transmit raw, unencoded, data through the network 14. However, such transmission typically leads to a high file size and requires a high bandwidth so that it is typically desirable to encode the data prior to the transmission.

[0089] The encoder 12 may encode the image data in a lossless manner or may encode the data a lossy manner. The encoder may apply inter-frame or intra-frame compression based on a currently-encoded frame and optionally one or more previously encoded frames. The encoder may be a multi-layer encoder, such as a low complexity enhancement video codec (LCEVC) enabled encoder.

[0090] Where the generated frames comprise depth map data, the encoder 12 may perform layered encoding on each instance of image data (e.g. each frame) to generate an encoded frame comprising a base depth map layer and an enhancement depth map layer. Encoding a depth map in this way may improve compression. In some applications, such as HDR video, depth maps are desirably highly detailed with a bit depth of up to twelve or fourteen bits, which is a significant increase in the data to be transmitted. As a result, providing ways to improve compression of the depth map can make more realistic depth map-based displays viable when performing rendering or transmission of rendered data in real-time. Furthermore, this type of layered encoding makes it easy to drop (and then pick back up) one or more of the layers, which provides flexibility and tools for bandwidth management.

[0091] Layered encoding is also helpful as the final decoder / user device (such as a user display device) can choose whether to process these extra layers. For example, in a non-layered approach, the best the end device (i.e. the receiver, decoder or display device associated with a user that will view the images) can do is determine that it does not have enough resources for a given quality (be it resolution, frame rate, inclusion of depth map) and then signal to the controller / renderer / encoder that it does not have enough resources. The controller then will send future images at a lower quality. In that alternative scenario, the end device still unfortunately has to process the higher quality data until the lower quality data arrives, if it can process the received images at all.

[0092] In some of the described embodiments, this situation is improved upon because when / if the end device determines for example that it does not have the processing capabilities to handle the highest level of quality, then it can drop and / or choose not to process certain layers. The end device may also signal to the controller that it needs a lower level of quality, but in the meantime the end device can only process the number of layers that it can handle. Therefore, the end device can react to conditions much more quickly.

[0093] In some cases, depth map data may be embedded in image data. In this case, the base depth map layer may be a base image layer with embedded depth map data, and the enhancement depth map layer may be an enhancement image layer with embedded depth map data.

[0094] Alternatively, when the generated images comprise a depth map layer separate from an image layer and multi-layer encoding is applied, the encoded depth map layers may be separate from the encoded image layers. This has the advantage that the encoded depth map layers can be dropped under some conditions while still retaining image layers that can be displayed (albeit with a lower level of realism). For example, the encoded depth map layers can be dropped by a transmitter or encoder when available communication resources are reduced, or can be dropped by an end device which lacks the processing resources to handle the highest level of quality.

[0095] Similarly, if some images comprise an audio base layer, a haptic feedback base layer, an audio enhancement layer or a haptic feedback enhancement layer, these can be processed or dropped flexibly.

[0096] Again similarly, if some images comprise an interactivity data base layer or an interactivity enhancement layer these can be processed or dropped flexibly. For example, certain interactions may only be possible where a threshold bandwidth is available, where complex interactions (e.g. those enabling a conversation with a digital object) may be disabled before less complex interactions (e.g. changing a pixel colour) are disabled.

[0097] Additionally or alternatively, where the image data comprises point cloud data, the encoder may apply a point cloud data encoding technique such as described in European patent application EP21386059.6, which is incorporated herein by reference. Such a point cloud encoder may act as a base encoder for a layered encoding technique such as LCEVC or VC-6. Notably LCEVC and VC-6 techniques encode and decode a layered signal, but are agnostic about the content type of data encoded in the signal. For example, the signal can include textures, video frames, geometry or depth data, meshes, point clouds, rendering attributes or physics engine attributes.

[0098] The transmitter 13 may be any known type of transmitter for wired or wireless communications, including an Ethernet transmitter or a Bluetooth transmitter.

[0099] The transmitter 13 may be configured to make decisions about how to transmit the image data, and / or may provide feedback to the encoder 12 or the image generator 11 . For example, the transmitter may determine available communication resources (e.g. bandwidth) for transmitting image data, and may drop one or more layers from an encoded frame, or indicate to the image generator and / or encoder that image data should be generated and encoded with fewer layers, when insufficient bandwidth is available for transmission of all generated data. As specific examples, the transmitter may be configured to drop a depth map layer, an LCEVC enhancement layer, or a VC-6 enhancement layer from a frame when insufficient communication resources are available.

[0100] The network 14 provides a channel for communication between the transmitter 13 and the receiver 15, and may be any known type of network such as a WAN or LAN or a wireless Wi-Fi or Bluetooth network. The network may further be a composite of several networks of different types. Many users only have access to a network with a bandwidth of 30MBps which can lead to latency jitter when streaming. The required bandwidth and the observed latency can be reduced by means of tactics such as forward-looking rendering and last-millisecond reprojection, which are enabled by improved compression.

[0101] The receiver 15 may be any known type of receiver for wired or wireless communications, including an Ethernet transmitter or a Bluetooth transmitter.

[0102] The decoder 16 is configured to receive and decode an encoded frame. The decoder may be implemented using executable software or may be implemented on specific hardware such as an ASIC.

[0103] The display device 17 may for example be a television screen or a VR headset. The timing of the display may be linked to a configured frame rate, such that the display device may wait before displaying the image. The display device may be configured to perform warping, that is, to obtain a final display window location, adjust a warpable image to obtain a final image corresponding to a final viewing direction of the user, and display the final image. In this regard, the image data is typically arranged to provide a warpable image for which a portion of the image that is displayed at the display device 17 is dependent on a position or orientation of a viewer. The warpable image may then be rendered before a most up to date viewing direction of the user is known. The warpable image may be transmitted to the display device, or the warpable image may be transmitted to a rendering node which is near to the display device, and the display device or rendering node may perform time warping to generate a displayed image portion based on the warpable image and the most up to date viewing direction of the user.

[0104] As mentioned above, a single device may provide a plurality of the described components. For example, a first rendering node may comprise the image generator 11 , encoder 12 and transmitter 13. Additional similar rendering nodes may be included in the system, and may work together to generate the sequence of frames.

[0105] In one case, multiple rendering nodes may each provide separate image data to an image data assembling node; for example, each rendering node may provide a part of a sequence of frames to a frame assembling node.

[0106] For example, the receiver 15, decoder 16 or display device 17 may be configured to assemble parts of image data from multiple sources to generate a sequence of images for display on the display device.

[0107] Alternatively, the image data assembling node may be separate from the receiver 15, decoder 16 and display device 17.

[0108] Additionally or alternatively, multiple rendering nodes may be chained. In otherwords, successive rendering nodes may add to a sequence of image data as it passes from rendering node to rendering node, and eventually a complete sequence of image data is then provided to the receiver 15. Furthermore, each rendering node may obtain components of a render from multiple upstream rendering nodes and / or distribute components of a render to multiple downstream rendering nodes.

[0109] A chain of rendering nodes may be useful for performing different rendering tasks that require different quantities of processing resources, or different frame rates. For example, a company may provide distributed processing in the form of a centralised hub which has abundant processing resources but is distant from users, and peripheral locations which have more scarce processing resources but are closer to users. Expensive but fairly static rendering features such as background lighting or environmental impact on sound may be generated at the central hub (for example using ray tracing), while features that require fewer resources but faster responses or higher frame rates may be generated closer to the user. In other words, the more responsive a rendering feature needs to be, the lower latency it needs between the rendering node which generates the feature and the user display and, in a chain of rendering nodes, the node which generates each rendering feature can be chosen based on a required maximum latency of that feature. On the other hand, if it is expensive to generate a rendering feature, then it may be preferable to generate the feature less frequency and with a higher maximum latency. For example, a static, high-quality background feature may be generated early in the chain of rendering nodes and a dynamic, but potentially lower-quality, foreground feature may be generated later in the chain of rendering nodes, closer to the user device. Here, environmental impact on sound means, for example, a set of surfaces may be constructed where each surface has different sound reflection and absorption properties depending upon material and shape. The frame rates may be matched by creating multiple frames with features generated at the lower frame rate, and combining them with the frames with features generated at the higher frame rate. In a nonlimiting embodiment, a preliminary rendering generates volumetric object data including motion vectors at a first (lowest) frame rate, then produces 2D rendered frames plus depth information for a specific user at a second (higher) frame rate, then transmits video plus depth data to the user device, which produces final frames for display via space warping (depth-based reprojections) at a third (highest) frame rate. One or more of these steps may be performed in combination with the other described embodiments. The viewing position of the user may change as additional rendering tasks are performed at different rendering nodes in the chain. Each or any rendering node may obtain an updated viewing position before performing its respective rendering task.

[0110] Additionally, the system may simultaneously generate multiple sequences of image data for different respective users or different respective display devices. For example, in the context of a VR or AR experience, each user or display device may view a different 3D environment, or may view different parts of a same 3D environment. When using a chain of rendering nodes, each node may serve multiple users or just one user.

[0111] For example, a starting rendering node (e.g. at a centralised hub) may serve a large group of users. For example, the group of users may be viewing nearby parts of a same 3D environment. In this case, the starting node may render a wide zone of view (“field of view”) which is relevant for all users in the large group.

[0112] The starting node may send this wide field of view to a first middle rendering node which renders additional aspects of the 3D environment. These additional aspects may for example be aspects which require less processing power to render, or may be aspects which are specific to individual users of the group. Additionally, the middle rendering node may render features in a smaller field of view than the starting node - this smaller field of view may be relevant to each user rather than the group of users. The first middle rendering node may additionally only serve a smaller number of users (e.g. half of the large group of users), with the remaining users being served by a second middle rendering node which also receives the wide field of view from the starting node.

[0113] The middle rendering node(s) may then send sequences of second partially or fully rendered frames to an end device for each user. The end device may perform further processes such as warping or focal distance adjustments, optionally using depth map data.

[0114] Preferably, each rendering node encodes the partially or fully rendered frames before transmitting them on to a next rendering node or to the receiver 15. This means that the required communication resources can be reduced when the rendering nodes are separated by one or more networks, or more generally are implemented in a distributed system such as a cloud.

[0115] However, each rendering node in a chain is encoding a different partially or fully rendered frame, with different data. Therefore, it may be advantageous for different rendering nodes to use different rendering formats and / or encoding formats. For example, the output from a first rendering node may be point cloud data which logically describes a 3D scene. This point cloud data can be encoded using the techniques of EP21386059.6. A second rendering node may then operate on the point cloud data to generate image data that is more readily displayed by a generic display device, without requiring the display device to model the 3D environment. This image data may be encoded using video coding techniques.

[0116] The chaining of rendering nodes may be extended to arbitrary tree structures, where a rendering node obtains partially rendered frames from more than one preceding rendering node, and generates further partially or fully rendered frames based on the multiple obtained sequences of partially rendered frames.

[0117] For example, a content rendering network (CRN) comprising numerous rendering nodes may be used to serve a volumetric event to a large number of same-time users, such as users participating in a shared virtual environment. Rendering the same event for each user is far more expensive in terms of computation time and power consumption than rendering the volumetric effect once and performing the rendering equivalent of multicasting the volumetric effect for multiple users. For example, each user may have a second rendering node (such as a VR headset), and the network may comprise a central first rendering node. The first rendering node may render the volumetric event, and distribute partially rendered frames depicting the volumetric event to the different second rendering nodes. The second rendering node for each user may then integrate the partially rendered frames depicting the volumetric event into a view of the virtual environment which is currently being shown to each user, based on parameters such as the user’s virtual position.

[0118] The receiver 15, decoder 16 and display device 17 may be consolidated into a single device, or may be separated into two or more devices. For example, some VR headset systems comprise a base unit and a headset unit which communicate with each other. The receiver 15 and decoder 16 may be incorporated into such a base unit.

[0119] In some embodiments, the network 14 may be omitted. For example, a home display system may comprise a base unit configured as an image source, and a portable display unit comprising the display device 17.

[0120] In the event that the decoder 16 or the display device 17 does not or cannot handle one or more layers, the receiver 15 or another transmitter associated with the decoder or display device may send a corresponding layer drop indication back through the network 14. The layer drop indication may be received by each rendering node. A rendering node which generates partially or fully rendered frames for that specific decoder or display device may cease generating the dropped layer. On the other hand, a rendering node which generates partially or fully rendered frames for multiple end devices may disregard a layer drop indication received from one end device (as the dropped layer is still needed for other devices). Alternatively, rendering nodes which serve multiple end devices may record received layer drop indications, and may cease generating the dropped layer only when all end devices served by the rendering node indicate that the layer is to be dropped.

[0121] In preferred examples, the encoders or decoders are part of a tier-based hierarchical coding scheme or format. Hierarchical coding enables frames to be communicated with higher resolution and / or higher frame rate than is possible in single-tier coding schemes. In hierarchical coding, one or more enhancement layers is communicated with base data, where the enhancement layers can be used to up-sample the base data at the decoder, for example providing up-sampling in a spatial ortemporal dimension. When combined with equivalent down-sampling of the original frames and generation of the enhancement layer at an encoder, hierarchical coding can overall provide lossless compression of data, with higher resolution and / or higher frame rate for a given transmission bit rate. Examples of a tier-based hierarchical coding scheme include LCEVC: MPEG-5 Part 2 LCEVC (“Low Complexity Enhancement Video Coding”) and VC-6: SMPTE VC-6 ST-2117, the former being described in PCT / GB2020 / 050695, published as WO 2020 / 188273, (and the associated standard document) and the latter being described in PCT / GB2018 / 053552, published as WO 2019 / 111010, (and the associated standard document), all of which are incorporated by reference herein. However, the concepts illustrated herein need not be limited to these specific hierarchical coding schemes.

[0122] A further example is described in WO2018 / 046940, which is incorporated by reference herein. In this example, a set of residuals are encoded relative to the residuals stored in a temporal buffer.

[0123] LCEVC (Low-Complexity Enhancement Video Coding) is a standardised coding method set out in standard specification documents including the Text of ISO / IEC 23094-2 Ed 1 Low Complexity Enhancement Video Coding published in November 2021 , which is incorporated by reference herein.

[0124] The system describes above is suitable for generating and presenting a representation of a scene, where this scene displays media content to a user. The scene typically comprises an environment, where the user is able to move (e.g. to move their head or to turn their head) to look around the environment and / or to move around the environment. For example, the scene may be a scene of a room in a building, where the user is able to move around the room (e.g. by moving in the real-world and / or by providing an input to a user interface) in orderto inspect various parts of the room. Typically, the scene is a XR (e.g. a VR) scene, where the user is able to move about the scene in three degrees of freedom (3DoF) or six degrees of freedom (6DoF) so as to experience the scene. As has been described with reference to Figure 1 , the image generator 11 may be arranged to determine point cloud data, where each point of the point cloud has a 3D position and one or more attributes. More generally, the image generator (or another component) is arranged to determine a three-dimensional representation of a scene, where this three-dimensional representation is thereafter used to generate two- dimensional images that are presented to a user at the display device 17.

[0125] While the points are typically points of a point cloud, more generally the disclosure extends to any point that is associated with a location and a value. Therefore, the points may, more generally, be considered to be data (or datapoints), which data is associated with a location and a value, and the ‘points’ may comprise polygons, planes (regular or irregular), Gaussian splats, etc.

[0126] Referring to Figure 3, there is described a method of determining (an attribute for) a point of such a three- dimensional representation. The method comprises determining the attribute using a capture device, such as a camera or a scanner. The scene may comprise a real scene, in which attribute values are captured using a camera, or a virtual scene (e.g. a three-dimensional model of a scene), in which attribute values are captured using a virtual scanner.

[0127] Where this disclosure describes ‘determining a point’ it will be understood that this generally refers to determining a point that has a location and an attribute value, where determining the point comprises determining the attribute value and / or storing a point that comprises at least an attribute value and a location value (these values may be indirect values, e.g. where the location is identified relative to another point). Once a plurality of points have been captured, these points can be stored as a three-dimensional representation (e.g. a point cloud) so as to enable the reconstruction of the three-dimensional scene based on this representation.

[0128] Typically, the scene comprises a simulated scene that exists only on a computer. Such a scene may, for example, be generated using software such as the Maya software produced by Autodesk®. The attributes determined using the methods described herein may then depend on virtual objects located within the scene as well as a virtual lighting arrangement used in the scene.

[0129] In a first step 31 , a computer device initiates a capture process for a capture device, the capture process being initiated with an initial azimuth angle (e.g. of 0°) and an initial elevation angle (e.g. of 0°).

[0130] In a second step 32, the computer device causes a point to be captured using the capture device at the current azimuth angle and current elevation angle. Capturing a point typically comprises assigning an attribute value to the point, which attribute value may, for example, be a color of the point and / or a transparency value of the point. Typically, the point has one or more color values associated with each of a left eye and a right eye of a viewer. Capturing the point may also comprise determining a normal value associated with the point, e.g. a normal of a surface on which the point lies. Typically, capturing the point further comprises determining a location of the point, e.g. by determining a distance of the point from the camera.

[0131] In practice, determining the point may comprise sending a ‘ray’ from the capture device and then stepping through a computer model to determine which surface of the computer model is impacted by the ray. The color, transparency, and normal of this surface are then recorded alongside the distance of the surface from the capture device.

[0132] In a third step, 33, the computer device determines whether a point has been captured for the capture device at each azimuth of a range of azimuths and in a fourth step 34, if points have not been captured at each azimuth, then the azimuth angle is incremented and the method returns to the second step 32 and another point is captured. The azimuth angle may, for example, be incremented by between 0.01 ° and 1 ° and / or by between 0.025° and 0.1 °. Typically, the range of azimuth angles is selected to be 360° (i.e. so that the capture device captures points surrounding the entirety of the capture device), but it will be appreciated that other ranges are possible.

[0133] Once a point has been captured for each azimuth, in a fifth step 35, the computer device determines whether a point has been captured for the capture device at each elevation of a range of elevations and in a sixth step 36, if points have not been captured at each elevation, then the azimuth angle is reset to the initial value, elevation angle is incremented and the method returns to the second step 32 and another point is captured. The elevation angles may, for example, be incremented by between 0.01 ° and 1 ° and / or by between 0.025° and 0.1 °. Typically, the range of elevation angles is selected to be 360° (i.e. so that the capture device captures points surrounding the entirety of the capture device), but it will be appreciated that other ranges are possible.

[0134] In a seventh step 37, once points have been captured for each azimuth angle and each elevation angle, the scanning process ends.

[0135] This method enables a capture device to capture points at a range of elevation and azimuth angles. This point data is typically stored in a matrix. The point data may then be used to provide a representation of the scene to a user, e.g. the three-dimensional representation formed by the point data may be processed to produce two-dimensional images for each eye of a user, with these images then being shown to a user via the display device 17 to provide a virtual reality experience to the viewer. By using the captured data, a video can be provided to a viewer that enables the viewer to move their head to look around the scene (while remaining at the location of the capture device).

[0136] It will be appreciated that the capture pattern (or scanning pattern) described with reference to Figure 3 is purely exemplary and that numerous capture patterns are possible. In general, the capture process for each capture device comprises capturing one or more points at one or more azimuth angles and / or one or more elevation angles.

[0137] The ‘points’ captured by the capture device are typically associated with a size, such as a height, a width, or a depth. That is, the points typically relate to two-dimensional planes / pixels and / or three-dimensional voxels. In this regard, there is necessarily some space between the locations of adjacent points (since if the points had no width, then an infinite number of points would be required to capture points at each angle). The size provides points that depict a non-negligible area of the three-dimensional space so that a plurality of points can be fit together to provide a depiction of the scene to a viewer.

[0138] The width and height of each point is typically dependent on the distance of that point from the capture device, where more distant points have a larger width / height. The width and height of each point is typically determined so that when each point is displayed, there is no space between adjacent points (indeed, there may be some overlap between points to ensure that no gaps appear between points). This height / width of each point can be determined at the time of capturing the points, or can be determined or defined after the capture of the points.

[0139] Typically, the points comprise a size value, which is stored as a part of the point data. For example, the points may be stored with a width value and / or a height value. Typically, the minimum width and the minimum height of a point are set by the angle increment of the azimuth angle and the elevation angle respectively. The size may be then specified in terms of this angle increment and / or in terms of this minimum width / minimum height (e.g. as being a multiple of the angle increment). In some embodiments, the size value is stored as an index, which index relates to a known list of sizes (e.g. if the size may be any of 1x1 , 2x1 , 1x2, 2x2, pixels this may be specified by using 3 bits and a list that relates each combination of bits to a size).

[0140] The size may be stored based on an underscan value. In this regard, where an object is very near to the viewing zone it may be captured using an unnecessarily dense arrangement of points. Therefore, certain surfaces or areas of the representation may be associated with an underscan value, which underscan value defines a reduction in the number of points captured as compared to a representation without underscan. The size of the points may be defined so as to indicate this underscan value. In an exemplary embodiment, the underscan value is an integer value between 0 and 3 and the size is stored as a combination of point dimensions (e.g. a width in the range [0,2]) and a height in the range ([0,2]) and an underscan factor (e.g. an underscan factor in the range [0,3]).

[0141] In some embodiments, the width and the height are dependent on the underscan factor. For example, when the underscan factor exceeds a threshold value, the possible height and width values may be limited. In a specific example, when the underscan factor is 3, the width and the height may be limited to the range [0,1]. The size may then be defined as size = underscan*9 + height*3 + width. Such a method provides efficient storage and indication of width, height, and underscan values.

[0142] As shown in Figure 4a, typically, for each capture step (e.g. each azimuth angle and / or each elevation angle), a plurality of sub-points SP1 , SP2, SP3, SP4, SP5 is determined. For example, where the azimuth angle increment is 0.1 ° then for an azimuth angle of 0°, sub-points may be determined at azimuth angles of -0.05°, -0.025°, 0, 0.025°, and 0.05° (and similar sub-points may be determined for a plurality of elevation angles). Attribute values of these sub-points may then be combined to obtain an attribute value for the point. For example, a maximum attribute value of the sub-points may be used as the value for the point, an average attribute value of the sub-points may be used as the value forthe point, and / or a weighted average of the sub-points may be used as the value forthe point. It will be appreciated that numerous other methods for combining the attribute values of the sub-points are possible.

[0143] By determining the attribute of a point based on the attributes of sub-points, the accuracy of the capture process can be increased. While it would be possible to simply reduce the increment of the angle steps to provide a higher resolution scene, by considering sub-points but only storing attributes for points, a balance can be struck between accuracy and file size (since storing every sub-point would lead to a substantial increase in the amount of data that needs storing).

[0144] With the example of Figure 4a, for each point of the three-dimensional representation that is captured by a capture device, this capture device may obtain attributes associated with each of the sub-points SP1 , SP2, SP3, SP4, SP5, combine these attributes to obtain a point attribute, and then store a point with a distance that is an average (e.g. a weighted average) of the distances of the sub-points from the capture device, at the nominal angle of the point, with the point attribute.

[0145] As shown in Figure 4b, where a plurality of sub-points SP1 , SP2, SP3, SP4, SP5 are considered, these points may have different distances from the location of the capture device. In some embodiments, the attributes of the sub-points may be combined in dependence on this distance, e.g. so that sub-points nearer to the capture device have higher weightings.

[0146] However, the possibility of sub-points with substantially different distances raises a potential problem. Typically, in order to determine a distance for a point, the distances for the sub-points are averaged. But where the sub-points have substantially different distances and / or are related to different surfaces in the scene, this may result in the point having a distance that does not correspond to any actual surface in the scene. Therefore, the point may seem to hang in space (e.g. to hang between the front and rear surfaces shown in Figure 4b.

[0147] Similarly, where the attribute values of the sub-points greatly differ, e.g. if the sub-points SP1 and SP2 are white in colour and the sub-points SP3 and SP4 are black in colour, then the attribute value of the point may be substantially different to the attribute value of other points in the scene. In an example, if the scene were composed of black and white objects, the point may appear as a grey point hanging in space between these objects. In some embodiments, the computer device is arranged to aggregate sub-points so as not to create any floating points. For example, the computer device may determine whether the sub-points are spatially coherent by employing a clustering algorithm (e.g. a k-means clustering algorithm). Where the sub-points are spatially coherent (e.g. where a difference in the distance of the sub-points is below a threshold value), these distances may be averaged to obtain a distance for the point. Where the sub-points are not spatially coherent, the sub-points may be processed to ensure that the distance of any point places it upon a surface; for example, in the system of Figure 4b, sub-points SP1 , SP2, and SP3 may be grouped into a first point and sub-points SP4 and SP5 may be grouped into a second point. Since each sub-point is associated with the same capture device and capture angle (all of these sub-points being associated with a capture step that has a particular azimuth angle and elevation angle), these points may be located at the same angle with respect to a capture device. Therefore, to ensure that each sub-point affects the representation considered, the first point (made up of sub-points SP1 , SP2, and SP3) may have a smaller distance value than the second point (made up of sub-points SP4 and SP5) and the first point may be assigned a nonzero transparency value so that the second point can be seen through the first point.

[0148] By capturing points at a plurality of azimuth angles and elevation angles, e.g. using the method described with reference to Figure 3, it is possible to provide a three-dimensional representation of the scene that can later be used to enable a viewer to view the scene from a plurality of angles. More specifically, given the three-dimensional points captured by the capture device, a computer device is able to render a two- dimensional representation (e.g. a two-dimensional image) of the scene for each eye of a viewer so as to provide a representation with an impression of depth. The computer device may render a series of two- dimensional representations to enable the viewer to look around the scene, where the two-dimensional representations are rendered based on an orientation of the viewer’s head. In this way, the determined representation is useable to provide, for example, a virtual reality (VR), mixed reality (MR), augmented reality (AR), and / or extended reality (XR) experience to the viewer.

[0149] To enable such a display, the display device 17 is typically a virtual reality headset, that comprises a plurality of sensors to track a head movement of the user. By tracking this head movement, the display device is able to update the images being displayed to the viewer as the viewer moves their head to look about the scene. Typically, this involves the display device sensing the sensor data to an external computer device (e.g. a computer connected to the display device via a wire). The external computer device may comprise powerful graphical processing units (GPUs) and / or computer processing units (CPUs) so that the external computer device is able to rapidly render appropriate two-dimensional images for the viewer based on the three-dimensional images and the sensor data. It will be appreciated that the use of a combination of a headset and an external device is exemplary. More generally, the processing of data and the rendering of images may be performed by various computer devices; for example, a standalone virtual reality headset may be provided, which headset is capable of processing data and rendering images without any connection to an external computer device.

[0150] In some embodiments, the external computer device may comprise a server device, where the display device 17 may be connected to this server device wirelessly. This enables the two-dimensional images to be streamed from the server to the display device so as to enable the display of high-quality images without the need for a viewer to purchase expensive computer equipment. In other words, operations that require large amounts of computing power, such as the rendering of two-dimensional images based on the three- dimensional representation, may be performed by the server, so that the display device is only required to perform relatively simple operations. This enables the experience to be provided to a wide range of viewers.

[0151] In some embodiments, a first two-dimensional image is provided to the display device 17 (and / or a connected device) and this first image is “warped’ in order to provide an image for viewing at the display device. The warping of the image comprises processing the image based on the sensor data in order to provide an image that matches a current viewpoint of the viewer. By performing the warping at the display device or another local device, the lag between a head movement of the user and an updating of the two- dimensional representation of the scene can be reduced.

[0152] One issue with the above-described method of capturing a three-dimensional representation is that it only enables a viewer to make rotational movements. That is, since the points are captured using a single capture device at a single capture location, there is no possibility of enabling translational movements of a viewer through a scene. This inability to move translationally can induce motion sickness within a viewer, can reduce a degree of immersion of the viewer, and can reduce the viewer’s enjoyment of the scene. However, enabling a viewer to move translationally through a three-dimensional representation that has been captured using a single capture device would lead to holes in the scene wherever a viewer moves away from the capture location of this single capture device (since this movement will cause parts of the scene that were not captured by the capture device to come into view of the viewer)

[0153] Therefore, it is desirable to enable translational movements through the scene while avoiding the display of these holes. To enable such movements, the three-dimensional representation of the scene may be captured using a plurality of capture devices placed at different locations (orthe same capture device placed at different locations). A viewer is then able to move around the scene translationally (e.g. by moving between these locations).

[0154] More generally, by capturing points for every possible surface that might be viewed by a viewer, a three- dimensional representation of a scene may be captured that allows a suitable two-dimensional representation of this scene to be rendered regardless of a location of a viewer (e.g. regardless of where a user is standing within a virtual room).

[0155] This need to capture points for every possible surface (so as to enable movement about a scene) greatly increases the amount of data that needs to be stored to form the three-dimensional representation.

[0156] Therefore, as has been described in the application WO 2016 / 061640 A1 , which is hereby incorporated by reference, the three-dimensional representation may be associated with a viewing zone, a zone of view (ZOV), or a zone of viewpoints (ZVP), where the three-dimensional representation is arranged to enable a user to move about the viewing zone so as to view the scene.

[0157] Figure 5 illustrates such a viewing zone 1 and illustrates how the use of a viewing zone limits the amount of image data that needs to be stored to provide a three-dimensional representation of the scene. With the scene shown in this figure, and the viewing zone 1 shown in this figure, it is not necessary to determine attribute data for the occluded surface 2 since this occluded surface cannot be viewed from any point in the viewing zone. Therefore, by enabling the user to only move within the viewing zone (as opposed to around the whole scene) the amount of data needed to depict the scene is greatly reduced, while still enabling the user to move to some extent and thereby to avoid the motion sickness that can be induced by augmented reality scenes with only three degrees of freedom.

[0158] While Figure 5 shows a two-dimensional viewing zone, it will be appreciated that in practice the viewing zone 1 is typically a three-dimensional zone or volume.

[0159] The viewing zone 1 may, for example, comprise a rectangular volume, or a rectangular parallelepiped, and the viewing zone may have a height of at least 30 cm, a depth of at least 30 cm, and / or a width of at least 30 cm, where these dimensions enable a user to move their head while remaining in the viewing zone. This is merely an exemplary arrangement ofthe viewing zone; it will be appreciated that viewing zones of various shapes and sizes may be used (e.g. spherical viewing zones). That being said, it is preferable that the viewing zone is limited so as to cover only a part of the volume of the scene, e.g. no more than 50% of the scene no more than 25% of the scene, and / or no more than 10% of the scene. In this regard, if the viewing zone is the same size as the scene, then the three-dimensional representation will simply be a standard representation for virtual reality (that enables a user to move freely about the scene) - and so the use of the viewing zone will not provide any reduction in file size.

[0160] The viewing zone 1 enables movement of a viewer around (a portion of) the scene. For example, where the scene is a room, the base representation may enable a user to walk around the room so as to view the room from different angles. In particular, the viewing zone enables a user to move through the scene with six degrees-of-freedom (6D0F) movement through the scene, where this aids in the provision of an immersive experience.

[0161] In some embodiments, the viewing zone 1 may be four-dimensional, where a three-dimensional location of the viewing zone changes over time - and in such embodiments the size and location of the occluded surface 2 may also change over time. More generally, it will be appreciated that viewing zones may be formed in any size or shape, with different sizes and shapes being suitable for different scenes.

[0162] The volume of the viewing zone 1 is typically selected so that a user is able to move to a degree sufficient to avoid motion sickness and to provide an immersive sensation, while still only enabling a limited amount of movement (where this leads to a smaller file size as compared to an implementation where a user is able to fully move about the scene). Typically, the viewing zone is arranged to enable a user to move their head while they are sitting or standing, but not to freely roam around a room.

[0163] The viewing zone 1 may have a (e.g. real-world) volume of less than five cubic metres (5m3), less than one cubic metre (1 m3), less than one-tenth of a cubic metre (0.1 m3) and / or less than one-hundredth of a cubic metre (0.01 m3).

[0164] The viewing zone 1 may also have a minimum size, e.g. the viewing zone may have a volume of at least 1 % of the volume of the scene, at least 5% of the volume of the scene, and / or at least than 10% of the volume of the scene. Similarly, the viewing zone may have a volume of at least one-thousandth of a cubic metre (0.01 m3); at least one-hundredth of a cubic metre (0.01 m3); and / or at least one cubic metre (1 m3).

[0165] The ‘size’ of the viewing zone 1 typically relates to a size in the real world, where if the viewing zone has a length of one metre this means that a user is able to move one metre in the real world while staying within the viewing zone. The size of the viewing zone in the scene may be greater than, equal to, or less than the size of the viewing zone in the real world. For example, the viewing zone may scale a real-world distance so that moving one metre in the real world moves the user less than (or more than) one metre in the scene. This enables the scene to provide different perceptions to the user (e.g. to make the user feel larger or smaller than they are in real life). Similarly, the viewing zone may scale a real-world angle so that rotating one degree in the real world rotates the user less than (or more than) one degree in the scene.

[0166] Therefore, a viewing zone with a volume of one cubic metre typically connotes a viewing zone in which the user is able to move about a one cubic metre volume in the real world while remaining in the viewing zone. And this may cause the user to move about a volume that is more than, or less than, one metre in the scene.

[0167] Referring to Figure 6a, in order to capture points for each surface and location that is visible from the viewing zone 1 , a plurality of capture devices C1 , C2 C9 may be used (e.g. a plurality of virtual scanners and / or a plurality of cameras). Each capture device is typically arranged to perform a capture process, e.g. as described with reference to Figure 3, in which the capture device captures points at a plurality of azimuth angles and elevation angles. By locating the capture devices appropriately, e.g. by locating a capture device at each corner of the viewing zone, it can be ensured that most (or all) points of a scene are captured.

[0168] Typically, a first capture device C1 is located at a centrepoint of the viewing zone 1. In various embodiments, one or more capture devices C2, C3, C4, C5 may be located at the centre ef faces of the viewing zone; and / or one or more capture devices C6, C7, C8, C9 may be located at edges of and / or corners of the viewing zone. Figure 6a shows a two-dimensional view (e.g. a plan view) of a rectangular viewing zone. It will be appreciated that within this viewing zone each capture device may be located on a shared plane. Equally, the various capture devices may be located on different planes. Referring, for example, to Figure 6b, there is shown a three-dimensional view of a cuboid viewing zone, where there is a capture device located: at the centre of the viewing zone; at the centre of each face of the viewing zone; and at each corner of the viewing zone.

[0169] With this arrangement, many locations in the scene (e.g. specific surfaces) will be captured by a plurality of capture devices so that there will be overlapping points relating to different capture devices. This is shown in Figure 7, which shows a first point P1 being captured by each of a first capture device C1 , a sixth capture device C6, and a seventh capture device C7. Each capture device captures this point at a different angle and distance and may be considered to capture a different ‘version’ of the point.

[0170] Typically, only a single version of the point is stored, where this version may be the highest quality version of the point and / or may be the version of the point associated with the nearest and / or least angled capture device.

[0171] In this regard, the highest ‘quality’ version of the point is captured by the capture device with the smallest distance and smallest angle to the point (e.g. the smallest solid angle). In this regard, as described with reference to Figures 4a and 4b, capturing a point for a given azimuth angle and elevation angle typically comprises capturing a plurality of sub-points at varying sub-point azimuth and elevation angles spread around the point azimuth and elevation angles. Due to the different spreads of sub-points, each capture device will capture a different version of the point (that has a different attribute) even when the points are at the same location. Capture devices that are close to the point and less angled with respect to the point typically have a smaller spread of sub-points and so typically obtain a version of a point that is sharper than a version of that point captured by more distant capture devices.

[0172] In some embodiments, a quality value of a version of the point is determined based on the spread of subpoints associated with this version (e.g. based on the perimeter formed by these sub-points and / or based on a surface area or volume bounded by these sub-points). The version of the point that is stored may depend on the respective quality values of possible versions of the points.

[0173] Regarding the ‘versions’ of the points, it will be appreciated that two ‘points’ in approximately the same location captured by each capture device may not have exactly the same location in the three-dimensional representation. More specifically, since each capture device typically projects a ‘ray’ at a given angle, the rays of differing capture devices may contact the surface at different locations for each capture device. Two points may be considered to be two ‘versions’ of a single point when they are within a certain proximity, e.g. a threshold proximity. For example, where the first capture device C1 captures a first point and a second point at subsequent azimuth angles, and the sixth capture device C6 captures a further point that is in between the locations of the first point and the second point, this further point may be considered to be a ‘version’ of one of the first point and the second point.

[0174] This difference in the points captured by different capture devices is illustrated by Figures 8a and 8b, which show the separate captured grids that are formed by two different capture devices. As shown by these figures, each capture device will capture a slightly different ‘version’ of a point at a given location and these captured points will have different sizes. Each capture step is associated with a particular range of angles (e.g. a nominal capture angle of 1 ° might encompass angles from 0.9° to 1.1 °), and therefore capture devices that are far from a point to be captured represent a wider region at the capture distance than capture devices closer to that point to be captured. As shown in Figure 8a, the capture device C1 would capture the points P1 and P2 in separate brackets, whereas for the capture device C2 these points are in the same bracket. Therefore, the capture device C2 might determine a single point that encompasses both points P1 and P2, whereas the capture device C1 would determine separate points for these two points. Considering then a situation in which points P1 and P2 are captured separately, and capture device C1 is used to capture point P1 while capture device C2 being used to capture point P2, it should be apparent that the ‘sizes’ of these captured points, and the locations in space that are encompassed by the captured points will be based on different grids. For example, the width of the captured point P2 captured by the capture device C2 will be larger than the width of the captured point P1 captured by the capture device C1. The capture process may be determined based on the existence of these different grids, and on the different bracket widths that occur at different distances from a capture device.

[0175] Figure 8a shows an exaggerated difference between grids for the sake of illustration. Figure 8b shows a more realistic embodiment in which the three-dimensional representation comprises a plurality of points associated with different capture devices, where these points lie on different grids associated with these different capture devices.

[0176] Distance-dependency

[0177] As has been described, e.g., with reference to Figure 4a, the capturing of a point by a capture device typically involves capturing a plurality of sub-points associated with this point and then combining these sub-points. And as has been described, e.g. with reference to Figure 4b, in some embodiments the subpoints may be associated with different distances from the capture device.

[0178] Typically, at short distances, viewers are able to identify details of small surfaces and small separations between surfaces. For example, if the two surfaces shown in Figure 4b were located 5 centimetres apart with the front surface being 50 centimetres from a viewer, then that viewer would be able to identify that the rear surface is slightly angled and the viewer would be able to roughly estimate the separation between the front surface and the rear surface. In contrast, at large distances, viewers are typically unable to identify small details of surface or separations between surfaces. For example, if the two surfaces of Figure 4b were located 5 centimetres apart with the front surface being 1 kilometre from the viewer, the viewer might only be able to identify that the rear surface is behind the front surface (without being able to provide any reasonable estimate of the separation between the front surface and the rear surface) or the two surfaces might even appear to the viewer to be as a single surface.

[0179] In relation to that Figure 4b, it has been described that combining the sub-points in the situation shown by Figure 4b can lead to an inaccurate combined point to be determined that seems to float away from a surface with an inaccurate colour.

[0180] While this float is problematic for nearby surfaces, this inaccuracy is not necessarily problematic for further surfaces. In practice, where a surface is far away, a user may not be able to identify a slight separation of a point from this surface, and so it may be acceptable to store points even where they do not correspond to a single surface. Therefore, the present disclosure envisages a method in which the determination of a point, and in particular the combining of sub-points to form a point, is dependent on a distance of that point (and / or the component sub-points) from one or more of: a capture device that is capturing the point; and a viewing zone. Being dependent on a distance from a viewing zone typically connotes being dependent on a minimum distance from a viewing zone (e.g. dependent on a distance between the point of the three- dimensional representation and a proximate point on the viewing zone).

[0181] Such a method of determining a point in dependence on a distance of that point from a viewing zone is described in Figure 9.

[0182] In a first step 41 , a computer device determines a plurality of sub-points that are associated with a point. Each of these sub-points is associated with a capture device that has used a different angle (e.g. a different azimuth angle and / or a different elevation angle) to capture the sub-points.

[0183] In a second step 42, a computer device determines a separation between the sub-points. The separation may be an absolute separation, may be a radial separation, and / or may be an axial separation (e.g. a difference in a depth of the points). Equally, the separation may comprise a difference between distance values of a sub-point to the capture device. For example, the separation may be the maximum difference in distance values between the sub-points.

[0184] In a third step 43, a computer device determines a distance of the sub-points from the viewing zone (and / or the capture device). The distance may, for example, be a minimum distance of a sub-point, a maximum distance of a sub-point, or an average distance of a sub-point. Typically, the distance is the distance of the closest sub-point.

[0185] In a fourth step 44, the computer device determines whether to combine the sub-points (in order to form a point) based on the separation and the distance. Typically, this comprises comparing the separation to a threshold separation and combining the sub-points in dependence on the separation being beneath the separation threshold.

[0186] Typically, the separation threshold is a function ofthe distance of the points from the capture device and / or the viewing zone. The separation threshold may increase linearly with the distance, but typically the threshold increases at an increasing rate as the distance increases (e.g. increases exponentially as the distance increases), based on a function, and / or based on an arctan function. The separation threshold may change (e.g. in a discrete step) from a first threshold for a first range of distances to a second threshold for a second range of distances.

[0187] The general principle being applied here is that a separation becomes less noticeable as distance from the viewing zone increases; for example, a separation of 5 cm in depth at a distance of 1 km will not be distinguished from any viewpoint inside the zone of viewpoints, on the contrary a separation of 5 cm at a distance of 50 cm will be highly perceived.

[0188] If, in the fourth step 44, the separation of the points is deemed too great for the points to be combined (e.g. because the separation exceeds a separation threshold), then the computer device may determine that it is appropriate to divide the point into a plurality of points. In this regard, points are typically formed from a combination of sub-points. While each point may, by default, be formed from the same number of subpoints, it may be possible to form points from different number of sub-points. In practice, this may lead to most points in the three-dimensional representation being formed of, e.g. four sub-points or sixteen subpoints, with certain points near the edges of surfaces being formed of fewer, e.g. 1 , or 4, sub-points.

[0189] While the above description has considered the combination of sub-points based on a separation of these sub-points and a distance of these sub-points from the viewing zone. The combination of the sub-points may alternatively, or additionally, depend on a difference between the attribute values of the points where this attribute difference may be compared to an attribute threshold, and this attribute threshold may also depend on the distance of the sub-points from the viewing zone.

[0190] For example, for a given distance, points that are highly separated, but have similar attribute values may be combined to form a point. Similarly, points that are close together, but have very different attribute values may not be combined. Each of the threshold separation value and the threshold attribute difference may depend on the distance of the sub-points from the capture device.

[0191] Selection of capture device

[0192] Where points are determined using capture devices placed at a plurality of locations, the method may comprise placing a first set of n capture devices in different locations (or, equally, placing a single capture device sequentially in n locations - it will be appreciated throughout this disclosure that these operations are alternatives that are functionally equivalent), with each capture device being arranged to capture j scanning points by iterating through a plurality of azimuth angles and elevation angles. The minimum number of capture devices and the minimum number of scanning angles is determined by the resolution of the display device intended for viewing the scene as well as the size of the viewing zone. Preferably the scanning pitch, that is the angle between two subsequent capture rays, is less than or equal to the angular resolution of the display screen.

[0193] Exemplary arrangements of capture devices are shown in Figures 6a and 6b, which show how a plurality of capture devices may be located about the viewing zone. Referring, for example, to Figure 6b, there is shown a three-dimensional view of a cuboid viewing zone, where there is a capture device located: at the centre of the viewing zone (C1); at the centre of each face of the viewing zone (C2_A, C2_B, C2_C, C2_D, C2_E, C2_F); and at each corner of the viewing zone (C3_A, C3_B, C3_C, C3_D, C3_E, C3_F, C3_G, C3_H.

[0194] With such arrangements, many points of the scene will be captured by a plurality of capture devices. This leads to certain points of the scene having multiple versions, where a first version of a point is captured by a first capture device and a second version of this point is captured by a second capture device. This has been described above with reference to Figure 7, which shows how a point P1 may be captured by each of a first capture device C1 , a sixth capture device C6, and a seventh capture device C7.

[0195] This capturing of multiple versions of points has a number of possible drawbacks. In particular, if each of these versions is stored, then there is an increase in the file size (as compared to an arrangement where only one version of each point is stored).

[0196] Therefore, it is generally desirable to store only a single version of each point (e.g. to store, for each location in the scene, a single point captured by a single capture device),

[0197] In some embodiments, for each location in the scene forwhich multiple versions of a point may be captured, a point is captured in dependence on a ‘quality’ of the points (e.g. solely in dependence on this quality). This quality may, for example, relate to the distance of each potential capture device from this location / their point.

[0198] However, there may be some differences in the versions of the points captured by the different capture devices (e.g. because these capture devices are capturing the point from a different location / angle). While these differences are typically small, they might nevertheless lead to perceptible boundaries between adjacent points captured by different capture devices. That is, if a first point is captured by a first capture device and a second point adjacent the first point is captured by a second capture device, there may be a noticeable boundary between these points that might not exist if both points had been captured by the same capture device. Equally, the use of the two different capture devices may necessitate further processing to ensure that such a boundary does not appear.

[0199] Therefore, as is described below, the version of each point that is recorded or stored may (e.g. also) depend on another feature of the capture devices and / or based on the capture process.

[0200] Referring to Figure 10, there is a described a method of determining a point of the three-dimensional representation in dependence on a quality factor associated with a capture device.

[0201] In a first step 51 , a computer device determines a first quality factor associated with a first capture device and a first version of point (a point to be captured).

[0202] The quality factor is associated with a capture quality of the second point. Typically, the quality factor is determined based on one or more of: a distance between the first capture device and the point and / or an angle between the first capture device and the second point (e.g. an angle between the first capture device and a normal associated with the second point); and a surface area associated with the point.

[0203] In some embodiments, the quality factor for a capture device is associated with (e.g. inversely proportional to) a size of a point that could be captured by that capture device. Therefore, points close to the camera device will, for a given angle, have a higher quality factor than points farther away from the capture device. Similarly, points with a small angle to the capture device will, for a given distance, have a higher quality factor than points with a higher angle to the capture device. The quality factor may then be determined as a function of one or more of (or all of): the distance of a point from a capture device; the angle between the point and the capture device; and the angle between the normal of the point and the capture device (e.g. between the normal of the point and a line passing from the capture device to the point).

[0204] Referring to Figures 11 a, 11 b, and 11 c, there are shown various situations in which (a version of) a first point 201 is captured by each of a first capture device C1 and a second capture device C2. As has been described above with reference to Figures 4a and 4b, the capture process typically comprises capturing a plurality of sub-points associated with the point to be captured and then combining these sub-points to determine an attribute value for the point.

[0205] Where each capture device is arranged to capture sub-points with the same angular spread a:

[0206] Figure 11a shows how a first capture device C1 that is closer to a captured point 201 than a second capture device C2 typically captures the point with higher ‘quality’ (e.g. with a lower spread between the sub-points so that the point formed from the sub-points more accurately reflects the actual value of the point at location 202). Essentially, by capturing a plurality of smaller points to cover an area instead of a single large point, this area can be represented more accurately.

[0207] Figure 11 b shows how a first capture device C1 that is at a smaller angle to a surface comprising the point 201 typically captures that point with higher quality.

[0208] Figure 11 c shows how a more distant capture device C2 may capture the point 201 with higher quality if the point lies on a surface that is angled to the capture devices.

[0209] Referring to Figure 11d, there is considered a generalised scenario where a capture device is capturing a point 201 , which point is a distance D from the capture device and has a normal that forms an angle k between the normal of the point and the capture device. The capture device is arranged to capture subpoints with an angular spread of a (so the sub-points cover a boundary from an angle of to an angle of

[0210] With this scenario, the spread of the sub-points covers a distance (from the ‘leftmost’ point to the ‘rightmost’ point) that is proportional to As k increases, cos(fc) decreases and so the covered distance increases. Similarly, as D increases, the covered distance increases.

[0211] A quality factor for each capture device may be determined based on this distance (e.g. based on an inverse of the spread).

[0212] It will be appreciated that the points are typically three-dimensional points, so that the sub-points will cover a surface area that depends on each of: a distance of the point to the capture device; an azimuthal spread of sub-points; and an elevational spread of sub-points. The quality factor is typically determined based on this distance, azimuthal spread of sub-points; and elevational spread of sub-points. This may comprise the quality factor being determined based on a size (e.g. an area, a surface area, a solid angle, and / or a height and / or width) of a point captured by a capture device.

[0213] The quality factor may be determined prior to the capturing of the second point using the first capture device, or the quality factor may be determined following the capturing of the second point using the first capture device. For example, the quality factor may depend on a difference between sub-points associated with the capture of the second point with the first capture device (e.g. based on a difference in the attribute values of these sub-points and / or based on a difference in the distances between the first capture device and these sub-points).

[0214] Returning to the method of Figure 10, in a second step 52, the computer device determines a second quality factor associated with a second capture device and the point. In a third step 53, the computer device determines a capture device to be used for capturing the second point based on the first quality factor and the second quality factor.

[0215] While this method refers to a first ‘version’ and a second ‘version’ of a single point, it will be appreciated that in practice each capture device typically captures separate points at a location. A first point captured by the first capture device and a second point captured by the second capture device may be considered to be two versions of the same ‘point’ if these points overlap (e.g. if these points both encompass a location, either a specific location or any location, in the three-dimensional space and / or if both of these points include, when the size of the points is considered, a coordinate in the three-dimensional space - e.g. a specific coordinate or any coordinate).

[0216] The method of Figure 10 can be used to ensure that each point is captured using a capture device that provides the highest quality point. This can provide a high-quality representation, but, as mentioned above, this may lead to discrepancies at the boundaries of points captured by different capture devices. Furthermore, this can increase the file size needed to store the representation as compared to an implementation in which a row of points is captured by a single capture device.

[0217] In a practical example, and referring to Figures 12a and 12b, where a row of capture devices C1 , C2, C3, C4 captures a row of points 201 , 202, 203, 204 on a surface that runs parallel to the capture devices this may result in each point on this surface being captured by a different capture devices. For a nearby surface, as in Figure 15a, this close capture of points may be beneficial. The increase in quality that is obtained from using a nearby capture device might be noticed by a user where the surface is close to the viewing zone and so the user can inspect the surface from close-up. However, for a far surface, as shown in Figure 15b, this increase in quality may be marginal at best and may not be noticed by a user, both because the differences in the quality of each version of each point are low and also because the surface is far away. Regarding this first consideration, where the surface is nearby, then there is a substantial difference in the distance between the first point 201 and the first capture device C1 and similarly there is a substantial difference in the angle between the first point 201 and the first capture device and the angle between the first point 201 and the second capture device C1 . But as the distance increases, these differences in distance and angle decrease (the distance decreases as a percentage of the overall distance of the point and the angle decreases absolutely).

[0218] Another disadvantage of frequently switching the capture device used to capture points is that this frequent switching can increase the number of points needed to form the three-dimensional representation.

[0219] This can be explained with reference to Figures 8a and 8b, which figure shows a first grid of points captured by a first capture device and a second grid of points captured by a second capture device. As can be seen, the different locations of the two capture devices, and the capturing process that is based on azimuth / elevation angles, causes each capture device to form a separate grid of points. One consequence of these separate grids is that the second point captured by a first capture device will cover a slightly different area than the second point captured by the first capture device. Where this disclosure considers two versions of the same ‘point’ that may be captured by different capture devices, it will be appreciated that these two (versions of) points overly a point in space in the scene. But since each point has a height and a width, these two (versions of) points may overlay different areas in the scene. A second consequence of these separate grids is that where there is a change in capture device, there will be a change in grid. This can cause holes in the three-dimensional representation that may eventually be noticed by a viewer of the scene. Or, to prevent holes forming, the computer device may need to capture additional points for the first capture device or the second capture device so that the two grids are superimposed. While this prevents the formation of holes between the grids, this leads to overlapping points that increase the file size of the representation beyond an arrangement without overlapping points. Due to each of these considerations, it is often desirable to reduce the amount of switching between capture devices. But it is desirable to do this without entirely discounting the quality factor (since this could cause a substantial reduction in the quality of the representation).

[0220] Due to these considerations, it may be desirable to use different capture devices for each point when the surface is near the viewing zone (as in Figure 12a), but to use a single capture device to capture each points when the surface is far from the viewing zone (as in Figure 12b).

[0221] Referring then to Figure 13, there is a described a method of determining a point of the three-dimensional representation in dependence on a quality factor associated with a capture device and also in dependence on a capture device that has been used to capture a further point of the representation.

[0222] In a first step 61 , a computer device causes a first capture device to capture a first point of the scene (e.g. at a specific elevation and azimuth).

[0223] In a second step 62, the computer device determines a quality factor associated with this first capture device and a second point of the scene. For example, the second point may be a point adjacent the first point, where the second point is associated with an increment in the azimuth angle or the elevation angle as compared to the first point.

[0224] More generally the first point and the second point may both be in a first region of the three-dimensional representation. For example, the separation between the first point and the second point may be less than 10%, less than 5%, less than 2%, and / or less than 1 % of a maximum dimension of the representation or the scene (e.g. a width, a height, a depth, or a maximum diagonal of the representation or the scene).

[0225] In a third step 63, the computer device determines a second quality factor associated with a second capture device and the second point.

[0226] In a fourth step 64, the computer device determines a capture device to be used for capturing the second point based on the first quality factor and the second quality factor.

[0227] In summary, as described with reference to Figure 10, determining the capture device to be used for capturing the second point could comprise determining the capture device with the highest quality factor. This typically is the capture device that is located closest to the second point and / or the capture device that is best aligned with a normal of the second point. This can cause frequent and / or unnecessary switching of capture devices. Therefore, the capture device used for a second point may also (or alternatively) be dependent on the capture device used for a first point.

[0228] In some embodiments, in the fourth step 64, the computer device may determine a capture device to be used for capturing the second point based on whether the first quality factor associated with the first device (used to capture the first point) is within a threshold of the second quality factor (e.g. within a threshold of a maximum quality factor). The threshold may, for example, be at least 10%, at least 20%, and / or 25%. This may equally be considered as the threshold being at least 50% of a maximum quality factor, at least 75% of a maximum quality factor, at least 80% of a maximum quality factor, and / or at least 90% of a maximum quality factor.

[0229] The method may comprise determining a quality factor for a plurality of capture devices for the second point, determining one or more of these capture devices that exceed a threshold quality factor (e.g. that are within a threshold of a maximum quality factor), and determining a capture point to be used for capturing the point from the set of capture devices that exceed a threshold quality factor.

[0230] In some embodiments, the capture devices are associated with a priority list, where the selection of the capture device to be used for capturing the second point is dependent on this priority list. For example, the capture devices may each be assigned a priority value, with the capture device being selected as the highest priority capture device that exceeds the threshold quality factor. Referring to Figure 14, there is shown a further method for determining a capture device to be used to capture the second point.

[0231] Similarly to the method of Figure 13, in a first step 71 , the computer device causes the first capture device to capture a first point and in a second step 72, the computer device determines a first quality factor associated with the first capture device and a second point.

[0232] In a third step 73, the computer device determines a maximum quality factor associated with the second point. For example, the computer device may determine a quality factor for each possible capture device for the second point (or for a subset of possible capture devices based on proximity to the second point).

[0233] In a fourth step 74, the computer device determines whether the first quality factor is within a threshold value of the maximum quality factor (e.g. within 75% of the maximum quality factor).

[0234] In a fifth step 75, if the first quality factor is within the threshold value, the first capture device is used to capture the second point.

[0235] In a sixth step 76, if the first quality factor is not within the threshold value, a second capture device is used to capture the second point.

[0236] The selection of the second capture device may be dependent on various factors. In some embodiments, the second capture device is selected as the capture device associated with the maximum quality factor. In some embodiments, the second capture device is selected in dependence on a priority value of the second capture device, for example the second capture device may be selected to be the highest priority capture device that has a quality factor above the threshold factor. In some embodiments, the second capture device is determined based on the relative numbers of points captured by the capture devices that exceed the quality factor threshold (e.g. so that the second capture device is determined as the capture device above the threshold that has captured the greatest number of other points).

[0237] Typically, the second point is proximal to, and / or adjacent, the first point and / or the second point is within a threshold distance of the first point. The second point may, for example, be within an azimuth increment and / or an elevation increment of the first point.

[0238] The priority list may comprise a continuous plurality list, where each capture device has a different priority. Equally, the priority list may comprise a plurality of discrete sections where, for example, one capture device may have a highest priority, a first plurality of capture devices may have a second highest priority, a second plurality of capture devices may have a third highest priority (e.g. where the second plurality is larger than the first plurality.

[0239] Various methods of determining the priority list are envisaged by the present disclosure. In some embodiments, the priority list may be predetermined prior to the determining of the three-dimensional representation, e.g. based on a user input. In some embodiments, a central capture device (e.g. the capture device C1 at the centre of the viewing zone) is assigned the highest priority.

[0240] In some embodiments, capture devices at the centres of the faces of the viewing zone (C2_A ... C2_F) may then be assigned the second highest priority. In some embodiments, the viewing zone may be separated into a plurality of (e.g. equally sized) quadrants, in which each quadrant may have a single capture device at the second highest priority and then each quadrant may be separated into a plurality of (e.g. equally sized) sub quadrants, in which each sub quadrant may have a single capture device at the third highest priority, etc.

[0241] Typically, the priority list is determined based on an expected number of points that each capture device will be able to capture (e.g. a capture device at the centre of the viewing zone may be expected to be able to capture the highest number of points, and so this central capture device may be assigned a highest priority). In some embodiments, the priority of a capture device may be dependent on a side of the viewing zone at which that capture device is placed. Typically, a scene is created with a ‘forwards’ direction that is intended to be focused on by a user and a ‘backwards’ direction (e.g. there may be an object of focus that is located in the forwards direction). The forwards direction may be associated with a first face of the viewing zone, with the backwards direction being associated with a second face of the viewing zone, the second face being opposite the first face. The priority of a capture device may be dependent on that capture device being located on, or near to, the first, forwards, face of the viewing zone (e.g. so that the capture device C2_A may be assigned a higher priority than the capture device C2_F).

[0242] In some embodiments, the priority list is determined based on a number of points for which each capture device is able to exceed a threshold quality value. For example, the process of determining the three- dimensional representation may comprise a multi-step process in which, in a first pass of the scene, the computer device computes a quality value for each capture device for a plurality of points of the scene. The computer device may then determine the priority list based on these quality values, where this enables the computer device to give the highest priorities to capture devices that have consistently high-quality values (e.g. the highest priority capture device may be the capture device with the highest average quality value and / or the device with the highest number of quality values that exceed a quality value threshold). Then, in a second pass of the scene, the computer device may capture a plurality of points based on the determined priority list.

[0243] In some embodiments, the priority list comprises a plurality of sub-lists, with the sub-lists each containing one or more of the capture devices. In some embodiments, the capture device used to capture the second point is dependent on one or more of these sub-lists. For example, if the first device is on a first sub-list, then the second capture device may be selected based on being on this same sub-list (such implementations may be used to reduce an amount of switching between sub-lists, where this may have a benefit when encoding the three-dimensional representation).

[0244] In various embodiments, the priority list may be determined (wholly, or in part) based on one or more of: a user input; a capability of the capture devices; a location of the capture devices (e.g. a location within the viewing zone); a number of points that are within sight of the capture devices; and a quality value distribution (e.g. an average and / or a median quality value) associated with the capture devices.

[0245] In some embodiments, the priority list is determined based on a capability of the capture devices. In some situations, the capture devices may comprise different types of capture devices (e.g. some capture devices may be able to capture higher resolutions). Therefore, devices with more capabilities may be given a higher priority.

[0246] In some embodiments, a plurality of priority lists are provided. For example, different regions of the scene may be associated with different priority lists.

[0247] Regarding the threshold that is used to determine a switching of a capture device, this threshold value may depend on a feature of the point. For example, the threshold value (e.g. the threshold quality factor) may depend on a distance of the point from the viewing zone. Viewers are typically less able to identify differences in the quality of points at longer distances and therefore, for points far from the viewing zone, the threshold quality may be lower than points closer to the viewing zone (though this is already accounted for within the determination of the quality factor, which typically depends on a distance of the point from the capture device).

[0248] As an example, a separation of 5 cm in depth between two objects at a distance of 1 km from the viewing zone will typically be indistinguishable to a user inside the viewing zone, on the contrary a separation of 5 cm at a distance of 50 cm will be highly perceived. Due to this, the threshold quality factor may be dependent on the distance of the user from the viewing zone where, for example, a first quality factor threshold is used for distances beneath 10 m, a second quality factor threshold is used for distances above 10 m, and / or a third quality factor threshold is used for distances above 1 km,

[0249] While the above methods primarily describe the use of a quality factor that depends on a distance and angle of a point from a capture device, it will be appreciated that other parameters may be considered. For example, the switching of a capture device may be dependent on a difference in an attribute of a first point and a second point, where a second capture device may be used to capture the second point in dependence on a difference between an attribute value of the second point exceeding an attribute value of the first point (where this attribute value may be determined using the first capture device or the second capture device). Such embodiments ensure that interfaces between surfaces are captured using a capture device with a high quality factor. Similarly, the second capture device may be used in dependence on a distance between the first point and the second point exceeding a threshold distance. Furthermore, a plurality of these conditions may be considered (where e.g. the meeting of any of the conditions may result in a second capture device being used for the second point).

[0250] In some embodiments, the capture device for a point may be determined in dependence on an obstructions between the potential capture devices and the point being captured. For example, where an object and / or a transparent object is located between a first capture device and the point and there is no obstruction between a second capture device and the point, then the second capture device may be chosen to determine the point. In practice, this may comprise the quality factor being determined in dependence on an obstruction (e.g. where the presence of an obstruction acts as a scaling factor to reduce the quality factor for a capture device).

[0251] In some embodiments, the method comprises determining a capture device to be used for capturing a point of the three-dimensional representations and then capturing the point using this capture device (e.g. capturing only a single version of the point). In some embodiments, the method comprises determining a plurality of versions of a point using different capture devices and then selecting one of these versions to be retained as the point (where the other versions may be deleted). For example, the retained version of the point may be determined based on the priority list described above. In other words, the selection of the capture device that is used to capture a point of the final three-dimensional representation may occur either before the point has been captured or after the point has been captured (e.g. as part of an initial three- dimensional representation).

[0252] Alternatives and modifications

[0253] It will be understood that the present invention has been described above purely by way of example, and modifications of detail can be made within the scope of the invention.

[0254] The representation is typically arranged to provide an extended reality (XR) experience (e.g. a representation that is useable to render a XR video). The term extended reality (XR) covers each of virtual reality (VR), augmented reality (AR), and mixed reality (MR) and it will be appreciated that the disclosures herein are applicable to any of these technologies.

[0255] Reference numerals appearing in the claims are by way of illustration only and shall have no limiting effect on the scope of the claims.

Claims

Claims1 . A method of determining a point of a three-dimensional representation, the method comprising: determining a first point of the representation with a first capture device; identifying a second point of the representation; determining a capture device to be used for capturing the second point in dependence on the first capture device; and capturing the second point using the determined capture device.

2. The method of claim 1 , comprising determining the capture device so as to minimise a number of capture devices required to capture the points of the representation.

3. The method of claim 1 or 2, wherein determining a capture device to be used for capturing the second point in dependence on the first capture device comprises deleting one or more points captured by one or more further capture devices.

4. The method of any preceding claim, comprising determining the capture device based on a number of points already captured by one or more available capture devices and / or based on a number of points already captured by the first capture device.

5. The method of any preceding claim, comprising: determining whether the first capture device is able to capture the second point; capturing the second point with the first capture device if the first device is able to capture the second point; and capturing the second point with a second capture device if the first device is not able to capture the second point.

6. The method of any preceding claim, comprising: determining a first quality value associated with the first capture device capturing the second point; capturing the second point with the first capture device if the first quality value exceeds a threshold value; and capturing the second point with a second capture device if the first quality value does not exceed the threshold value.

7. A method of determining a point of a three-dimensional representation, the method comprising: determining a first quality factor associated with a first capture device that is able to capture the point; determining a second quality factor associated with a second capture device that is able to capture the point; and determining whether to capture the point with the first capture device or the second capture device based on the first quality factor and the second quality factor.

8. The method of claim 6 or 7, wherein the point is a second point, and wherein the method comprises: determining a first point with the first capture device; and determining whether to capture the second point with the first capture device or the second capture device based on whether the first quality factor exceeds a threshold value.

9. The method of any of claims 6 to 8, wherein the threshold quality factor is associated with a maximum possible quality factor, the maximum possible quality factor being the quality factor associated with a maximum quality capture device.

10. The method of claim 9, wherein the threshold quality factor is at least 50%, at least 80%, and / or at least 90% of the maximum possible quality factor.11 . The method of any of claims 6 to 10, comprising: in response to determining that the first quality factor is below the threshold quality factor, capturing the point using the second capture device.

12. The method of any of claims 6 to 11 , comprising determining the second capture device.

13. The method of any of claims 6 to 12, wherein the threshold quality factor depends on the first capture device, preferably wherein one or more capture devices are associated with different threshold quality factors.

14. The method of any of claims 6 to 13, wherein the threshold quality factor depends on a distance of the point from a viewing zone and / or from a capture device.

15. The method of any of claims 6 to 14, wherein the quality value for each capture device is dependent on a distance of that capture device from the point.

16. The method of any of claims 6 to 15, wherein the quality value for each capture device is dependent on an angle of the capture device to a normal value of the point and / or an angle of the capture device to an angle of a surface that includes the point.

17. The method of any of claims 6 to 16, wherein determining the second capture device comprises: determining a set of capture devices that have a quality factor higher than the threshold quality factor; and determining the capture device in this set that has the highest quality factor.

18. The method of any preceding claim, comprising determining the (e.g. second) capture device as the capture device that provides the maximum quality factor.

19. The method of any preceding claim, comprising determining the (e.g. second) capture device based on a priority list associated with the capture devices.

20. A method of determining a point of a three-dimensional representation, the method comprising: identifying a point of the representation; identifying a plurality of capture devices capable of capturing points of the representation; determining a capture device to be used for capturing the point in dependence on a priority list associated with the capture devices; and capturing the point using the determined capture device.

21. The method of claim 19 or 20, wherein the priority list is associated with a viewing zone, preferably wherein a highest priority capture device is located at a centre of the viewing zone, more preferably wherein a plurality of second highest priority capture devices are located at the centres ef faces of the viewing zone.

22. The method of any of claims 19 to 21 , comprising identifying a plurality of priority lists, preferably wherein each of the priority lists is associated with a different region of the scene.

23. The method of any of claims 19 to 22, wherein the priority list is determined based on a number of points that can be captured by each of the priority devices.

24. The method of any of claims 19 to 23, comprising: in a first step, determining a number of points for which each capture device exceeds a / the threshold quality factor; and in a second step, determining the priority list based on said number of points.

25. The method of any of claims 19 to 24, wherein the priority list is based on one or more capabilities of the capture devices.

26. The method of any of claims 19 to 25, wherein the priority list is dependent on a user input.

27. The method of any preceding claim, wherein the first point and the second point are adjacent points of the three-dimensional representation.

28. The method of any preceding claim, wherein the first point and the second point are both in a first region of the three-dimensional representation and / or where the separation between the first point and the second point is less than 10% of a maximum dimension of the representation, preferably less than 5%, less than 2%, and / or less than 1%.

29. The method of any preceding claim, wherein each capture device is arranged to capture a plurality of sub-points associated with the point, the sub-points being associated with an angular spread, wherein the quality value for each capture device is dependent on an area bounded by the sub-points captured by that capture device.

30. The method of any preceding claim, wherein the three-dimensional representation is associated with a viewing zone, the viewing zone comprising a subset of the scene and / or the viewing zone enabling a user to move through a subset of the scene, preferably wherein the user is able to move within the viewing zone with six degrees of freedom (6DoF).31 . The method of claim 30, wherein: the viewing zone has a volume of less than 50% of the volume of the scene, less than 20% of the volume of the scene, and / or less than 10% of the volume of the scene; and / or the viewing zone has, or is associated with, a volume, preferably a real-world volume, of less than five cubic metres (5m3), less than one cubic metre (1 m3), less than one-tenth of a cubic metre (0.1 m3) and / or less than one-hundredth of a cubic metre (0.01 m3).

32. The method of any preceding claim, wherein the three-dimensional representation comprises a point cloud.

33. The method of any preceding claim, comprising storing the three-dimensional representation and / or outputting the three-dimensional representation, preferably outputting the three-dimensional representation to a further computer device.

34. The method of any preceding claim, comprising generating an image and / or a video based on the three- dimensional representation.

35. The method of any preceding claim, comprising forming one or more two-dimensional representations of the scene based on the three-dimensional representation, preferably comprising forming a two- dimensional representation for each eye of a viewer.

36. The method of any preceding claim, wherein the point is associated with one or more of: a location; an attribute; a transparency; a colour; and a size.

37. The method of any preceding claim, wherein the point is associated with an attribute for a right eye and an attribute for a left eye.

38. The method of any preceding claim, wherein the scene comprises one or more of: an extended reality (XR) scene; a virtual reality (VR) scene; an augmented reality (AR) scene; and a mixed reality (MR) scene.

39. A computer program product comprising software code that, when executed on a computer device, causes the computer device to perform the method of any preceding claim.

40. A machine-readable storage medium that includes instructions that, when executed by one or more processors of a machine, cause the machine to perform the method of any of claims 1 to 38.41 . A system for carrying out the method of any of claims 1 to 38, the system comprising one or more of: a processor; a communication interface; and a display.

42. An apparatus for processing a three-dimensional representation of a scene, the apparatus comprising: means for identifying a plurality of points of a three-dimensional representation; means for determining a coordinate system for the representation; means for determining a plurality of containers associated with the coordinate system, wherein each container covers a volume of the three-dimensional representation, and wherein a depth of each container is dependent on a distance of that container from a centre of the coordinate system; and means for allocating each of the plurality of points to one of the containers.

43. An apparatus for determining a point of a three-dimensional representation, the apparatus comprising: means for determining a first point of the representation with a first capture device; means for identifying a second point of the representation; means for determining a capture device to be used for capturing the second point in dependence on the first capture device; and means for capturing the second point using the determined capture device.

44. An apparatus for determining a point of a three-dimensional representation, the apparatus comprising: means for determining a first quality factor associated with a first capture device that is able to capture the point; means for determining a second quality factor associated with a second capture device that is able to capture the point; and means for determining whether to capture the point with the first capture device or the second capture device based on the first quality factor and the second quality factor.

45. An apparatus for determining a point of a three-dimensional representation, the apparatus comprising: means for identifying a point of the representation; means for identifying a plurality of capture devices capable of capturing points ofthe representation; means for determining a capture device to be used for capturing the point in dependence on a priority list associated with the capture devices; and means for capturing the point using the determined capture device.

Images