Augmented reality overlays

Abstract

A computer-implemented method is disclosed for facilitating the generation of augmented reality (AR) overlays on a first group of AR-enabled devices located in a first area. The first group of AR-enabled devices comprises more than one AR-enabled device. The method comprises determining (502), for the first group of AR-enabled devices, first information for generating an AR overlay. The first information is determined based on a location of the first group of AR-enabled devices. The method further comprises sending (504), to one or more first wireless devices, at least part of the first information. Each of the first wireless devices is one of: a respective AR device in the first group of AR-enabled devices; or an intermediate wireless device connected to one or more of the first group of AR-enabled devices.

Description

[0001] AUGMENTED REALITY OVERLAYS

[0002] TECHNICAL FIELD

[0003] Embodiments of the present disclosure relate to augmented reality (AR) overlays, and particularly to methods, apparatus, and computer-readable media for facilitating the generation of AR overlays on a group of AR-enabled devices.

[0004] BACKGROUND

[0005] AR can be used to improve user experience. For example, screens of AR-enabled devices (sometimes referred to as “transparent screens”) can be configured to allow users to view a scene of interest whilst receiving additional information relating to the scene (e.g., via an AR overlay on the screen). The screen may be either an optical screen or a “pass-through” screen, which can be configured to display a video of the scene of interest captured by a camera (e.g., a camera located on the AR-enabled device). An AR-enabled device may, for example, be a pair of glasses or a mobile device.

[0006] AR-enabled devices may, for example, be utilized in sports arenas; a user may look through the screen of the AR-enabled device and be provided with additional information (e.g., information about a person the user is focusing on). This additional information is displayed on the screen (e.g., via an AR overlay). For example, when utilized for a football match, the additional information could be the name of a player, whether the player has scored any goals, whether the player has received any warnings, and so on. The additional information may be displayed in proximity to the players as viewed on the screen, in a virtual display or jumbotron in the arena, or at a virtual plane in the AR- enabled device.

[0007] Where AR is provided via an AR overlay displayed on the screen, the overlay may be rendered according to one of the following methods:

[0008] 1) A (central) server may render the AR overlay and transmit the AR overlay to an AR-enabled device. In this case, the server is informed as to where the AR-enabled device is located so that it can determine a view from the AR-enabled device (and therefore create an appropriate AR overlay). This AR overlay is then displayed by the AR-enabled device, allowing the user to have a more detailed experience.

[0009] 2) The AR-enabled device may render the AR overlay. In this case, information on what to render (also referred to herein as rendering information) is received by the AR- enabled device (e.g., from a server). The rendering information may indicate where important objects should be located in the AR overlay and what they look like. The AR overlay is then rendered / generated directly at the device.

[0010] 3) An intermediate wireless device (e.g., a users’ mobile phone) connected to an AR-enabled device is used to render the AR overlay. In this case, rendering information is received by the intermediate wireless device (e.g., from a server), which then renders the AR overlay and sends the rendered AR overlay to the AR-enabled device. The AR- enabled device may then display the AR overlay.

[0011] To exemplify the above methods, a use case of AR in a sports context is discussed, where a sports game is played in a first arena, and people watch the sports game in a second arena using AR-enabled devices. The players are inserted onto the pitch / court / field of the second arena via an AR overlay. According to the first rendering method discussed above, a server: generates / renders a video-based AR overlay for an AR-enabled device based on positioning information received from the AR-enabled devices; and sends the video-based AR overlay to the AR-enabled devices. The AR- enabled devices then decode the video-based AR overlay and display it. According to the second rendering method, the server transmits rendering information (including information about the players, such as their position and what they look like) directly to the AR-enabled devices, and each AR-enabled device determines the video-based AR overlay to render based on the current viewing angle (also referred to herein as a viewing direction) of the AR-enabled device. According to the third rendering method, both of the above approaches are combined. The server: transmits the rendering information to an intermediate wireless device connected to one of the AR-enabled devices; and based on positioning information received by the intermediate wireless device from the AR- enabled device, the intermediate wireless device renders a video-based AR overlay for the AR-enabled device. The intermediate wireless device then sends the video-based AR overlay to the AR-enabled device.

[0012] AR may also be used in other contexts, such as for events comprising large audiences (e.g., music concerts, political or corporate rallies, etc.). For example, AR can be used during music concerts to display lyrics of a song or additional information (such as dancers) to enhance a user experience.

[0013] Regarding video-based AR overlays, new video compression standards have been developed over the past decade. Each new generation has increased compression efficiency through technological advances, but this improvement has been at the cost of increased computational complexity of encoders, decoder software, and decoder hardware. To offset this increased computational complexity, improvements in processing power have been made. As a result, new video compression standards allow for improved video compression in comparable times to older video compression standards.

[0014] Recent standards, such as High Efficiency Video Coding (HEVC) and Versatile Video Coding (WC), comprise the following steps:

[0015] - dividing each picture (which may also referred to herein as an image or a frame) of a video into coding blocks;

[0016] - creating a prediction for each coding block, either from other parts of the same picture (called intra coding) or from one or two previously decoded pictures (called inter coding);

[0017] - calculating a residual between the prediction and actual sampled values; performing transform coding and quantization of the residual; and using entropy coding to efficiently compress the quantized values.

[0018] In HEVC and WC a picture comprises one or more slices. Slices in HEVC and WC are typically independent, meaning decoding of a slice in a picture is independent of other slices of the same picture. One purpose of slices is to enable resynchronization in case of data loss. A slice is carried in a single network abstraction layer (NAL) unit.

[0019] A picture in HEVC and WC may also be divided into several tiles distributed over M tile rows x N tile columns. Tiles are typically used for parallel processing of video data in a picture.

[0020] In WC, a picture may also comprise one or more subpictures, where a subpicture (which may also be referred to herein as a subimage or subframe) may be defined as a rectangular region of one or more slices within a picture. A subpicture is independent from other subpictures of the picture and are, in contrast to slices, guaranteed to be independent over time. That is, a subpicture in a current picture may only be predicted from the same subpicture location of a previously decoded picture, and not from subpicture belonging to other locations. This allows subpictures to be extracted and merged at a bitstream level without having to first decode the (whole) picture. A similar functionality is supported in HEVC through temporal motion constrained tile sets (MCTS) Supplemental Enhancement Information (SEI) messages, which constrain a prediction for specified groups of tiles. SEI messages are typically optional for the decoder to support.

[0021] One extension to HEVC and WC is scalable video coding. In HEVC, support for scalable coding was added in version 2 of the standard in the Scalable Main and Scalable Main 10 profiles, whereas in WC scalable coding was part of the first version of the standard in the Multilayer Main 10 and Multilayer Main 10 4:4:4 profiles. The concept behind scalable video coding is to have a base layer (BL) at a lower quality and an enhancement layer (EL) that contains additional information that the decoder can use to increase the quality of the decoded video. Scalable video coding can be used for:

[0022] - spatial or temporal resampling, and / or

[0023] - quality improvements such as higher bit depth, different chroma subsampling, or signal-to-noise (SNR) scalability, which typically includes varying the level of quantization of the residual (i.e., varying the quantization parameter).

[0024] Other aspects that can be considered when using scalable video coding are additional error resiliency and graceful declines in quality.

[0025] Many AR-enabled devices, such as glasses, provide audio support in addition to video support (e.g., in the form of open ear headphones). That is, they can output an audiobased AR overlay and / or display a video-based AR overlay. As such, real-world or outdoor noises can also be provided to a user whilst they are viewing a video-based AR overlay. The three methods of AR overlay rendering discussed above are applicable to audio-based AR overlays.

[0026] An audio-based AR overlay may be rendered based on a location of the AR-enabled device to provide contextual information (e.g., spatially dependent information), or rendered as a background score without considering the location of the AR-enabled device.

[0027] For example, audio-based AR overlays may comprise spatial audio, which refers to technology that gives a listener the impression that sound is coming from points on a sphere around their head. It provides an immersive experience by rendering audio either binaurally via headphones or as audio objects via loudspeakers. Binaural rendering has become common and there are many headsets and earphones in the market that have the ability to deliver spatial audio. More advanced capabilities of AR-enabled devices may include binaurally rendering spatial audio, which improves the user’s immersive experience. It may have three Degrees of Freedom (DoF) (i.e., changing with head orientation) or six DoF (i.e., changing with head orientation and movement in space).

[0028] It is possible for audio-based AR overlays and video-based AR overlays to be completely decoupled and rendered individually.

[0029] SUMMARY

[0030] In current solutions all users / AR-enabled devices receive an AR stream that is specific to them. If there are many AR-enabled devices in close proximity to each other, all AR- enabled devices will get a slightly different stream compared to the AR-enabled devices next to them. This has several implications:

[0031] 1 . In applications where only rendering information is transmitted, there will be many AR-enabled devices that render nearly the same scene. As rendering requires significant processing power, different devices will each expend energy to create nearly the same result.

[0032] 2. In applications where the AR overlay is transmitted from the server, the server will use a lot of processing power to generate many instances of nearly the same AR overlay for users that are close to each other.

[0033] 3. The capacity of the network may not be large enough to handle ‘single casting’ of AR streams to each user, or the quality of the AR stream for some or all AR- enabled devices may need to be reduced in order to stay within the network capacity.

[0034] Certain aspects of the disclosure and their embodiments may provide solutions to these or other challenges. In embodiments of the present disclosure, AR-enabled devices that are geographically close to each other are clustered or grouped together, such that one or more groups of AR-enabled devices are obtained. The clusters or groups may be static (i.e., the devices in the cluster or group does not change over time), or dynamic (i.e., the devices in the cluster or group may change over time). Information (referred to as ‘first information’) for generating an AR overlay is determined for each group based on a location of the respective group. All users in a respective group receive at least part of the corresponding first information. Based on a viewing direction of an AR-enabled device in a respective group, the AR-enabled devices may display a subset (i.e., a part) of the corresponding first information. The first information may be for generating a videobased AR overlay and / or an audio-based AR overlay.

[0035] According to a first aspect of the present disclosure, there is provided a computer- implemented method for facilitating the generation of AR overlays on a first group of AR- enabled devices located in a first area, the first group of AR-enabled devices comprising more than one AR-enabled device. The method comprises determining, for the first group of AR-enabled devices, first information for generating an AR overlay, wherein the first information is determined based on a location of the first group of AR-enabled devices. The method further comprises sending, to one or more first wireless devices, at least part of the first information. Each of the first wireless devices is one of: a respective AR device in the first group of AR-enabled devices; or an intermediate wireless device connected to one or more of the first group of AR-enabled devices.

[0036] According to a second aspect of the present disclosure, there is provided a method performed by a first wireless device for generating an AR overlay on an AR-enabled device. The method comprises receiving first information for generating the AR overlay, wherein the first information comprises a rendered video bitstream comprising a plurality of sequential frames, each frame being made up of one or more frame segments. The method further comprises selecting at least part of the rendered video bitstream, wherein each frame of the at least part of the rendered video bitstream comprises at least one frame segment corresponding to a viewing direction of the AR-enabled device. The first wireless device is one of: the AR-enabled device; or an intermediate wireless device connected to the AR-enabled device.

[0037] According to a third aspect of the present disclosure, an apparatus is provided for facilitating the generation of AR overlays on a first group of AR-enabled devices located in a first area, the first group of AR-enabled devices comprising more than one AR- enabled device. The apparatus comprises processing circuitry configured to cause the apparatus to perform any of the embodiments of the first aspect.

[0038] According to a fourth aspect of the present disclosure, a first wireless device for generating an AR overlay on an AR-enabled device, wherein the first wireless device is one of: the AR-enabled device; or an intermediate wireless device connected to the AR- enabled device. The first wireless device comprising processing circuitry configured to cause the apparatus to perform any of the embodiments of the second aspect. According to a fifth aspect of the present disclosure, a computer program product is provided. The computer program product comprises a computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform the method of any of the embodiments of the first or second aspect.

[0039] Certain embodiments may provide one or more of the following technical advantage(s): processing resources can be conserved at servers and / or base stations, which do not need to determine rendering information for an AR overlay per AR-enabled device in a group of AR-enabled devices in a given area. processing resources can be conserved at servers and / or base stations, which do not need to render an AR overlay per AR-enabled device in a group of AR-enabled devices in a given area.

[0040] If the server transmits a rendered AR overlay to AR-enabled devices, the AR- enabled devices can save a significant amount of energy as they do not need to perform any rendering; they can instead decode the transmitted rendered AR overlay and display it. Furthermore, the server does not need to run an encoding instance (for the rendered AR overlay) per AR-enabled device, as all AR-enabled devices in the group receive at least part of the same (encoded) first information.

[0041] If the server transmits information for rendering an audio-based AR overlay or a rendered audio-based AR overlay to AR-enabled devices in a group, the AR-enabled devices in the group can apply relevant filters based on their orientation for spatial audio rendering.

[0042] The server traffic can fit within a given network capacity, or alternatively the network capacity can be scaled to fit a “broadcasting” scenario of AR traffic.

[0043] BRIEF DESCRIPTION OF THE DRAWINGS

[0044] For a better understanding of examples of the present disclosure, and to show more clearly how the examples may be carried into effect, reference will now be made, by way of example only, to the following drawings in which:

[0045] Figure 1 is a schematic diagram of an AR-enabled device in communication with a server;

[0046] Figure 2 is a schematic diagram of an AR-enabled device in communication with a server via an intermediate wireless device; Figure 3 illustrates a first example of groups of AR-enabled devices;

[0047] Figure 4 illustrates a second example of groups of AR-enabled devices;

[0048] Figure 5 is a flowchart showing a method in accordance with embodiments of the disclosure;

[0049] Figure 6 illustrates a viewing direction of an AR-enabled device;

[0050] Figure 7 is a schematic diagram illustrating a picture which is divided into a plurality of subpictures;

[0051] Figure 8 illustrates sub-groups of AR-enabled devices;

[0052] Figure 9 illustrates one or more sub-groups of AR-enabled devices;

[0053] Figure 10 is a flowchart showing a method in accordance with embodiments of the disclosure;

[0054] Figure 11 is a schematic diagram of an apparatus in accordance with embodiments of the disclosure;

[0055] Figure 12 is a schematic diagram of a first wireless device in accordance with embodiments of the disclosure; and

[0056] Figure 13 is a schematic diagram of a virtualization environment in which embodiments of this disclosure can be implemented.

[0057] DETAILED DESCRIPTION

[0058] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Other embodiments, however, are contained within the scope of the subject matter disclosed herein. The disclosed subject matter should not be construed as limited to only the embodiments set forth herein; rather, these embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0059] As an overview, in embodiments of the present disclosure, first information for generating an AR overlay is determined for one or more groups of AR-enabled devices, where each group contains AR-enabled devices that are geographically close to one another and the first information determined for the group is based on a location of the group. Respective AR-enabled devices in a group receive at least part of the first information determined for the group. Based on the viewing direction of a respective AR-enabled device in the group, the AR-enabled device may determine which part of the at least part of the first information to display and / or render. In embodiments of the present disclosure, an AR-enabled device may be any one of: a head mounted display (HMD); optical AR glasses; pass-through AR glasses; a screen device (i.e., a device with a screen); a mobile phone; a tablet; and a computer. An AR- enabled device is also referred to herein as an “AR device”.

[0060] As illustrated by Figure 1 , an AR-enabled device 102 may be able to communicate ‘directly’ with a server 104 (or, equivalently, a base station or other suitable network node). As used herein, communicating ‘directly’ means that there is no intermediate processing of the information exchanged between the AR-enabled device 102 and the server 104 for the purposes of determining and generating an AR overlay by the AR- enabled device. As such, communicating directly does not necessarily mean that the AR-enabled device 102 is physically connected to the server 104, or has a wireless connection direct to the server 104. The AR-enabled device 102 may be able to send information (e.g., positioning information, including location and / or viewing direction information) directly to the server 104, and receive information (e.g., rendering information or a rendered AR overlay) determined by the server 104 directly from the server 104. The AR-enabled device 102 may be responsible for rendering the AR- overlay.

[0061] As an alternative to the ‘direct’ communication shown in Figure 1 , embodiments of the present disclosure provide for ‘indirect’ communications between the AR-enabled device and the server / base station via an intermediate wireless device. The intermediate wireless device performs some processing of the information exchanged between the AR-enabled device and the server. An intermediate wireless device may be any one of: a user device, such as a mobile phone, capable of communication with (i.e., connecting to) one or more AR-enabled devices; a device capable of serving (i.e., connecting to) one or more AR-enabled devices, such as an edge node. An intermediate wireless device is also capable of connecting to a server or base station of a network. An intermediate wireless device is also referred to herein as an “intermediate device”.

[0062] Figure 2 illustrates an AR-enabled device 202 communicating indirectly with a server 204 (or, equivalently, a base station or other suitable network node) via an intermediate wireless device 206 to which the AR-enabled device is connected. When an AR-enabled device 202 is connected to the intermediate wireless device 206, typically all data is routed via the intermediate wireless device 206. For example, the AR-enabled device 202 may send information (e.g., positioning information, including location and / or viewing direction information) indirectly to the server 204 via the intermediate wireless device 206. The intermediate wireless device 206 may then receive information (e.g., rendering information or a rendered AR overlay) determined by the server 204 directly from the server 204, and forward this information to the AR-enabled device 202. The intermediate wireless device 206 may itself determine information (e.g., render an AR overlay or decode rendering information) based on the information it receives from the server 204, and send this information directly to the AR-enabled device 202. As such, processing (e.g., rendering) for generating an AR overlay can be offloaded from the AR-enabled device 202 to the intermediate wireless device 206. As such, the energy consumption and / or resource usage of the AR-enabled device 202 can be reduced.

[0063] A group of AR-enabled devices comprises AR-enabled devices that have been assigned to the group. A group may also be referred to herein as a cluster. One or more groups of AR-enabled devices may be determined by assigning AR-enabled devices to a group based on the locations of the AR-enabled devices (or the associated intermediate wireless device).

[0064] For example, Figure 3 illustrates a first example of assigning AR-enabled devices to a number of groups 302, 304, 306, 308. In figure 3, AR-enabled devices in use by an audience 310 to view a scene of interest 312 (e.g., a sports match) are distributed over a range of areas around the scene 312. Although not required, each area may be defined by fixed borders (e.g., borders defined by Global Positioning System (GPS) coordinates or similar positioning systems).

[0065] The AR-enabled devices are divided into groups (i.e., grouped together) based on the geographical (i.e., physical) position or location of each respective AR-enabled device or their associated intermediate device, if present. In the following, reference to the grouping and / or location of AR-enabled devices should be understood as also or alternatively referring to the grouping and / or location of a respective intermediate device where the AR-enabled device is connected to an intermediate device that handles some of the processing for generating the AR overlay. The geographical position of an AR- enabled device may, for instance, be determined from GPS coordinates of the AR- enabled device, or a proximity of the AR-enabled device to a radio or network node connected to the server (e.g., an edge node). An AR-enabled device located or positioned within a first area (of the plurality of areas) is assigned to a first group 302 that includes other devices associated with the first area, an AR-enabled device located or positioned within a second area of the plurality of areas is assigned to a second group 304 that includes other devices associated with the second area, and so on.

[0066] The grouping shown in Figure 3 may be static, meaning that once an AR-enabled device has been assigned to a group (e.g., for a specified period of time), the AR-enabled device remains in its assigned group even if it enters a region located within an area associated with a different group. This may be useful in embodiments where respective areas associated with two or more of the groups 302, 304, 306, 308 overlap. For example, an AR-enabled device may be assigned to a first group 302 associated with a first area, and the AR-enabled device may then enter a region of a second area associated with the second group 304 that overlaps with the first area. By being statically assigned to the first group 302, the AR-enabled device can be prevented from being assigned to the second group 304 whilst it is located in the overlap region. Static grouping can also be useful where it is expected that the location of AR-enabled devices is unlikely to significantly change, e.g., in the case of the user of the AR-enabled device having an allocated seat in a sports stadium.

[0067] Figure 4 illustrates a second example of assigning AR-enabled devices to a number of groups 402, 404, 406. In Figure 4, AR-enabled devices in use by members of an audience 408 for viewing a scene of interest 410 are divided into groups 402, 404, 406 based on a number of users in an area. That is, AR-enabled devices in close proximity to one another may be allocated or assigned to a group (e.g., a first group 402), and the first group 402 may have a maximum number of AR-enabled devices that can be assigned to it (e.g., 10). Other AR-enabled devices in proximity to one another may be assigned to a second group 404, where the second group 404 may have a maximum number of AR-enabled devices that can be assigned to it (e.g., 30). For example, a group of AR-enabled devices may be determined by assigning an AR-enabled devices to the group based on a proximity of the AR-enabled device to an AR-enabled device already assigned to the respective group.

[0068] The AR-enabled devices may be dynamically grouped; e.g., the groups 402, 404, 406 may be configured such that, if AR-enabled devices assigned to a group are no longer in close proximity to one another and instead start to cover a larger area with other users in-between (i.e., the AR-enabled devices in a group have a reduced concentration due to disbursement of the AR-enabled devices), the group might be discontinued. The AR- enabled devices of the discontinued group may be assigned to a new group.

[0069] One or more groups of AR-enabled devices may also be determined by assigning AR- enabled devices to a respective group (e.g., the first group) based on a connection of the AR-enabled devices to an intermediate wireless device associated with the first group (e.g., an edge node).

[0070] In some embodiments, different groups / groupings can be used for video AR overlays and audio AR-overlays, which means that an AR-enabled device may be assigned to more than one group. For example, an AR-enabled device may be assigned to a group for the purposes of receiving at least part of first information for generating a video-based AR overlay, and assigned to a different group for the purposes of receiving at least part of first information for generating an audio-based AR overlay.

[0071] Figure 5 illustrates a computer-implemented method according to embodiments of the present disclosure. The method is for facilitating the generation of AR overlays on a first group of AR-enabled devices located in a first area, the first group of AR-enabled devices comprising more than one AR-enabled device (e.g., the AR-enabled devices 102, 202 of Figures 1 and 2). The method of figure 5 is described below as being performed by a server (e.g., a central server of a network, such as server 104, 204 of Figures 1 and 2). However, the method may also be performed by a base station or any other suitable network node capable of communication with an AR-enabled device and / or an intermediate wireless device.

[0072] The first group may be determined according to the methods discussed above (e.g., in relation to Figures 3 and 4). For example, the first group may be any of groups 302, 304, 306, 308 of Figure 3, or any of groups 402, 404, 406 of Figure 4. 3. For example, the first group of AR-enabled devices may be determined by assigning an AR-enabled device to the first group based on any one of: a proximity of the AR-enabled device to an AR- enabled device already assigned to the first group; the AR-enabled device being located within a geographical area associated with the first group; and a connection of the AR- enabled device to an intermediate wireless device associated with the first group. Whilst not illustrated, the method of Figure 5 may begin with a step of obtaining an indication of a location of one or more AR-enabled devices (e.g., from the AR-enabled devices themselves or from an intermediate wireless device to which the AR-enabled device are connected). For example, the indication may indicate a location of an intermediate wireless device connected to the AR-enabled device. This location indication can be used to group the AR-enabled devices, or otherwise assign an AR- enabled device to a group.

[0073] At step 502, the method comprises determining, for the first group of AR-enabled devices, first information for generating an AR overlay. The first information is determined based on a location of the first group.

[0074] The location may be based on an estimated centre of the first area (in which the first group is located). The estimated centre may be determined based on GPS coordinates of the borders of the first area (or any other similar positioning coordinates). This embodiment is illustrated by Figure 6, in which an estimated centre of a first group of AR-enabled devices 602 is shown as the intersection point 603. As a result, for the purposes of determining the first information, the assumed position of an AR-enabled device in the first group 602 is the middle or centre of the first area (i.e., the intersection point 603).

[0075] Returning to step 502, the location may additionally or alternatively be based on a location of an intermediate wireless device connected to one or more of the AR-enabled devices of the first group. The server may be configured with this information, or the intermediate wireless device may transmit this information to the server.

[0076] The first information may comprise information for rendering a video stream (also referred to herein as video rendering information), e.g., for a video-based AR overlay. The video rendering information may include information for any one or more of: objects to render; positions of the objects to render; positions and / or viewing directions of an AR- enabled device in the first group (if the server has previously received or otherwise determined this information); a location of the centre of the first group; and a position of an edge node in the first group. The server may be able to re-use parts of the video rendering information of the first information for other groups of AR-enabled devices. Alternatively, the first information may comprise a rendered video bitstream. That is, the server may first determine the video rendering information for the first group, and use the video rendering information to render a video-based AR overlay for display by the AR-enabled devices. The server may then encode the video-based AR overlay (e.g., using a video encoder) to obtain a rendered video bitstream, e.g., to facilitate transmission of the video-based AR overlay to the AR-enabled devices of the first group. The rendered video bitstream provides an AR overlay for a scene of interest, and part of the rendered video bitstream is selectable (e.g., by an AR-enabled device, intermediate device, or server) for display by the AR-enabled device.

[0077] The rendered video bitstream may comprise a plurality of sequential pictures (which are also referred to as “frames”), and each picture / frame may be made up of one or more picture / frame segments. A segment may correspond to a (independently coded) subpicture, subframe, slice, or a tile of a picture. For example, Figure 7 illustrates a picture 702 of a rendered video bitstream divided vertically into several subpictures 704, 706, 708.

[0078] Returning to figure 5, the segments are selectable / suitable for selection (e.g., by an AR- enabled device, intermediate device, or server) to be displayed by an AR-enabled device based on a viewing direction of the AR-enabled device. A segment may correspond to a subpicture, slice, or a tile of a picture.

[0079] A viewing direction may be defined as the direction of a vector perpendicular to the plane of a view port (e.g., the screen) of the AR-enabled device. A viewing direction may also be referred to herein as a viewing orientation, viewing angle, orientation, or head orientation. As illustrated by Figure 6, an AR-enabled device having a particular viewing direction corresponds to a user of the AR-enabled device having a particular viewing area 606. The particular viewing area 606 may overlap with all or part of a scene of interest 612 (see where the dotted line 604 overlaps with the viewing area 606). As such, the AR-enabled device (through which a user of audience 610 intends to view the scene of interest 612) should at least display (on a view port of the AR-enabled device) an AR overlay corresponding to the part of the scene of interest overlapping with the viewing area 606 of the AR-enabled device.

[0080] Returning to step 502, the rendered video bitstream may also be a layered video bitstream. A layered video bitstream comprises a base layer (BL) and at least one enhancement layer (EL). The base layer, once decoded, provides each picture of the rendered video bitstream at a basic picture quality. The at least one enhancement layer is divided (e.g., vertically) into a plurality of EL segments. Each EL segment, once decoded, provides a picture segment of the rendered video bitstream at a higher picture quality than the BL. That is, the at least one EL has a higher picture quality than the BL. A higher picture quality may correspond to a larger spatial resolution, higher temporal resolution, a higher bit depth, a higher SNR quality, and / or a closer resemblance to a pre-encoded version of the rendered video bitstream.

[0081] Displaying a part of the rendered layered video bitstream may comprise displaying the base layer and a part of the at least one enhancement layer corresponding to the part of the rendered video bitstream. The part of the at least one enhancement layer may comprise at least one of: at least one independently coded subpicture; at least one slice; and at least one tile.

[0082] A rendered layered video bitstream is beneficial in embodiments where a viewing direction of an AR-enabled device changes quickly. For example, if the AR-enabled device quickly changes viewing direction, it is able to display a video-based AR overlay for the new viewing direction (i.e., the BL) even if the AR-enabled device has not received / decoded / rendered the EL segments corresponding to the new viewing direction of the AR-enabled device, albeit at only a basic video quality.

[0083] Additionally or alternatively to the above embodiments wherein the first information comprises video rendering information or a rendered video bitstream, the first information may comprise information for rendering an audio stream (also referred to herein as audio stream rendering information) or a rendered audio bitstream. For example, the server may determine audio rendering information for the first group, and either: use the audio rendering information to create / render an audio-based AR overlay; or

[0084] - transmit at least part of the audio rendering information to respective AR-enabled devices in the first group, or to one or more intermediate wireless devices to which AR-enabled devices in the first group are connected.

[0085] It should be appreciated that audio and video AR overlays can be decoupled and / or rendered individually. The server may encode the audio-based overlay (e.g., using an audio encoder) into a rendered audio bitstream (e.g., to facilitate the transmission of the audio-based AR overlay to the AR-enabled devices of the first group). The rendered audio bitstream provides an AR overlay for a scene of interest, and all or part of the rendered audio bitstream is selectable (e.g., by an AR-enabled device, intermediate device, or server) for output by the AR-enabled device.

[0086] At step 504 of Figure 5, the method comprises sending, to one or more first wireless devices (e.g., a respective AR device in the first group of AR-enabled devices or an intermediate wireless device connected to one or more of the first group of AR-enabled devices) at least part of the first information.

[0087] The at least part of the first information sent to one of the one or more first wireless devices may be based on: a viewing direction of the first wireless device, if the first wireless device is an AR-enabled device; or a viewing direction of an AR-enabled device connected to the first wireless device, if the first wireless device is an intermediate wireless device. That is, the server may determine which parts of the first information are needed for an AR-enabled device or intermediate wireless device connected thereto to render and / or generate an AR overlay to be displayed / output by an AR-enabled device based on its viewing direction. For example, as illustrated by Figure 6, an AR-enabled device may only need an AR overlay for the part of the viewing area 606 that overlaps with the scene of interest.

[0088] For this purpose, the server may determine a viewing direction and / or position of an AR- enabled device of the first group relative to a scene of interest (e.g., by receiving periodic updates from the AR-enabled device or an intermediate wireless device connected thereto, which indicate the position and / or viewing direction of the AR-enabled device).

[0089] In step 504, the at least part of the first information may comprise all or part of the rendered video bitstream.

[0090] Once the one or more one or more first wireless devices have received all or part of the rendered video bitstream, the one or more first wireless devices may select different parts of the rendered video bitstream for display by the AR-enabled devices (e.g., depending on the viewing directions of the AR-enabled device). If the first AR-enabled device receives all or part of the rendered video bitstream (either from the server or an intermediate device), selecting the parts of the rendered video bitstream for display by the AR-enabled devices may comprise the AR-enabled device decoding only parts (e.g., picture segments) that are to be displayed by the AR-enabled based on its viewing direction. In the event that the rendered video bitstream is a layered video bitstream, this comprises decoding the BL and decoding only EL segments that are to be displayed by the AR-enabled based on its viewing direction. Using the decoded parts of the rendered video bitstream, the AR-enabled device is able to obtain a videobased AR overlay and display the video-based AR overlay.

[0091] If an intermediate wireless device connected to an AR-enabled device receives all or part of the rendered video bitstream, selecting the parts of the rendered video bitstream for display by the AR-enabled devices may comprise the intermediate wireless device routing (i.e., sending), to an AR-enabled device, only the parts (e.g., picture segments) that are to be decoded and displayed by the AR-enabled device based on the viewing direction of the AR-enabled device. In the event that the rendered video bitstream is a layered video bitstream, this comprises routing, to the AR-enabled devices, the BL and only EL segments that are to be displayed by the AR-enabled based on its viewing direction. The AR-enabled can then decode the parts of the rendered video bitstream to obtain a video-based AR overlay and display the video-based AR overlay.

[0092] If an intermediate wireless device connected to an AR-enabled device receives all of the parts of the rendered video bitstream, selecting the parts of the rendered video bitstream for display by the AR-enabled devices may comprise the intermediate wireless device decoding only the parts (e.g., picture segments) that are to be displayed by the AR- enabled device based on the viewing direction of the AR device, and creating / rendering the video-based AR overlay for the AR-enabled device. In the event that the rendered video bitstream is a layered video bitstream, this comprises decoding the BL and decoding only EL segments that are to be displayed by the AR-enabled based on its viewing direction. The intermediate device can then send the video-based AR overlay to the AR-enabled device. The AR-enabled can then display the video-based AR overlay.

[0093] Step 504 may comprise sending, to the one or more first wireless devices, all or part of the video rendering information.

[0094] Once the one or more first wireless devices have received all or part of the video rendering information, the one or more first wireless devices may select different parts of the video rendering information for decoding and using to render a video-based AR overlay for display by the AR-enabled devices (e.g., depending on the viewing directions of the AR-enabled device).

[0095] If an AR-enabled device receives all or part of the video rendering information (either from the server or the intermediate device), selecting the parts of the video rendering information may comprise the AR-enabled device decoding only parts of the video rendering information needed to render picture segments that are to be displayed by the AR-enabled device based on its viewing direction. The AR-enabled may use the decoded parts of the video rendering information to render and display a video-based AR overlay. If an intermediate wireless device connected to an AR-enabled device receives all or part of the video rendering information, selecting the parts of the video rendering information may comprise the intermediate wireless device routing (i.e., sending) only parts of the video rendering information needed to render picture segments that are to be displayed by the AR-enabled device based on its viewing direction. Once the AR- enabled has received and decoded the relevant parts of the video rendering information, it may render and display a video-based AR overlay.

[0096] If an intermediate wireless device connected to an AR-enabled device receives all of part of the video rendering information, selecting the parts of the video rendering information may comprise the intermediate wireless device decoding only parts of the video rendering information needed to render picture segments that are to be displayed by the AR-enabled device based on its viewing direction, and creating / rendering the videobased AR overlay for the AR-enabled device. The intermediate device can then send the video-based AR overlay to the AR-enabled device. The AR-enabled device can then display the video-based AR overlay.

[0097] In step 504, the at least part of the first information may comprise all or part of the rendered audio bitstream.

[0098] Once the one or more first wireless devices have received all or part of the rendered video bitstream, the one or more first wireless devices may select different parts of the rendered audio bitstream for output by the AR-enabled devices (e.g., depending on their orientation), such that a rendered audio-based AR overlay is suited for a particular head orientation. If an AR-enabled device receives all or part of the rendered audio bitstream direct from the server, selecting the parts of the rendered audio bitstream for output by the AR- enabled devices may comprise the AR-enabled device decoding only parts of the rendered audio bitstream that are to be output by the AR-enabled based on its orientation. Using the decoded parts of the rendered video bitstream, the AR-enabled device is able to obtain an audio-based AR overlay and output the audio-based AR overlay.

[0099] If an intermediate wireless device connected to an AR-enabled device receives all or part of the rendered audio bitstream, selecting the parts of the rendered audio bitstream for output by the AR-enabled devices may comprise the intermediate wireless device routing (i.e., sending), to an AR-enabled device, only the parts of the rendered audio bitstream that are to be decoded and output by the AR-enabled device based on the orientation of the AR-enabled device. The AR-enabled can then decode the parts of the rendered audio bitstream to obtain an audio-based AR overlay and output the audiobased AR overlay.

[0100] If an intermediate wireless device connected to an AR-enabled device receives all of part of the rendered audio bitstream, selecting the parts of the rendered audio bitstream for output by the AR-enabled devices may comprise the intermediate wireless device decoding only the parts of the rendered audio bitstream that are to be output by the AR- enabled device based on the orientation of the AR device, and creating / rendering the audio-based AR overlay for the AR-enabled device. The intermediate wireless device can then send the audio-based AR overlay to the AR-enabled device. The AR-enabled can then display the audio-based AR overlay.

[0101] In step 504, the at least part of the first information may comprise all or part of the audio rendering information.

[0102] Once the one or more first wireless devices have received all or part of the audio rendering information, the one or more first wireless devices may select different parts of the audio rendering information for decoding and using to render an audio-based AR overlay for display by the AR-enabled devices (e.g., depending on the orientation of the AR-enabled device). If an AR-enabled device receives all or part of the audio rendering information (either from the server or the intermediate wireless device), selecting the parts of the audio rendering information may comprise the AR-enabled device decoding only parts of the audio rendering information needed to render an audio-based AR overlay that is to be output by the AR-enabled device based on its orientation. The AR-enabled may use the decoded parts of the audio rendering information to render and display an audio-based AR overlay.

[0103] If an intermediate wireless device connected to an AR-enabled device receives all or part of the audio rendering information, selecting the parts of the audio rendering information may comprise the intermediate wireless device routing (i.e., sending) only parts of the audio rendering information needed to render an audio-based AR overlay that is to be output by the AR-enabled device based on its orientation. Once the AR- enabled device has received and decoded the relevant parts of the audio rendering information, it may render and display an audio-based AR overlay.

[0104] If an intermediate wireless device connected to an AR-enabled device receives all of part of the audio rendering information, selecting the parts of the audio rendering information may comprise the intermediate wireless device decoding only parts of the audio rendering information needed to render an audio-based AR overlay that is to be output by the AR-enabled device based on its orientation, and creating / rendering the audiobased AR overlay for the AR-enabled device. The intermediate wireless device can then send the audio-based AR overlay to the AR-enabled device. The AR-enabled can then display the audio-based AR overlay.

[0105] For the purposes of step 504, an AR-enabled device may be configured to periodically update the intermediate wireless device as to its position and viewing direction / orientation.

[0106] In embodiments where the one or more first wireless devices receive part of the first information, the one or more first wireless devices may be in various sub-group of the first group. That is, respective parts of the first information may be sent to different subgroups of AR-enabled devices of the first group, wherein AR-enabled devices in a respective sub-group have similar or the same viewing direction. A sub-group may be referred to herein as a sub-cluster. As such, the method of figure 5 may further comprise assigning the AR-enabled devices of the first group to one or more sub-groups (e.g., according to the method illustrated by Figure 8, discussed below). This may involve a step of determining a viewing direction of each of the AR-enabled devices of the first group of AR-enabled devices relative to a scene of interest, and grouping (i.e., dividing) the first group into a plurality of sub-groups of AR-enabled devices based on the determined viewing directions. That is, AR-enabled devices with similar or the same viewing direction may be assigned to a particular subgroup. The similarity of viewing directions of AR-enabled devices may be determined by the server based on information received from the AR-enabled devices or intermediate wireless devices connected thereto. That is, AR-enabled devices (and / or intermediate devices connected thereto) may be configured to periodically update the server as to the position and viewing direction / head orientation of the AR-enabled devices (e.g., according to an update frequency). The sub-grouping of the first group may be performed periodically or dynamically.

[0107] For example, Figure 8 illustrates assignment of AR-enabled devices in a first group / cluster 804 to one or more sub-groups. In Figure 8, the server 802 receives, from the AR-enabled devices of the first group 804 (or intermediate wireless devices connected thereto), information relating to the viewing directions of the AR-enabled devices of the first group 804 (i.e., the “head orientation information”). The server 802 then assigns the AR-enabled devices with the same or similar viewing directions to a particular sub-group (e.g., to sub-cluster 1 806).

[0108] If the first information is for generating both audio-based and video-based AR overlays on the AR-enabled devices of the first group, one or more of the sub-groups of the first group may overlap (e.g., one or more first wireless devices may be in more than one sub-group). For example, as illustrated in Figure 9, a first wireless devices (i.e., AR Device 11 900) of first group 902 may be in a first sub-group (e.g., “audio cluster 1” 904) in order to receive information for generating a video-based AR overlay, and the first wireless devices may also be in a second sub-group (e.g., “video cluster 1” 906) in order to receive information for generating an audio-based AR overlay.

[0109] Returning to figure 5, in embodiments where the first group is divided into sub-groups, step 504 may comprise sending (as the at least part of the first information) at least part of the rendered video bitstream to one or more first wireless devices associated with a respective sub-group. Each picture of the at least part of the rendered video bitstream may comprise at least one picture segment corresponding to a viewing direction associated with the respective sub-group. In this embodiment, the AR-enabled devices (or intermediate wireless devices connected thereto) may not need to determine, per AR- enabled device, which picture segments are to be displayed by the AR-enabled based on its viewing direction.

[0110] Step 504 may instead comprise sending (as the at least part of the first information) at least part of the video rendering information to one or more first wireless devices associated with a respective sub-group, wherein the at least part of the first information corresponds to a viewing direction associated with the respective sub-group. In this embodiment, the server may not need to determine, per AR-enabled device, which parts of the video rendering information to send to the AR-enabled devices.

[0111] Additionally or alternatively, step 504 may comprise sending (as the at least part of the first information) at least part of the rendered audio bitstream to one or more first wireless devices associated with a respective sub-group, wherein the at least part of the rendered audio bitstream corresponds to a viewing direction associated with the respective subgroup. As such, the part of the rendered audio bitstream output by the AR-enabled device may be suited to a particular viewing direction of the AR-enabled device. In this embodiment, the AR-enabled devices (or intermediate wireless devices connected thereto) may not need to determine which part of the rendered audio bitstream to output based on its viewing direction. This is beneficial in scenarios where it would be computationally expensive for each AR-enabled device in a respective sub-group to process the rendered audio stream or process the audio rendering information to generate an AR overlay with 3 or 6 DoF.

[0112] Step 504 may instead comprise sending at least part of the audio rendering information to one or more first wireless devices associated with a respective sub-group, wherein the at least part of audio rendering information corresponds to a viewing direction associated with the respective sub-group. As such, the part of the audio rendering information output by the AR-enabled device is suited to a particular viewing direction of the AR-enabled device. In this embodiment, the server may not need to determine, per AR-enabled device, which parts of the audio rendering information to send the AR- enabled devices. It should be appreciated that, whilst the above embodiments are discussed in relation to first information being determined for a first group of AR-enabled devices based on the location of the first group, embodiments of the present disclosure also further facilitate the generation of AR overlays on a second group of AR-enabled devices located in a second area. The second group of AR-enabled devices also comprises more than one AR-enabled device, and at least one of the AR-enabled devices of the second group is not included in the first group.

[0113] Such embodiments may facilitate the generation of AR overlays on AR-enabled devices of the second group in a similar manner to the way in which embodiments facilitate the generation of AR overlays on AR-enabled devices of the first group.

[0114] For example, the method of figure 5 may further comprise:

[0115] - determining, for the second group of AR-enabled devices, second information for generating an AR overlay based on a location of the second group of AR-enabled devices; and

[0116] - sending, to one or more second wireless devices, at least part of the second information.

[0117] Each of the second wireless devices is one of: a respective AR-enabled device in the second group of AR-enabled devices; or an intermediate wireless device connected to one or more of the second group of AR-enabled devices.

[0118] As with the first information, the second information may comprise: information for rendering a video stream (e.g., the above discussed video rendering information) or a rendered video bitstream (e.g., the above discussed rendered video bitstreams); and / or information for rendering an audio stream (e.g., the above discussed audio rendering information) or a rendered audio bitstream (e.g., the above discussed rendered audio bitstreams).

[0119] The embodiments for determining the second information may correspond to the above discussed embodiments for determining the first information. However, the second information is different than the first information. For example, the first information may be further determined based on at least one first characteristics associated with the first group, and / or the second information may be further determined based on at least one second characteristics associated with the second group.

[0120] The at least one first characteristic and at least one second characteristic relate to AR overlay rendering or generation. For example, the at least one first characteristic may belong to any one or more of the following characteristic categories: a type of AR overlay (e.g., video or audio), frame quality, a level of AR features, a language, and priority. The at least one second characteristic may belong to any one or more of the following characteristic categories: a type of AR overlay (e.g., video or audio), a frame quality, a level of AR features, a language, and priority.

[0121] The at least one first characteristic and at least one second characteristic may be different characteristics within the same characteristic category.

[0122] For example, the at least one first characteristic may indicate that (the at least part of) the first information received by the first group is of a better quality than (the at least part of) the second information received by the second group. A “better quality” may refer to any one or more of: a higher bitrate, a higher resolution, a higher frame rate, lower latency, a higher dynamic range, and / or more feature rich AR overlays (such as overlays comprising 3D objects rather than flat overlays / overlays comprising 2D objects). This embodiment may be utilised when users of AR-enabled devices in the first group have paid more for their seats (or are, by some other means, premium users) and therefore their AR-enabled devices receive (parts of) the determined first information that is of better quality than the (parts of) the determined second information received by AR- enabled devices in the second group.

[0123] In another example, the at least one first characteristic may indicate that AR-enabled devices in the first group are to use a first language for audio-based and / or video-based AR overlays, whereas AR-enabled devices in the second group use a second language for audio-based and / or video-based AR overlays.

[0124] In another example, the at least one first characteristic may indicate that AR-enabled devices in the first group have a higher network priority than AR-enabled devices in the second group. This embodiment may be utilised when network resources are scarce, and groups with the higher network priority are prioritized with regards to generating AR overlays. In some embodiments, the first information may further comprise metadata. Metadata may comprise information indicating a position, translation and / or transformation of objects (e.g., a generative face video) for rendering an AR overlay. In some embodiments, a group or set of clusters receive the same video / image data, but each cluster receives different metadata. For example, the same video / image data may be determined for multiple clusters (according to any of the above embodiments), but respective metadata may be determined for each cluster (e.g., based on a location of a respective cluster). The at least part of the first information received by AR-enabled devices in a cluster may then include the video / image data (which is the same for each cluster) and metadata determined for the cluster.

[0125] Likewise, in some embodiments, AR-enabled devices of a cluster receive the same video / image data, but individual AR-enabled devices receive different metadata. For example, the same video / image data may be determined for a cluster, but different metadata may be determined for each AR-enabled device in the cluster (e.g., based on their position / orientation). The at least part of the first information received by an AR- enabled device in a cluster may then include the video / image data (which is the same for each AR-enabled device in the cluster) and metadata determined for the AR-enabled device.

[0126] For instance, each cluster / AR-enabled device may receive the same video / image for the overlay on a football field, where the video / image may comprise in-game visualizations such as a virtual offside line, virtual ball trajectories, etc. Each cluster / AR-enabled device may also receive different metadata with position and translation information to align the video / image with the football field.

[0127] Figure 10 illustrates a method performed by a first wireless device (e.g., one of the one or more first wireless devices discussed above in relation to Figure 5) for generating an AR overlay on an AR-enabled device. The first wireless device is one of: the AR-enabled device; or an intermediate wireless device connected to the AR-enabled device. It should be appreciated that the method of Figure 10 may comprise any of the steps performed by the one or more first wireless devices discussed above in relation to Figure 5.

[0128] Whilst not illustrated, the method of Figure 10 may begin with: - determining a location of the first wireless device and / or a viewing direction of the AR-enabled device; and / or

[0129] - sending, to a base station, a server, or an intermediate wireless device connected to the first wireless device (if the first wireless device is the AR-enabled device), a location of the first wireless device and / or a viewing direction of the AR-enabled device.

[0130] At step 1002, first information is received for generating the AR overlay, wherein the first information comprises a rendered video bitstream comprising a plurality of sequential frames, each frame being made up of one or more frame segments. The rendered video bitstream may, for example, be the rendered video bitstream discussed above in relation to Figure 5. The first information may be received from at least one of the base station, the server, and the intermediate wireless device connected to the first wireless device, if the first wireless device is the AR-enabled device. Step 1002 may correspond to step 504 of Figure 5.

[0131] At step 1004, at least part of the rendered video bitstream is selected, wherein each frame of the at least part of the rendered video bitstream comprises at least one frame segment corresponding to a viewing direction of the AR-enabled device. Selecting at least part of the rendered video bitstream may comprise decoding the at least part of the rendered video bitstream. If the first wireless device is an intermediate wireless device connected to the AR-enabled device, the method further may comprise sending, to the AR-enabled device, the decoded at least part of the rendered video bitstream.

[0132] The method may further comprise generating the AR overlay using the at least part of the rendered video bitstream (e.g., after it has been decoded).

[0133] If the first wireless device is an intermediate wireless device connected to the AR- enabled device, the method may further comprise sending the AR overlay to the AR- enabled device.

[0134] If the first wireless device is the AR-enabled device, the method further may further comprise displaying the AR overlay.

[0135] Figure 11 is a schematic diagram of an apparatus 1100, such as a server or base station, for facilitating the generation of AR overlays for a first group of AR-enabled devices located in a first area. The first group of AR-enabled devices comprises more than one AR-enabled device. The apparatus comprises memory 1102, processing circuitry 1104, and interface(s) 1106. The processing circuitry 1104 may be configured such that the apparatus 1100 may be operable to perform the methods of the disclosure discussed in relation to Figures 1-9. Apparatus 1100 may be a server or a base station of a network configurable to communicate with an AR-enabled device or an intermediate wireless device connected thereto.

[0136] It will be appreciated that the apparatus 1100 may comprise one or more virtual machines running different software and / or processes. The apparatus 1100 may therefore comprise, or be implemented in or as, one or more servers, switches and / or storage devices and / or may comprise cloud computing infrastructure that runs the software and / or processes.

[0137] The processing circuitry 1104 controls the operation of the apparatus 1100 to implement the relevant part of the methods described herein. The processing circuitry 1104 can comprise one or more processors, processing units, multi-core processors or modules that are configured or programmed to control the apparatus 1100 in the manner described herein. In particular implementations, the processing circuitry 1104 can comprise a plurality of software and / or hardware modules that are each configured to perform, or are for performing, individual or multiple steps of the method described herein in relation to the apparatus 1100.

[0138] The interface(s) / communications interface(s) 1106 is for use in enabling communications with other apparatus, network nodes, computers, servers, etc. For example, the communications interface 1106 can be configured to transmit to and / or receive from other apparatus or nodes requests, acknowledgements, information, data, signals, or similar. The communications interface 1106 can use any suitable communication technology.

[0139] The processing circuitry 1104 may be configured to control the communications interface 1106 to transmit to and / or receive from other nodes, etc. requests, acknowledgements, information, data, signals, or similar, according to the methods described herein.

[0140] Figure 12 is a schematic diagram of a first wireless device 1200 for generating an AR overlay for an AR-enabled device. The first group of AR-enabled devices comprises more than one AR-enabled device. The apparatus comprises memory 1202, processing circuitry 1204, and interface(s) 1206. The processing circuitry 1204 may be configured such that the first wireless device 1200 may be operable to perform the methods embodiments of the disclosure discussed in relation to Figure 10. The first wireless device 1200 may itself be an AR-enabled device, or it may be an intermediate wireless device connected to an AR-enabled device.

[0141] If the first wireless device 1200 is an AR-enabled device, the first wireless device 1200 further comprises a screen via which a scene of interest may be viewed. For example, the screen may be an optical (transparent) screen, such that the scene of interest is viewable through the screen, or a pass through screen, such that it can display a video of the scene of interest as captured by a camera (e.g., a camera of the first wireless device 1200). The screen is configurable to display an AR overlay to enhance the scene of interest as viewed on or through the screen by the user. The AR-enabled device is configurable to communicate directly with a server or base station of a network (such as apparatus 1100) or indirectly with a server or base station of a network via an intermediate wireless device.

[0142] If the first wireless device 1200 is an intermediate wireless device, the intermediate wireless device is configurable to communicate with a server or base station of a network (such as apparatus 1100) and configurable to communicate with an AR-enabled device.

[0143] It will be appreciated that the first wireless device 1200 may comprise one or more virtual machines running different software and / or processes. The first wireless device 1200 may therefore comprise, or be implemented in or as one or more servers, switches and / or storage devices and / or may comprise cloud computing infrastructure that runs the software and / or processes.

[0144] The processing circuitry 1204 controls the operation of the first wireless device 1200 to implement the relevant part of the methods described herein. The processing circuitry 1204 can comprise one or more processors, processing units, multi-core processors or modules that are configured or programmed to control the first wireless device 1200 in the manner described herein. In particular implementations, the processing circuitry 1204 can comprise a plurality of software and / or hardware modules that are each configured to perform, or are for performing, individual or multiple steps of the method described herein in relation to the first wireless device 1200. The interface(s) / communications interface(s) 1206 is for use in enabling communications with other apparatus, network nodes, computers, servers, etc. For example, the communications interface 1206 can be configured to transmit to and / or receive from other apparatus or nodes requests, acknowledgements, information, data, signals, or similar. The communications interface 1206 can use any suitable communication technology.

[0145] The processing circuitry 1204 may be configured to control the communications interface 1206 to transmit to and / or receive from other nodes, etc. requests, acknowledgements, information, data, signals, or similar, according to the methods described herein.

[0146] In some embodiments, the memory 1202 can be configured to store program code that can be executed by the processing circuitry 1204 to perform the methods described herein. Alternatively or in addition, the memory 1202 can be configured to store any requests, acknowledgements, information, data, signals, or similar that are described herein. The processing circuitry 1204 may be configured to control the memory 1202 to store such information therein.

[0147] Figure 13 is a block diagram illustrating a virtualization environment 1300 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments 1300 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a network node, UE, core network node, or host. Further, in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized. In some embodiments, the virtualization environment 1300 includes components defined by the O-RAN Alliance, such as an O- Cloud environment orchestrated by a Service Management and Orchestration Framework via an 0-2 interface. Virtualization may facilitate distributed implementations of a network node, UE, core network node, or host. Applications 1302 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment Q400 to implement some of the features, functions, and / or benefits of some of the embodiments disclosed herein.

[0148] Hardware 1304 includes processing circuitry, memory that stores software and / or instructions executable by hardware processing circuitry, and / or other hardware devices as described herein, such as a network interface, input / output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 1306 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 1308a and 1308b (one or more of which may be generally referred to as VMs 1308), and / or perform any of the functions, features and / or benefits described in relation with some embodiments described herein. The virtualization layer 1306 may present a virtual operating platform that appears like networking hardware to the VMs 1308.

[0149] The VMs 1308 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer 1306. Different embodiments of the instance of a virtual appliance 1302 may be implemented on one or more of VMs 1308, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment.

[0150] In the context of NFV, a VM 1308 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs 1308, and that part of hardware 1304 that executes that VM, be it hardware dedicated to that VM and / or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs 1308 on top of the hardware 1304 and corresponds to the application 1302. Hardware 1304 may be implemented in a standalone network node with generic or specific components. Hardware 1304 may implement some functions via virtualization. Alternatively, hardware 1304 may be part of a larger cluster of hardware (e.g. such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 1310, which, among others, oversees lifecycle management of applications 1302. In some embodiments, hardware 1304 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system 1312 which may alternatively be used for communication between hardware nodes and radio units.

[0151] It should be noted that the above-mentioned examples illustrate rather than limit the disclosure, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended embodiments. The word “comprising” does not exclude the presence of elements or steps other than those listed in an embodiment or claim, “a” or “an” does not exclude a plurality, and a single processor or other unit may fulfil the functions of several units recited in the embodiments. Any reference signs in the claims shall not be construed so as to limit their scope.

[0152] The following numbered statements set out some exemplary and non-limiting embodiments of the present disclosure:

[0153] 1 . A method performed by a server for serving multiple AR devices with AR data, the method comprising: a. clustering at least two AR devices into at least a first and a second cluster, b. sending a first set of AR data to the AR devices in the first cluster, and c. sending a second set of AR data to the AR devices in the second cluster.

[0154] 2. The method of embodiment 1 , further comprising determining the physical location of the at least two AR devices and wherein the clustering is based on the physical location of the at least two AR devices. 3. The method of any of the previous embodiments wherein the first cluster comprises a first intermediate device (e.g., edge node or mobile phone), the server sends the first set of AR data to the first intermediate device, the first intermediate device processes the first set of AR data to produce a first set of processed AR data and the first intermediate device sends the first set of processed AR data to the first AR device in the first cluster.

[0155] 4. The method of embodiment 3, wherein the first intermediate device processing the first set of AR data comprises decoding the first set of AR data and rendering the first set of AR data.

[0156] 5. The method of any of the previous embodiments, further comprising determining a cluster ID for each of the at least first and second cluster.

[0157] 6. The method of any of the previous embodiments, wherein the clustering is different for video AR data and audio AR data.

[0158] 7. The method of any of the previous embodiments, wherein the first set of AR data has different characteristics compared to the second set of AR data.

[0159] 8. The method of embodiment 7, wherein characteristics is at least one of quality, level of AR features, language, or priority.

[0160] 9. A method performed by an AR device for receiving AR data from a server or an intermediate device, the method comprising: a. sending information of the physical location to the server, b. receiving AR data from the server or the intermediate device, based on the physical location, c. determining the orientation of the AR device, d. rendering the AR data based on the orientation of the AR device, and e. displaying the rendered AR data.

[0161] 10. The method of embodiment 9, where sending information of the physical location to the server comprises: a. the intermediate device sending information of its physical location to the server; and receiving the AR data from the server further comprises a. the intermediate device receiving the AR data from the server.

[0162] 11 . The method of embodiment 10, where rendering the AR data further comprises a. receiving transmitted AR data from the intermediate device, and b. rendering the transmitted AR data on the AR device.

[0163] 12. The method of embodiment 10, where rendering the AR data further comprises a. sending the orientation to the intermediate device, b. the intermediate device processing the AR data it received from the server, c. receiving processed AR data from the intermediate device, and d. displaying the processed AR data on the AR device

[0164] 13. The method of any of the previous embodiments wherein an AR device is an HMD (e.g., optical AR glasses or pass-through AR glasses), or a screen device (e.g., a mobile phone, a tablet or a computer).

[0165] 14. The method of any of the previous embodiments wherein AR data comprises video data.

[0166] 15. The method of embodiment 14 wherein the video data is multilayer video data and / or subpicture video data.

[0167] 16. The method of any of the previous embodiments wherein AR data comprises audio data.

[0168] 17. The method of any of the previous embodiments wherein the AR data is comprised in a compressed bitstream.

[0169] 18. A method performed by a server or base station serving multiple AR devices with AR data, the method comprising one or more of the following steps: clustering at least two AR devices into at least a first and a second cluster, wherein the clustering may be different for video AR data and audio AR data; sending a first set of AR data to the AR devices in the first cluster; sending a second set of AR data to the AR devices in the second cluster, wherein the first set of AR data may have different characteristics compared to the second set of AR data, wherein characteristics may include one or more of quality, level of AR features, language and priority;

[0170] - determining the physical location of the at least two AR devices, wherein the clustering may be based on the physical location of the at least two AR devices; sending the first set of AR data to an intermediate device and the intermediate device sends the first set of AR data to an AR device in the first cluster, wherein the intermediate device may be an edge node or a mobile phone, wherein the intermediate device may be used to offload the rendering for the AR device, wherein the intermediate device may process the first set of AR data to produce a first set of processed AR data before sending the first set of processed AR to the AR device in the first cluster, wherein the processing may comprise decoding the first set of AR data and rendering the first set of AR data; determining a cluster ID for the first cluster and / or the second cluster.

[0171] 19. A method performed by an AR device comprising one or more of the following steps:

[0172] - sending information of physical location to a server or an intermediate device (alternatively, an intermediate device sends information about its own physical location to the server); determining the orientation of the AR device;

[0173] - sending the orientation to the intermediate device; receiving AR data from the server or an intermediate device, wherein if the orientation of the AR device was sent to the intermediate device, the AR data may have been rendered remotely on the intermediate device (to offload the AR device) based on the orientation of the AR device before sending it to the AR device; rendering the AR data on the AR device; displaying the AR data on the AR device.

[0174] 20. The method of embodiment 18 or 19, wherein an AR device is an HMD (e.g., optical AR glasses or pass-through AR glasses), or a screen device (e.g., a mobile phone, a tablet or a computer). 21. The method of any of embodiments 18-20, wherein AR data comprises video data, and wherein the video data may be single picture single layer data multilayer video data and / or subpicture video data. 22. The method of any of embodiments 18-21 , wherein AR data comprises audio data.

[0175] 23. The method of any of embodiments 18-22, wherein AR data is comprised in a compressed bitstream.

Claims

CLAIMS1. A computer-implemented method for facilitating the generation of augmented reality, AR, overlays on a first group of AR-enabled devices located in a first area, the first group of AR-enabled devices comprising more than one AR-enabled device, the method comprising: determining (502), for the first group of AR-enabled devices, first information for generating an AR overlay, wherein the first information is determined based on a location of the first group of AR-enabled devices; and sending (504), to one or more first wireless devices, at least part of the first information, wherein each of the first wireless devices is one of: a respective AR device in the first group of AR-enabled devices; or an intermediate wireless device connected to one or more of the first group of AR-enabled devices.

2. The method of claim 1 , wherein the location is based on: an estimated centre of the first area; and / or a location of an intermediate wireless device connected to one or more of the AR- enabled devices of the first group of AR-enabled devices.

3. The method of claim 1 or 2, wherein the first group of AR-enabled devices is determined by assigning an AR-enabled device to the first group based on any one of: a proximity of the AR-enabled device to an AR-enabled device already assigned to the first group; the AR-enabled device being located within a geographical area associated with the first group; and a connection of the AR-enabled device to an intermediate wireless device associated with the first group.

4. The method of claim 3, further comprising obtaining an indication of a location of the AR-enabled device.

5. The method of claim 4, wherein the indication of the location of the AR-enabled device is obtained from: the AR-enabled device; or an intermediate wireless device to which the AR-enabled device is connected.

6. The method of any of claims 1-5, wherein the first information comprises: information for rendering a video stream; or a rendered video bitstream.

7. The method of claim 6, wherein the rendered video bitstream is encoded using a video encoder.

8. The method of claim 6 or 7, wherein the rendered video bitstream comprises a plurality of sequential frames, each frame being made up of one or more frame segments for selection to be displayed by an AR-enabled device based on a viewing direction of the AR-enabled device.

9. The method of claim 8, further comprising: determining a viewing direction of each of the AR-enabled devices of the first group of AR-enabled devices relative to a scene of interest.

10. The method of claim 9, wherein the at least part of the first information sent to one of the one or more first wireless devices is based on: a viewing direction of the first wireless device, if the first wireless device is an AR- enabled device, or a viewing direction of an AR-enabled device connected to the first wireless device, if the first wireless device is an intermediate wireless device.

11. The method of claim 9, further comprising grouping, based on the determined viewing directions, the first group of AR-enabled devices into a plurality of first subgroups of AR-enabled devices.

12. The method of claim 11 , wherein sending the at least part of the first information to the one or more first wireless devices comprises: sending at least part of the rendered video bitstream to one or more first wireless devices associated with a respective sub-group, wherein each frame of the at least part of the rendered video bitstream comprises at least one frame segment corresponding to a viewing direction associated with the respective sub-group; or sending at least part of the information for rendering the video stream to one or more first wireless devices associated with a respective sub-group, wherein the at leastpart of the information for rendering the video stream corresponds to a viewing direction associated with the respective sub-group.

13. The method of any of claims 6-12, wherein sending the at least part of the first information to the one or more first wireless devices comprises: sending the rendered video bitstream to the one or more first wireless devices.

14. The method of claims 6-13, wherein the rendered video bitstream provides an AR overlay for a scene of interest, and a part of the rendered video bitstream is selectable by a first wireless device of the one or more first wireless devices for display.

15. The method of claim 14, wherein the rendered video bitstream is a layered video bitstream comprising a base layer and at least one enhancement layer, and displaying the part of the rendered video bitstream comprises: displaying the base layer and; a part of the at least one enhancement layer corresponding to the part of the rendered video bitstream.

16. The method of claim 15, wherein the part of the at least one enhancement layer comprises at least one of: at least one independently coded subpicture; at least one slice; and at least one tile.

17. The method of claim 16, wherein the at least one enhancement layer has a higher frame quality than the base layer.

18. The method of claim 17, wherein the higher frame quality comprises at least one of: a larger spatial resolution; a higher temporal resolution; a higher bit depth; a higher SNR quality; and a closer resemblance to a pre-encoded version of the rendered video bitstream.

19. The method of any of claims 1-18, wherein the first information comprises: information for rendering an audio stream; or a rendered audio bitstream.

20. The method of claim 19, further comprising: determining a viewing direction of each of the AR-enabled devices of the first group of AR-enabled devices relative to a scene of interest; andgrouping, based on the determined viewing directions, the first group of AR- enabled devices into a plurality of first sub-groups of AR-enabled devices.

21. The method of claim 20, wherein sending the at least part of the first information to the one or more first wireless devices comprises: sending at least part of the rendered audio bitstream to one or more first wireless devices associated with a respective sub-group, wherein the part of the rendered audio bitstream corresponds to a viewing direction associated with the respective sub-group; or sending at least part of the information for rendering an audio stream to one or more first wireless devices associated with a respective sub-group, wherein the at least part of the information for rendering the audio stream corresponds to a viewing direction associated with the respective sub-group.

22. The method of any of claims 1-21 , the method further facilitating the generation of AR overlays on a second group of AR-enabled devices located in a second area, the second group of AR-enabled devices comprising more than one AR-enabled device, where at least one of the AR-enabled devices of the second group is not included in the first group, the method further comprising: determining, for the second group of AR-enabled devices, second information for generating an AR overlay, wherein the second information is different from the first information and is determined based on a location of the second group of AR-enabled devices; and sending, to one or more second wireless devices, at least part of the second information, wherein each of the second wireless devices is one of: a respective AR device in the second group of AR-enabled devices; or an intermediate wireless device connected to one or more of the second group of AR-enabled devices.

23. The method of claim 22, wherein the first information is further determined based on at least one first characteristic associated with the first group, and / or the second information is further determined based on at least one second characteristic associated with the second group.

24. The method of claim 23, wherein the at least one first characteristic belongs to any one or more of the following characteristic categories: a type of AR overlay, a frame quality, a level of AR features, a language, and priority, and / or the at least one secondcharacteristic belongs to any one or more of the following characteristic categories: a type of AR overlay, a frame quality, a level of AR features, a language, and priority.

25. The method of any of claims 1-24, wherein an AR-enabled device is at least one of: a head mounted display, HMD; optical AR glasses; pass-through AR glasses; a screen device; a mobile phone; a tablet; and a computer.

26. The method of any of claims 1 -25, wherein the method is performed by a server or a base station.

27. A method performed by a first wireless device for generating an augmented reality, AR, overlay on an AR-enabled device, the method comprising: receiving (1002) first information for generating the AR overlay, wherein the first information comprises a rendered video bitstream comprising a plurality of sequential frames, each frame being made up of one or more frame segments; and selecting (1004) at least part of the rendered video bitstream, wherein each frame of the at least part of the rendered video bitstream comprises at least one frame segment corresponding to a viewing direction of the AR-enabled device; wherein the first wireless device is one of: the AR-enabled device; or an intermediate wireless device connected to the AR-enabled device.

28. The method of claim 27, further comprising determining a location of the first wireless device and / or a viewing direction of the AR-enabled device.

29. The method of claim 27 or 28, wherein the first information is received from at least one of: a base station; a server; and an intermediate device connected to the first wireless device, if the first wireless device is the AR-enabled device.

30. The method of claim 29, further comprising sending, to the server, base station and / or intermediate wireless device from which the first information is received, a location of the first wireless device and / or a viewing direction of the AR-enabled device.

31. The method of any of claims 27-30, wherein selecting the at least part of the rendered video bitstream comprises decoding the at least part of the rendered video bitstream.

32. The method of claim 31 , wherein if the first wireless device is an intermediate wireless device connected to the AR-enabled device, the method further comprises sending, to the AR-enabled device, the decoded at least part of the rendered video bitstream.

33. The method of any of claims 27-31 , the method further comprises generating the AR overlay using the at least part of the rendered video bitstream.

34. The method of claim 33, wherein if the first wireless device is an intermediate wireless device connected to the AR-enabled device, the method further comprises sending the AR overlay to the AR-enabled device.

35. The method of claim 33 or 34, wherein if the first wireless device is the AR-enabled device, the method further comprises displaying the AR overlay.

36. An apparatus (1100) for facilitating the generation of augmented reality, AR, overlays on a first group of AR-enabled devices located in a first area, the first group of AR-enabled devices comprising more than one AR-enabled device, the apparatus comprising processing circuitry (1104) configured to cause the apparatus to perform any of claims 1-26.

37. The apparatus of claim 35, wherein the apparatus is a base station or a server.

38. A first wireless device (1200) for generating an augmented reality, AR, overlay on an AR-enabled device, wherein the first wireless device is one of: the AR-enabled device; or an intermediate wireless device connected to the AR-enabled device, the first wireless device comprising processing circuitry (1204) configured to cause the apparatus to perform any of claims 27-35.

39. A computer program product comprising a computer readable medium having computer readable code embodied therein, the computer readable code beingconfigured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform the method of any of claims 1-35.

Images