Output of virtual content

Abstract

Embodiments of the present disclosure relate to the output of virtual content. This specification describes an apparatus, method, and computer program related to virtual reality, particularly augmented reality (AR) or mixed reality (MR). The method may include: providing different first and second sets of virtual content based on a position associated with a display component for overlapping display at a display component at a first time; and determining that the first set of virtual content takes precedence over the second set of virtual content. Based on the determination, the method may include giving priority to the display of the first set of virtual content over the display of the second set of virtual content at the display component, and enabling the display of the second set of virtual content at a subsequent second time.

Description

Technical Field

[0001] Example embodiments relate to the field of outputting virtual content, such as display of augmented reality (AR) or mixed reality (MR) content. Example embodiments also relate to associated methods, computer programs and apparatus. Background Art

[0002] The term extended reality (XR) is sometimes used to refer to a range of technologies and methods involving a combination of real and virtual environments. Common examples are virtual reality (VR), augmented reality (AR), and mixed reality (MR). VR can refer to the rendering of a virtual scene in terms of video and / or audio content through a VR headset or handheld device, where the virtual scene can be updated based on user movement. AR is similar, but involves the output of overlapping virtual content to enhance the view of the real world seen through glasses, goggles, or a camera of a handheld device. Thus, users of an AR display component can be able to view the real-world environment around them, which is enhanced or supplemented with content that can be provided based on their location. Virtual content can include multimedia content such as pictures, photos, videos, charts, textual information, and auditory content. An example use case is navigation instructions. MR is similar to AR, but can be considered more advanced because some content is inserted into the real scene at anchor points to give the illusion that the content is part of the real scene. Summary of the invention

[0003] The scope of protection sought by various embodiments of the present invention is set out by the independent claims. Embodiments and features described in this specification that do not fall within the scope of the independent claims, if any, are to be construed as examples useful for understanding various embodiments of the present invention.

[0004] According to a first aspect, the present specification describes a device comprising components for: providing different first and second groups of virtual content based on a position associated with a display component for overlapping display at the display component at a first time; determining that the first group of virtual content takes precedence over the second group of virtual content; based on the determination: enabling display of the first group of virtual content to take precedence over display of the second group of virtual content at the display component; and enabling display of the second group of virtual content at a subsequent second time.

[0005] The component may be configured to prioritize display by causing display of the first set of virtual content rather than the second set of virtual content at the display component.

[0006] The component may be configured to determine that the first set of virtual content takes precedence over the second set of virtual content if the first set of virtual content was displayed before the first time (and, for example, if the second set of virtual content was not displayed before the first time).

[0007] The component may be configured to determine that the first set of virtual content takes precedence over the second set of virtual content if the first set of virtual content and the second set of virtual content would conflict if both were displayed at a first time.

[0008] The component may be configured to identify that the first set of virtual content and the second set of virtual content will conflict if respective positions of the first set of virtual content and the second set of virtual content will be within a predetermined distance of each other or will overlap.

[0009] The component may be configured to automatically enable display of the second set of virtual content in response to the first set of virtual content no longer being displayed.

[0010] The component may be configured to enable display of the second set of virtual content in response to receiving a user selection of the second set of virtual content.

[0011] The component may also be configured to cause the capture of a real-world image at the first time or close to the first time in time (e.g., within a predetermined time period) by a camera associated with the display component, and to associate at least a segment of the captured image with the second virtual content to display the captured image segment with the second virtual content at the second time. The component may also be configured to provide metadata associated with the image segment, which metadata defines how the captured image segment will be displayed when displayed at the display component. After the first time, the component may also be configured to determine a similarity between the captured image segment and a portion of a current or most recent (e.g., substantially current, such as within a defined time period) real-world image captured by the camera, and cause the display of the captured image segment at the display component in response to determining the predetermined similarity level. The component may be configured to cause the display of the captured image segment so that the captured image segment overlaps a similar portion of the real-world image. The component may be configured to identify that a similar portion of a current or recent real-world image is outside a predetermined display area of ​​the display component, and in response to the identification, provide one or more prompts via the display component, the one or more prompts indicating the camera movement required to move the similar portion of the current or recent real-world image toward the predetermined display area. One or more prompts may change appearance as the camera moves to indicate progress toward the predetermined display area. The component may be configured to provide a modified form of the second set of virtual content and / or the captured image segment as a prompt. The modified form may include a stretched form that is not stretched as the camera moves to indicate progress toward the predetermined display area. The modified form may include a partial view displayed simultaneously with the first set of virtual content, the size of the partial view increasing as the camera moves to indicate progress toward the predetermined display area while reducing the size of the first set of virtual content. The component may also be configured to indicate the availability of the second set of virtual content for subsequent display at the display component. The component may be configured to provide a plurality of different second sets of virtual content, and display an indicator at the display component, the indicator including an indication of the number of second sets of virtual content that can be subsequently displayed. The component may be configured to display a gallery view of a second, different set of virtual content in response to a user selection associated with the indicator, and to receive a selection of the second set for display via the gallery view.

[0012] According to a second aspect, the present specification describes a method, comprising: providing different first and second groups of virtual content based on a position associated with a display component for overlapping display at the display component at a first time; determining that the first group of virtual content takes precedence over the second group of virtual content; based on the determination: enabling display of the first group of virtual content to take precedence over display of the second group of virtual content at the display component; and enabling display of the second group of virtual content at a subsequent second time.

[0013] The method may prioritize display by causing display of the first set of virtual content rather than the second set of virtual content at the display component.

[0014] If the first set of virtual content is displayed before the first time, the method may determine that the first set of virtual content takes precedence over the second set of virtual content.

[0015] The method may determine that the first set of virtual content takes precedence over the second set of virtual content if the first set of virtual content and the second set of virtual content would conflict if both were displayed at a first time.

[0016] The method may identify that the first set of virtual content and the second set of virtual content will conflict if respective locations of the first set of virtual content and the second set of virtual content will be within a predetermined distance of each other or will overlap.

[0017] The method may automatically enable display of the second set of virtual content in response to the first set of virtual content no longer being displayed.

[0018] The method may enable display of the second set of virtual content in response to receiving a user selection of the second set of virtual content.

[0019] The method may also include causing, by a camera associated with the display component, the capture of a real-world image at or near the first time in time, and associating at least a segment of the captured image with the second virtual content to display the captured image segment with the second virtual content at the second time. The method may provide metadata associated with the image segment, the metadata defining how the captured image segment will be displayed when displayed at the display component. After the first time, the method may determine a similarity between the captured image segment and a portion of a current or most recent real-world image captured by the camera, and cause the display of the captured image segment at the display component in response to determining a predetermined similarity level. The method may cause the display of the captured image segment so that the captured image segment overlaps the similar portion of the real-world image. The method may identify that the similar portion of the current or most recent real-world image is outside a predetermined display area of ​​the display component, and in response to the identification, provide one or more prompts via the display component, the one or more prompts indicating the camera movement required to move the similar portion of the current or most recent real-world image toward the predetermined display area. The one or more prompts may change appearance as the camera moves to indicate progress toward the predetermined display area. The method may provide a modified form of the second set of virtual content and / or the captured image fragment as a prompt. The modified form may include a stretched form that is not stretched as the camera moves to indicate progress toward a predetermined display area. The modified form may include a partial view displayed simultaneously with the first set of virtual content, the size of which increases as the camera moves to indicate progress toward a predetermined display area, while reducing the size of the first set of virtual content. The method may indicate the availability of the second set of virtual content for subsequent display at the display component. The method may provide a plurality of different second sets of virtual content, and display an indicator at the display component, the indicator including an indication of the number of second sets of virtual content that can be subsequently displayed. The method may display a gallery view of different second sets of virtual content in response to a user selection associated with the indicator, and receive a selection of the second group for display via the gallery view.

[0020] According to a third aspect, the present specification describes a computer-readable medium including a computer program code stored thereon, the computer-readable medium and the computer program code being configured to execute a method when run on at least one processor, the method comprising: providing different first and second groups of virtual content based on a position associated with a display component for overlapping display at the display component at a first time; determining that the first group of virtual content takes precedence over the second group of virtual content; based on the determination: enabling display of the first group of virtual content to take precedence over display of the second group of virtual content at the display component; and enabling display of the second group of virtual content at a subsequent second time.

[0021] According to a fourth aspect, the present specification describes a computer readable medium including computer program code stored thereon, the computer readable medium and the computer program code being configured to perform a method of any operation related to the second aspect when executed on at least one processor.

[0022] According to a fifth aspect, the present specification describes a device comprising: at least one processor; and at least one memory comprising computer program code, which, when executed by the at least one processor, causes the device to: provide different first and second groups of virtual content based on a position associated with a display component for overlapping display at the display component at a first time; determine that the first group of virtual content takes precedence over the second group of virtual content; based on the determination: prioritize the display of the first group of virtual content over the display of the second group of virtual content at the display component; and enable the display of the second group of virtual content at a subsequent second time.

[0023] According to a sixth aspect, the present specification describes an apparatus comprising: at least one processor; and at least one memory comprising computer program code, which, when executed by the at least one processor, causes the apparatus to perform any operations related to the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Example embodiments will now be described, by way of example only, with reference to the accompanying drawings.

[0025] Figure 1 is a schematic diagram of an example AR system that can represent a user terminal device;

[0026] Figure 2 is a block diagram illustrating functional components of an example AR system, the AR system including a content provider and a network;

[0027] Figure 3 is a perspective view of an augmented reality scene that may be useful in understanding some example embodiments;

[0028] Figure 4a-4c It is an instruction Figure 1 A schematic diagram of a user interface of a display component;

[0029] Figure 5 is a flow chart illustrating processing operations according to some example embodiments;

[0030] Figure 6a-6d is a schematic diagram of a user interface of a display component indicating operation according to some example embodiments;

[0031] Figure 7a-7cis a schematic diagram of a user interface of a display component indicating operation according to some example embodiments;

[0032] Figure 8 is a schematic diagram of an augmented reality scenario that may be useful in understanding some example embodiments;

[0033] Figure 9a-9c is a schematic diagram of a user interface of a display component indicating operation according to some example embodiments;

[0034] Figure 10a-10c is a schematic diagram of a user interface of a display component indicating operation according to some example embodiments;

[0035] Figures 11a-11c is a schematic diagram of a user interface of a display component indicating operation according to some example embodiments;

[0036] Fig.12 is a schematic diagram illustrating functional components of an apparatus according to some example embodiments; and

[0037] Fig.13 is a plan view of a non-transitory medium for storing computer readable code for executing some example embodiments. DETAILED DESCRIPTION

[0038] Example embodiments relate to methods, computer programs, and devices related to extended reality (XR), such as augmented reality (AR) and / or mixed reality (MR).

[0039] For ease of explanation, the term augmented reality (AR) will be used hereinafter, but it should be understood that AR is also intended to cover MR and any related technologies involving the output of virtual content overlaid onto a portion of a real-world scene.

[0040] For example, virtual content may include multimedia content, such as pictures, photos, videos, charts, textual information, and auditory content. Virtual content may be represented in any suitable data format. Example embodiments focus on visual content rather than auditory content, but do not exclude the latter. Virtual content may be generated from a captured scene, whether the entire scene or one or more segments of the scene, or may be generated by a designer using an appropriate content creation tool.

[0041] As described above, AR can use an AR display component, such as glasses or goggles with one or more transparent lenses, into (or onto) which one or more images can be projected. Another form of AR display component can be a portable computing device, such as a mobile phone or tablet with a camera that is used to capture a viewport of the real world for display in a lens manner. An AR device that may or may not include an AR display component can provide a presentation of virtual content superimposed on a user's view of the real world, whether it is a view seen through a transparent lens or through a camera. Thus, users of AR can be able to view the real-world environment around them, which is enhanced or supplemented with content provided by a content provider. AR content can also include auditory content.

[0042] Figure 1 is a schematic diagram of an example AR system 1 representing a user-end device. The AR system 1 may include a display component in the form of AR glasses 20, which have one or more perspective (i.e., transparent) lenses for displaying received video content by overlaying (e.g., projecting) the received video content onto the one or more lenses using known techniques. The term "lens" does not necessarily indicate any optical correction. The AR glasses 20 may be of any suitable type. As previously described, a mobile phone or tablet may provide the same functionality using a camera on one side as a lens to provide a real-time image on a display on the other side.

[0043] The AR glasses 20 can receive data representing virtual content from the media player 10. In the broadest sense, the media player 10 is a processing component for handling and processing data. The media player 10 can be part of a separate device connected to the AR glasses 20 via a wired or wireless connection. For example, the media player 10 may include a game console, a PC, a laptop, a digital assistant, a mobile phone, or a tablet computer. The media player 10 can communicate with the AR glasses 20 using any wired or wireless communication method, such as using WiFi, Bluetooth, or similar technology.

[0044] Alternatively, the media player 10 may form part of the AR glasses 20 .

[0045] Alternatively, the media player 10 may not be part of the user-end device and may be an edge or cloud-based system that provides virtual content directly to the AR glasses 20 over a data network, such as by using cellular communications, for example, using 3G, LTE, 4G, 5G or any similar future communications technology.

[0046] The AR system 1 may also include, or be associated with, one or more cameras for capturing static and / or moving images in use. For example, if the AR system 1 includes a mobile phone or tablet as a display component, one or more cameras may be provided as a lens into the real world. In the case of AR glasses 20, one or more cameras may be mounted on the AR glasses 20, for example, at a position indicated by reference numeral 28. Images captured by the one or more cameras 28 may be processed by the AR glasses 20 and / or may be transmitted to the media player 10 for processing.

[0047] The AR system 1 may also include components for determining the user's location, for example by determining the location of the AR glasses 20 or a mobile phone or tablet. The term "location" may refer to a geographic location that may use global positioning technologies such as GNSS (e.g., GPS), positioning / triangulation via WiFi or cellular base station signals, but may also refer to a location relative to a reference point and / or, for example, an orientation of the user's head, which indicates a field of view (FOV). In addition to determining the user's location in space, motion measurements over continuous time frames may also be calculated and stored. Such a device may include a portion of the media player 10 and / or a portion of the AR glasses 20 or a mobile phone or tablet. For example, the AR glasses 20 or a mobile phone or tablet may incorporate a GNSS receiver or may even carry a radio positioning tag that can be detected by one or more high-precision radio positioning readers. The AR glasses 20, mobile phone or tablet may include one or more motion tracking sensors, which may include one or more of a gyroscope, an accelerometer, and a structured light system. These sensors may generate location data, from which the current visual FOV may be determined, and the location data is updated with the user, so that the AR glasses 20, mobile phone or tablet changes position and / or orientation.

[0048] The AR glasses 20 may include two see-through lenses that may incorporate corresponding digital displays, such as LED or OLED displays. Alternatively, the digital display may be projected onto the surface of the lens at an appropriate location (e.g., directly or via a prism).

[0049] In some embodiments, the AR system 1 may use six degrees of freedom (6DoF) to determine the position and / or orientation of the user's head. Figure 1 As shown, with respect to the illustrated AR glasses 20, these may include measurements of pitch 22, roll 23, and yaw 24, as well as translational motion in Euclidean space along left-right, front-back, and up-down axes 25, 26, 27. In this way, determining the user's FOV is relatively straightforward.

[0050] In use, a user wearing AR glasses 20 or carrying a mobile phone or tablet can move within a real-world space. Depending on their location, virtual content can be displayed as one or more overlays onto the relevant display components (i.e., lenses or display screens) to enhance the real-world view based on the aforementioned location. For example, if the location corresponds to the location of a particular monument or landmark, the media player 10 can provide virtual content related to that location (and therefore the monument or landmark). The virtual content may already be stored on the media player 10, or may be retrieved by sending the location information to a remote source of the virtual content (i.e., a remote content provider) and then receiving and displaying the relevant virtual content.

[0051] refer to Figure 2 , the remote content provider 30 can store and transmit virtual content as discrete files or as packetized data streams or continuous data streams, which in the context of the example embodiment are data representing virtual content for AR overlay using a display component (e.g., AR glasses 20). In response to receiving or downloading a request sent by the media player 10, the content provider 30 sends the virtual content over a data network 40, which can be any network, such as an IP network (such as the Internet) or a cellular network.

[0052] A storage device 29 (which may be any form of storage device (solid-state, magnetic or optical)) may be provided for access by the media player 10 or the AR display component. The purpose of the storage device 29 is explained later. In summary, the storage device 29 provides a way to collect AR virtual content and possibly captured real-world images that would be seen by the user at a given time / location, but are not displayed due to the prioritization methods described herein. The storage device 29 may be part of the media player 10 or may be an edge or cloud-based device accessed via a network.

[0053] In the example use case, Figure 3 Two similar scenes 50a, 50b are shown in perspective. In the first scene 50a, Figure 1 The user of the AR glasses 20 is shown interacting with the real world including a tree object 52. In the second scene 50b, the user of the mobile phone 54 positions the mobile phone in such a way that the rear camera of the mobile phone is pointed at the tree object 52 with a display screen, which acts as a lens equivalent together with the camera. The user may or may not wear headphones or earphones / earbuds to obtain relevant audio data.

[0054] Figure 4a-4c Yes, users can use Figure 1Schematic diagram of what is seen by the user interface 60 of the AR system 1 via the display component. For ease of illustration, the user interface 60 is the user interface shown in the mobile phone 54, but the same or similar user interface is also applicable to the AR glasses 20, although it may be a stereoscopic or volumetric representation. For example, these examples may involve using more than one AR service at the same time or consuming more than one AR content from one or more sources. For example, a user can utilize a specific AR service that can be location-based or context-based and corresponds to a wide area with suitable adaptation of content based on user movement (e.g., navigation) and a highly location-based AR service with limited applicability in other physical locations (e.g., local information services).

[0055] refer to Figure 4a , the media player 10 provides a first set of virtual content 60a, 60b in the form of navigation instructions, which may be part of a navigation application that guides the user to a particular destination. The user's current location determines the navigation instructions, i.e., what to do next. In this case, the first set of virtual content 60a, 60b includes two portions of related content that are intended to be displayed together at a given time and will progress as the user changes location. However, a set of virtual content may include only one or more than two portions of related content. Figure 4b A different second set of virtual content 60c, 60d is also shown in the form of text and image information associated with the tree object 52 based on the same or similar location of the mobile phone 54 and possibly based on the user orientation. Figure 4c 6 shows how the first and second sets of virtual content 60a-60d may conflict when they are simultaneously displayed on the user interface 60. There is at least some overlap, but even proximity (e.g., within some predetermined distance) and the number of different virtual content portions may result in conflicts that may overwhelm, distract, or confuse a user.

[0056] Therefore, example embodiments are directed to avoiding or reducing this problem by prioritizing one set of virtual content over another set of virtual content, while collecting lost content for later reuse or "resurrection."

[0057] Figure 5 1 is a flow chart indicating processing operations according to an example embodiment. The operations may be performed in hardware, software, firmware, or a combination thereof. A greater or lesser number of operations may be involved. The operations may be performed by, for example, a media player 10.

[0058] The first operation 5.1 may include providing different first and second sets of virtual content for overlapping display at the display component at a first time based on a location associated with the display component. The location may be a location of a user, media player 10, and / or display component (if different from the media player). The location may be a geographic location.

[0059] The second operation 5.2 may include determining that the first set of virtual content takes precedence over the second set of virtual content.

[0060] The third operation 5.3 may include: based on the determination, giving priority to display of the first set of virtual contents over display of the second set of virtual contents at the display component, and enabling display of the second set of virtual contents at a subsequent second time.

[0061] The term "providing" may mean "receiving", for example, from another source.

[0062] In some embodiments, prioritizing display may involve displaying a first group of virtual content at a display component instead of a second group of virtual content. Determining that the first group of virtual content takes precedence over the second group may be based on, for example, that the first group has been displayed when the second group is provided. For example, assuming that the navigation instructions include a first group of virtual content, if the above instructions have been overlapped when the second group is provided, the first group may be prioritized and the second group is not displayed but saved or collected for later display. Other rules for determining priorities may be predetermined or even learned over time using, for example, machine learning or artificial intelligence methods. Examples may include rules that prioritize the most relevant virtual content over less relevant virtual content, or prioritize the content that occupies the least or most space on the user interface over the opposite situation, or rules implemented by understanding what type of virtual content the user selected over other types in a previous time period.

[0063] In some embodiments, if both the first set of virtual content and the second set of virtual content are displayed simultaneously, a prioritization determination may be performed in response to detecting a conflict situation. This may occur if their respective locations will be within a predetermined distance of each other or will overlap on the user interface, and / or if more than a predetermined number of virtual object portions will occupy the user interface, and / or if greater than a predetermined percentage of the user interface will be occupied by virtual objects.

[0064] In some embodiments, a first group may take precedence over multiple second groups of virtual content. For example, assuming that navigation instructions include a first group of virtual content, when the user follows the instructions, multiple other groups of virtual content may be provided due to the user's changing position. These multiple second groups may be collected (e.g., stored) for later viewing. Figure 2The storage component 29 shown is stored. In fact, for a specific user or device identifier, a collection of one or more groups of other virtual content is stored locally or at an edge or cloud device for later playback or reproduction. This later reproduction can be initiated automatically, for example, when the display component stops displaying the first virtual content. For example, when the user arrives at their destination, the navigation instructions of the first virtual content can stop and this can trigger one or more of the collected second virtual content to be automatically displayed. These can be displayed in a specific queue order, for example, first in first out or last in first out, based on the location closest to the current location, or based on some other preferences that can be defined by user settings.

[0065] In some embodiments, the user himself can be informed of the availability of the collected second virtual content through some visual, auditory and / or tactile indicators, and can cause the display of the first virtual content to stop at any time through user input (e.g., touch or voice command) to enable access and / or display of one or more second virtual contents. In one example, a gallery view of the second virtual content can be presented to the user interface for user selection.

[0066] exist Figure 5 In the fourth optional operation 5.4 indicated in , an additional operation 5.4 may be performed, which causes the capture of a real-world image at the above-mentioned first time or close to the above-mentioned first time in time by a camera associated with the display component, and associates at least a segment of the captured image with the second virtual content to display the above-mentioned captured image segment with the second virtual content at the above-mentioned second time. Operation 5.4 may be performed in response to the second or third operation 5.2, 5.3. In this way, the relevant real-world context is captured to be displayed with the second virtual content, making the enhancement more appropriate and information-rich. The captured image segment may include a subset of pixels of the captured image, such as related to a specific object or multiple objects, which is different from the entire image. The captured image or image segment may be stored locally or at an edge or cloud device together with a collection of one or more other virtual contents for later playback or reproduction. Metadata associated with the above-mentioned image or image segment may also be generated and stored, for example, to reflect the specific time and / or location of the capture and / or define how the captured image segment will be displayed when the captured image segment is displayed at the display component.

[0067] Example embodiments will now be described in more detail with reference to user interface scenarios.

[0068] Figure 6a 1 shows a user interface 70 according to some example embodiments, which may be displayed by a display component (e.g., AR glasses 20) associated with a device (e.g., media player 10). The user interface 70 is similar to Figure 4aThe user interface shown, but in this case, other AR content with a lower priority than the current virtual content 60a, 60b is indicated by a visual indicator 72 that has been saved for later playback. The visual indicator 72 can be located anywhere in the user interface 70 and can take any form. When it appears and / or updates, it can be accompanied by an auditory and / or tactile signal. It can also indicate the number of groups of virtual content saved in the user collection and stored in the storage device 29. In the current case, as shown by the "1" number, one group of virtual content is saved.

[0069] Figure 6b An example user interface 74 is shown resulting from a user selection to replay saved virtual content. The user interface 74 includes a diagram showing Figure 4b A gallery view of a set of thumbnails 76 of virtual content 60c, 60d is shown. The virtual content 60c, 60d is saved at a first time based on the user's location being close to the real-world tree object 52 and the virtual content having a lower priority than the currently displayed virtual content. The thumbnail 76 can be selected at any time after the first time, such as by touch or voice input. In other embodiments, there may be multiple thumbnails 77 representing other replayable virtual content, as indicated by dotted lines. Associated with each thumbnail 76, 77 may be metadata 78 that describes relevant information such as the save / capture time / location. By selecting the thumbnail 76 at a second time, the user can cause its associated virtual content to be replayed.

[0070] For example, refer to Figure 6c , selecting a thumbnail 76 may cause the associated virtual content 60c, 60d to be displayed on the user interface 70. Note that the virtual content 60c, 60d may be overlaid onto the current real-world scene 79, in which case the real-world scene 79 is different from the scene at the time of capture because the user has changed position. However, the user is able to replay the virtual content 60c, 60d while avoiding screen conflicts.

[0071] As previously mentioned, improved context may be provided by saving a snapshot of the real world scene as seen at or near the time when the virtual content is saved for playback. The snapshot may include one or more images or image fragments, and may even capture a short period of video and / or audio.

[0072] For example, including Figure 3 The scene of the tree object 52 in the video may have been captured at the above time, and the tree object 52 is saved as an image segment so as to overlap with the saved virtual content 60c, 60d during playback. Figure 6d, where an image segment 80 representing a tree object 52 is shown overlapping the current scene with virtual content 60c, 60d, which further improves the environment.

[0073] In some example embodiments, the user may wish to replay or reproduce the saved virtual content based on their current real-world environment. For example, the similarity between the captured image segment (such as the captured and segmented tree object 80) and a portion of the current or most recent real-world image captured by the camera may be used. In response to determining a predetermined level of similarity with, for example, a tree currently or most recently in view, the media player 10 may cause the captured image segment, such as the image segment 80, to be displayed at the display component.

[0074] Determining the similarity may use any suitable and known image processing algorithm. Known examples include, for example, using sum-of-squares distance, Mahalanobis distance, Hamming distance, cross-correlation, convolutional neural networks, etc. The predetermined distance threshold may be predefined and possibly modified using software user settings to set the desired similarity.

[0075] In this case, if Figure 7a-7c As shown, in response to identifying object 84 in the user's FOV, media player 10 may determine that it has a predetermined similarity (e.g., greater than 70%) with image segment 80 and therefore determine to overlap image segment 80 at the location of user interface 70' corresponding to object 84 ( Figure 7b ). If the user moves their FOV, the position of the image segment 80 can be tracked accordingly to remain superimposed on the object 84. Figure 7c As shown, the overlap of related virtual contents 60c, 60d completes the enhancement.

[0076] In some example embodiments, we consider the situation where a similar portion of the current or most recent real-world image (e.g., object 84) is outside the predetermined display area of ​​the display component. This may include object 84 being completely or partially outside the display or capture area of ​​the display component. In response to this, the media player 10 may provide one or more prompts via the display component that indicate the camera movement required to capture the content or the above-mentioned similar portion of the most recent real-world image. The prompt may be visual, auditory, and / or tactile. The prompt may indicate the direction and / or degree of movement in the above-mentioned direction. In this way, the user may be guided to make one or more additional inputs, such as a change in position, to cause the media player 10 to replay or reproduce the virtual content. For example, one or more prompts may change appearance as the camera moves to indicate progress in the desired direction.

[0077] Figure 8It shows a situation where the user rotates to the right so that the previously captured image of the object 92 is no longer in the capture area. Figure 9a-9c A progression of example visual cues indicating in real time how a user should move to trigger playback of stored location-related virtual content is shown. Figure 9a , a portion of image segment 80' is shown on the left side of user interface 70" to indicate that movement in the left-hand direction will trigger the display of some virtual content. Image segment 80' is shown as being stretched or otherwise distorted in some manner to indicate the amount of movement required. Note that in this case, a modified form of image segment 80' is used, but it may also or alternatively include a modified form of the associated virtual content itself. For example, Figure 9b As shown, as the user begins to move their mobile phone 54 to the left, more of the image segment 80' appears in the user interface 70", but with less distortion. In addition, at least some virtual content 60c', 60d' begins to appear. Fig.9c 8 shows when the correct amount of movement is achieved, whereby the captured image (image segment 80) is shown overlapping the virtual content 60c, 60d, as shown in FIG. Figure 7c The unstretched indication of the image segment 80' as it moves along the mobile phone 54 progresses toward the predetermined display area, thereby guiding the user with continuous feedback. Where at least some virtual content 60c', 60d' is also shown, this may be extended in terms of content volume (e.g., volume of words) or size.

[0078] In some example embodiments, prioritization may allow two sets of virtual content to appear simultaneously, but with one set of virtual content being shown more prominently than the other. Figure 10a-10c This is a follow-up to the above example. In this case, if the first set of virtual content 60a, 60b is being displayed, movement of the mobile phone 54 to the left causes it to correspondingly shrink in size on the user interface 70"' as the second set of virtual content 60c, 60d expands (which may be at the same or different rates). Fig.10b The user interface 70'" is shown as the user progresses to the left, where it can be seen that the first set of virtual content 60a', 60b' begins to shrink and a portion of the second set of virtual content 60c', 60d' begins to appear. Fig.10c The user interface 70"' is shown when the correct amount of movement is reached, similar to Fig.9c , although minimized versions 60a" of at least some of the first set of virtual content 60a remain. In some embodiments, the first set of virtual content 60a may be completely removed, leaving only the second set of virtual content 60c, 60d.

[0079] In some example embodiments, a distinction may be made between saved virtual content that may be displayed (based on a current or recently captured image or image portion, as in FIGS. 7-10 ) and saved virtual content that is not associated with any current or recently captured image of the current or recent location (i.e., not related to the current location or surroundings). In this case, two different forms of indicators 72 ′, 72 ″ may be provided on the user interface 70 ″″, such as Figures 11a-11c shown.

[0080] The first type of indicator 72' may indicate availability, and possibly the number of sets of virtual content that may be consumed at any time in response to a user selection. For example, its selection may take the user to a Figure 6b The gallery view shown.

[0081] The second type of indicator 72" may indicate the availability and possible number of sets of virtual content that may now be consumed, as well as additional context provided by any of the embodiments of FIGS. 7-10. For example, the second type of indicator 72" may indicate that a Fig.11c The direction and amount of positional change required for the correct position shown, for example, by position, size and / or deformation.

[0082] In addition, more than one second type indicator 72" may be used to indicate corresponding other virtual content that may be replayed or reproduced by moving in different corresponding directions. For example, in addition to the second type indicator 72" on the left side of the user interface 70"", there may be another second type indicator (not shown) on the right side of the user interface.

[0083] Except for the case where indication is made by visual, auditory and / or tactile feedback, any or all of the above operations and embodiments can be performed in the background so that the user is unaware of the processing being performed. The virtual content and related data stored on the memory device 29 can be automatically deleted when consumed, after a predetermined time period and / or when the user's position has moved to a sufficiently far position for a predetermined time period. Alternatively or additionally, the above virtual content and associated data can be compressed and stored in cloud storage after consumption or after a predetermined time period. In some embodiments, the above virtual content can be shared with other users.

[0084] In some example embodiments, the overlap with a captured image or image fragment may be determined based on the relevance to the virtual content. For example, along with trees, the virtual content may be related to specific trees at relevant locations. The virtual content may specify when a specific tree was planted and by whom. In this case, it may be important to show a captured image of a specific tree. On the other hand, the virtual content may be general, such as about nature conservation, in which case the overlap of the segmented object photo may not seem very obvious, or is not directly used for visual presentation (and is therefore only used for content matching). Such information about the importance or relevance of the object can be derived from metadata that is part of the virtual content, or can be derived, for example, based on AI analysis of text and / or audio content.

[0085] Fig.12 An apparatus according to an embodiment is shown, which may include a media player 10 as described herein. The component may be configured to perform the operations described herein, for example, with reference to the operations described in any previous process. The apparatus includes at least one processor 120 and at least one memory 121 directly or closely connected to the processor. The memory 121 includes at least one random access memory (RAM) 121a and at least one read-only memory (ROM) 121b. Computer program code (software) 125 is stored in ROM 121b. The apparatus may be connected to a transmitter (TX) and a receiver (RX). The apparatus may optionally be connected to a user interface (UI) for indicating the apparatus and / or for outputting data. At least one processor 120 is arranged together with at least one memory 121 and computer program code 125 to cause the apparatus to perform at least a method according to any of the foregoing processes. At least one processor 120 may communicate with AR glasses, goggles, mobile phones, or tablets described herein.

[0086] Fig.13 A non-transitory medium 130 according to some embodiments is shown. The non-transitory medium 110 is a computer readable storage medium. It may be, for example, a CD, a DVD, a USB memory stick, a Blu-ray disc, etc. The non-transitory medium 130 stores computer program code to cause the apparatus to perform the method of any of the aforementioned processes.

[0087] The user input may be a gesture, including one or more of: a tap, a swipe, a slide, a press, a hold, a rotation gesture, a static hover gesture of a user interface close to the device, a moving hover gesture close to the device, bending at least a portion of the device, squeezing at least a portion of the device, a multi-finger gesture, tilting the device, or flipping the control device. Additionally, the gesture may be any free-space user gesture using the user's body (such as their arm) or a stylus or other element suitable for performing a free-space user gesture.

[0088] The devices shown in the above examples can be portable electronic devices, laptop computers, mobile phones, smart phones, tablet computers, personal digital assistants, digital cameras, smart watches, smart glasses, pen computers, non-portable electronic devices, desktop computers, displays, smart TVs, servers, wearable devices, virtual reality devices, or modules / circuit devices for one or more of the above.

[0089] Any of the mentioned devices and / or other features of the specifically mentioned devices may be provided by devices that are arranged so that they are configured to perform the desired operation only when enabled, such as power on, etc. In this case, they may not necessarily load appropriate software into active memory when not enabled (e.g., closed state), but only load appropriate software when enabled (e.g., open state). The device may include hardware circuitry and / or firmware. The device may include software loaded onto the memory. Such software / computer program may be recorded on the same memory / processor / functional unit and / or one or more memories / processors / functional units.

[0090] In some examples, the specifically mentioned devices may be pre-programmed with appropriate software to perform the desired operations, and where the appropriate software may be enabled for use by a user who downloads a "key", e.g., to unlock / enable the software and its associated functionality. Advantages associated with such examples may include reduced requirements for downloading data when the device requires more functionality, and this may be useful in examples where the device is deemed to have sufficient capacity to store such pre-programmed software for functionality that may not be enabled by the user.

[0091] Any mentioned device / circuitry / component / processor may have other functions in addition to the functions mentioned, and these functions may be performed by the same device / circuitry / component / processor. One or more aspects disclosed may include related computer programs recorded on appropriate carriers (e.g., memory, signal) and electronic distribution of computer programs (which may be source / transport encoded).

[0092] Any "computer" described herein may include a collection of one or more individual processors / processing elements that may or may not be located on the same circuit board, or the same area / location of a circuit board, or even on the same device. In some examples, one or more of any processors mentioned may be distributed across multiple devices. The same or different processors / processing elements may perform one or more functions described herein.

[0093] The term "signaling" may refer to one or more signals transmitted as a series of transmitted and / or received electrical / optical signals. The series of signals may include one, two, three, four or even more individual signal components or different signals to constitute the above-mentioned signaling. Some or all of these individual signals may be transmitted / received simultaneously, sequentially and / or in a manner such that they overlap each other in time by wireless or wired communication.

[0094] With reference to any discussion of any computer and / or processor and memory (including, for example, ROM, CD-ROM, etc.) mentioned, these may include computer processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and / or other hardware components that have been programmed in a manner capable of carrying out the functions of the present invention.

[0095] Applicants hereby disclose separately each individual feature described herein and any combination of two or more such features, so that such features or combinations can be performed based on the specification as a whole, in view of the general knowledge of those skilled in the art, regardless of whether such features or feature combinations solve any problems disclosed herein, and without limiting the scope of the claims. Applicants point out that the disclosed aspects / examples may consist of any such individual features or feature combinations. In view of the foregoing description, it is clear to those skilled in the art that various modifications may be made within the scope of the present disclosure.

[0096] Although the basic novel features applied to the examples thereof have been shown and described and pointed out, it should be understood that various omissions and substitutions and changes in the form and details of the described apparatus and methods may be made by those skilled in the art without departing from the scope of the present disclosure. For example, all combinations of elements and / or method steps that perform substantially the same function in substantially the same manner to achieve the same results are within the scope of the present disclosure. In addition, it should be recognized that the structures and / or elements and / or method steps shown and / or described in conjunction with any disclosed form or example may be incorporated into any other disclosed or described or suggested form or example as a general design matter of choice. Moreover, in the claims, means-plus-function clauses are intended to cover structures described herein as performing the functions described, and not only structural equivalents, but also equivalent structures.

Claims

1. A device for outputting virtual content, comprising components for the following operations: providing different first and second sets of virtual content based on a location associated with a display component for overlapping display at the display component at a first time; determining that the first set of virtual content takes precedence over the second set of virtual content; Based on the determination: Prioritizing display of the first set of virtual content over display of the second set of virtual content at the display component; causing display of an indicator via the display component that the second set of virtual content has been saved for later playback; as well as The display of the second set of virtual contents is enabled at a subsequent second time. 2 . The apparatus of claim 1 , wherein the component is configured to prioritize display by causing display of the first set of virtual content rather than the second set of virtual content at the display component at the first time. 3 . The apparatus of claim 1 , wherein the component is configured to determine that the first set of virtual content takes precedence over the second set of virtual content if the first set of virtual content is displayed before the first time.

4. The apparatus of claim 1 or claim 2, wherein the component is configured to determine that the first set of virtual content takes precedence over the second set of virtual content if both the first set of virtual content and the second set of virtual content would conflict when displayed at the first time.

5. The apparatus of claim 4, wherein the component is configured to identify that the first group of virtual content and the second group of virtual content will conflict if respective positions of the first group of virtual content and the second group of virtual content will be within a predetermined distance of each other and / or will overlap.

6. The apparatus of claim 1 or claim 2, wherein enabling display of the second set of virtual contents at the second time comprises: Display of the second set of virtual content is automatically enabled in response to the first set of virtual content no longer being displayed.

7. The apparatus of claim 1 or claim 2, wherein enabling display of the second set of virtual contents at the second time comprises: Display of the second set of virtual content is enabled in response to receiving a user selection of the second set of virtual content.

8. An apparatus according to claim 1 or claim 2, wherein the component is further configured to cause capture of a real-world image at the first time or at a time close to the first time by a camera associated with the display component, and is configured to associate at least a fragment of the captured image with the second set of virtual content for displaying the at least fragment of the captured image with the second set of virtual content at the second time.

9. An apparatus according to claim 8, wherein after the first time, the component is further configured to determine a similarity between the at least segment of the captured image and a portion of a current or most recent real-world image captured by the camera, and configured to cause display of the captured image segment at the display component in response to determining at least a predetermined level of similarity.

10. An apparatus according to claim 9, wherein the component is configured to cause display of the captured image segment such that the captured image segment overlaps a similar portion of the real-world image.

11. An apparatus according to claim 9, wherein the component is configured to identify that the similar portion of the current or most recent real-world image is outside a predetermined display area of ​​the display component, and is configured to provide one or more prompts via the display component in response to the identification, the one or more prompts indicating the camera movement required to move the similar portion of the current or most recent real-world image toward the predetermined display area.

12. The apparatus of claim 11, wherein the one or more cues change appearance in response to movement of the camera to indicate progress toward the predetermined display area.

13. The apparatus of claim 12, wherein the component is configured to provide a modified form of the second set of virtual content and / or the captured image segment as the hint.

14. A method for outputting virtual content, comprising: providing different first and second sets of virtual content based on a location associated with a display component for overlapping display at the display component at a first time; determining that the first set of virtual content takes precedence over the second set of virtual content; Based on the determination: Prioritizing display of the first set of virtual content over display of the second set of virtual content at the display component; causing display of an indicator via the display component that the second set of virtual content has been saved for later playback; as well as The display of the second set of virtual contents is enabled at a subsequent second time.

15. A computer readable medium comprising a computer program code stored thereon, the computer readable medium and the computer program code being configured to perform the following method when executed on at least one processor: providing different first and second sets of virtual content based on a location associated with a display component for overlapping display at the display component at a first time; determining that the first set of virtual content takes precedence over the second set of virtual content; Based on the determination: Prioritizing display of the first set of virtual content over display of the second set of virtual content at the display component; causing display of an indicator via the display component that the second set of virtual content has been saved for later playback; as well as The display of the second set of virtual contents is enabled at a subsequent second time.

Images