Augmented Reality

Abstract

The present application relates to augmented reality. According to an example aspect, there is provided an apparatus comprising at least one processing core and at least one memory comprising computer program code, the at least one memory and the computer program code being configured to, together with the at least one processing core, cause the apparatus to at least: obtain a user's movement context; based at least in part on the movement context, rank at least two objects based on their visibility to the user; and based at least in part on the rank, determine at least one of the at least two objects as a placement object for an augmented reality information element.

Description

[0001] This application is a divisional application of the patent application with application number 201610160153.1 and invention name “Augmented Reality” filed on March 21, 2016. Technical Field

[0002] The invention relates to the field of information management for augmented reality systems and / or camera systems. Background Art

[0003] Augmented reality is an arrangement in which a direct or indirect view of a real physical environment is supplemented or enhanced with additional or supplementary information. The supplementary information may relate to objects in the view of the physical environment, such that the supplementary information may provide additional information related to the real physical objects in the view of the physical environment.

[0004] An example of augmented reality is a smartphone camera being used to capture a view of a landscape, such as a city. The view may be provided to a screen included in the smartphone as if the user had panned the smartphone to capture a digital photograph of the landscape using the camera. Software running on the smartphone may determine at least some of the content of the view, for example, based on the orientation and position of the smartphone, both of which may be determined using sensor data from sensors included in the smartphone.

[0005] Using the orientation and location of the smartphone, the software can insert supplemental information about objects in the view into the view of the landscape. For example, where the software is related to operating a machine, the inserted supplemental information can be related to how the user should interact with the machine controls to use the machine.

[0006] In addition to smartphones, the user may wear glasses that include a transparent display, allowing the user to see through the glasses to view the outside world, and augmented reality software may be overlaid on the transparent display information related to objects in the view that the user is able to see.

[0007] In some cars, the windshield is used, at least in part, to convey information to the user. In this case, information related to operating the car can be reflected to the driver's eyes via the windshield. In the case where the user is able to view the actual physical view through the display, the view is direct, while in the case where the user views the physical view on a non-transparent display, as in the case of a smartphone, the view can be considered indirect. Summary of the invention

[0008] The invention is defined by the features of the independent claims. Some particular embodiments are defined in the dependent claims.

[0009] According to a first aspect of the present invention, a device is provided, comprising at least one processing core and at least one memory comprising computer program code, wherein the at least one memory and the computer program code are configured to, together with the at least one processing core, enable the device to at least: obtain a user's movement context; at least partially based on the movement context, grade at least two objects based on their visibility to the user; and at least partially based on the grade, determine at least one of the at least two objects as a placement object for an augmented reality information element.

[0010] Various embodiments of the first aspect may include at least one feature from the following bulleted list:

[0011] the at least one memory and the computer program code being configured to, with the at least one processing core, cause the apparatus to rank the at least two objects based at least in part on: estimating a field of view of the user based at least in part on the movement context, and estimating how long each of the at least two objects is in the field of view;

[0012] The at least one memory and the computer program code are configured to, with the at least one processing core, cause the apparatus to: estimate the field of view based at least in part on a navigation route included in the mobile context;

[0013] The at least one memory and the computer program code are configured to, with the at least one processing core, cause the apparatus to: estimate the field of view based at least in part on a direction of movement included in the movement context of the user;

[0014] The at least one memory and the computer program code are configured to, with the at least one processing core, cause the apparatus to: rank the at least two objects based at least in part on a movement context of at least one of the at least two objects;

[0015] The movement context of the user includes information about the past locations of the user;

[0016] The movement context further includes information about where the user has looked in the past;

[0017] The at least one memory and the computer program code are configured to, with the at least one processing core, cause the apparatus to: rank the at least two objects based at least in part on an object type data structure;

[0018] The at least one memory and the computer program code are configured to, with the at least one processing core, cause the apparatus to: select the augmented reality information element according to the identity of the user;

[0019] The augmented reality information element includes at least one of: a traffic announcement, a personal message, an advertisement, a general public notice, and a status message related to the device;

[0020] The at least one memory and the computer program code are configured to, with the at least one processing core, cause the apparatus to: rank the at least two objects based at least in part on object noticeability values;

[0021] The at least one memory and the computer program code are configured to, with the at least one processing core, cause the apparatus to: rank the at least two objects based at least in part on a three-dimensional neighborhood model used to determine visibility of the at least two objects to the user.

[0022] According to a second aspect of the present invention, there is provided an apparatus comprising at least one processing core and at least one memory comprising computer program code, wherein the at least one memory and the computer program code are configured to, together with the at least one processing core, enable the apparatus to at least: identify an object of interest; predict at least one of a movement path, a movement speed, and a movement pace of the object of interest; and adjust at least one camera setting based on the movement path of the object of interest and based on a user's movement context.

[0023] Various embodiments of the second aspect may include at least one feature from the following bulleted list:

[0024] The at least one memory and the computer program code are configured to, with the at least one processing core, cause the apparatus to: trigger the camera in response to determining that there is a clear shot of the object of interest;

[0025] The at least one memory and the computer program code are configured to, with the at least one processing core, cause the apparatus to: adjust at least one of a focus camera setting and a shutter speed camera setting.

[0026] According to a third aspect of the present invention, a method is provided, comprising: obtaining a user's movement context; grading at least two objects based on their visibility to the user, at least in part based on the movement context; and determining at least one of the at least two objects as a placement object for an augmented reality information element, at least in part based on the grading.

[0027] Various embodiments of the third aspect may include at least one feature corresponding to features from the aforementioned bulleted list presented in connection with the first aspect.

[0028] According to a fourth aspect of the present invention, a method is provided, which includes: identifying an object of interest; predicting at least one of a movement path, a movement speed, and a movement pace of the object of interest; and adjusting at least one camera setting based on the movement path of the object of interest and based on a user's movement context.

[0029] Various embodiments of the fourth aspect may include at least one feature corresponding to the features from the aforementioned bulleted list presented in connection with the second aspect.

[0030] According to a fifth aspect of the present invention, a device is provided, comprising: a component for obtaining a user's movement context; a component for grading at least two objects based on their visibility to the user at least in part based on the movement context; and a component for determining at least one of the at least two objects as a placement object for an augmented reality information element based at least in part on the grading.

[0031] According to a sixth aspect of the present invention, there is provided an apparatus comprising: a component for identifying an object of interest; a component for predicting at least one of a moving path, a moving speed, and a moving pace of the object of interest; and a component for adjusting at least one camera setting based on the moving path of the object of interest and based on a user's moving context.

[0032] According to a seventh aspect of the present invention, there is provided a non-volatile computer-readable medium having a set of computer-readable instructions stored thereon, wherein the set of computer-readable instructions, when executed by at least one processor, causes the device to at least: obtain a user's movement context; based at least in part on the movement context, rank at least two objects based on their visibility to the user; and based at least in part on the rank, determine at least one of the at least two objects as a placement object for an augmented reality information element.

[0033] According to an eighth aspect of the present invention, there is provided a non-transitory computer-readable medium having a set of computer-readable instructions stored thereon, which, when executed by at least one processor, causes a device to at least: identify an object of interest; predict at least one of a movement path, a movement speed, and a movement pace of the object of interest; and adjust at least one camera setting based on the movement path of the object of interest and based on a user's movement context.

[0034] According to a ninth aspect of the present invention, a method is provided, comprising: identifying at least two objects in a user's field of view; predicting at least one moving path of at least one of the at least two objects; determining a speed of at least one of the at least two objects; determining a movement context of the user; estimating, for each of the at least two objects, a time that the object will remain in the user's field of view based at least in part on at least one of the moving path and speed of the object and on the user's movement context; and selecting, from among the at least two objects, a first object that is estimated to stay in the user's field of view for the longest time.

[0035] Various embodiments of the ninth aspect may include at least one feature from the following bulleted list:

[0036] The method further comprises: based at least in part on the movement path of the first object,

[0037] Adjust at least one camera setting;

[0038] The method further comprises: using the first object as a placement object for an augmented reality information element;

[0039] The method further comprises selecting an augmented reality information element for placement on the first object based on a time that the first object will remain in the user's field of view.

[0040] According to the tenth aspect of the present invention, a device is provided, which includes at least one processing core and at least one memory including computer program code, and the at least one memory and the computer program code are configured to, together with the at least one processing core, enable the device to at least: identify at least two objects in a user's field of view; predict at least one moving path of at least one of the at least two objects; determine the speed of at least one of the at least two objects; determine the user's movement context; estimate the time that each of the at least two objects will remain in the user's field of view based at least in part on at least one of the moving paths and speeds of the objects and on the user's movement context; and select a first object from the at least two objects that is estimated to stay in the user's field of view for the longest time.

[0041] Various embodiments of the tenth aspect may include at least one feature corresponding to features from the aforementioned bulleted list presented in connection with the ninth aspect.

[0042] According to an eleventh aspect of the present invention, there is provided a computer program configured to implement a method according to at least one of the third, fourth or ninth aspects.

[0043] Industrial Applicability

[0044] At least some embodiments of the present invention find industrial application in facilitating communications using the principles of augmented reality. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 An example system capable of supporting at least some embodiments of the present invention is shown;

[0046] Figure 2 illustrates example use cases in accordance with at least some embodiments of the present invention;

[0047] Figure 3 An example apparatus capable of supporting at least some embodiments of the present invention is shown;

[0048] Figure 4 shows signaling according to at least some embodiments of the present invention;

[0049] Figure 5 is a first flow chart of a first method according to at least some embodiments of the present invention; and

[0050] Figure 6 is a second flow chart of a second method according to at least some embodiments of the present invention. DETAILED DESCRIPTION

[0051] By ranking objects based on their visibility and / or prominence to the user, the effectiveness of messages displayed in augmented reality can be improved. In this case, for example, the message can be related to the user or the user's environment. In other words, unlike in conventional augmented reality applications, the supplemental information does not need to be directly related to the real-life object to which the supplemental information is visually associated in the augmented view.

[0052] Figure 1 An example system is shown that can support at least some embodiments of the present invention. Figure 1 There is a map showing an intersection of streets 100 and 140. The intersection is four-way in the sense that both streets continue through the intersection, which means that the intersection can be exited from four different directions. At the four corners of the intersection, in clockwise order, are building 102, building 104, building 106, and building 108. Sidewalks 102SW and 108SW are deployed between the streets and buildings 102 and 108, respectively. Sidewalks are also deployed between buildings 104 and 106 and the streets, but are shown in the figure for clarity. Figure 1 These are not numbered.

[0053] Users can move around the city where the intersection is located. Figure 1In the example of FIG. 1 , the route taken by the user is shown by arrow 120. As shown by arrow 120, the user walks along street 100 from left to right on the sidewalk, crossing street 140 at the intersection. There may be a pedestrian crossing to enable the user to make such a crossing, for example, such a pedestrian crossing is optionally equipped with traffic lights configured to control the flow of traffic.

[0054] In this example, the user uses an augmented reality system, such as an augmented reality system based on glasses including a display mechanism. The augmented reality system may include a display mechanism and an augmented reality application running on at least one processing core. The augmented reality application may be run locally on a user device, or it may be remotely run at least partially on a server or in a cloud computing environment. Such an augmented reality system may allow light to travel from an object in the user's environment to the user's eyes, and also allow the display mechanism to provide supplementary visual information to the user. The supplementary information may be provided in a manner that associates the supplementary information with the objects included in the view. For example, the display mechanism may include a small projector that is arranged to reflect light from the inner surface of the glasses to one or a pair of eyes of the user, so that the light reflected to the user can transmit supplementary information spatially associated with the object in the real life view. Thus, the view experienced by the user may include both light from real objects in the user's environment and light provided by the display mechanism. Supplementary visual information may, for example, enhance features displayed in the user's environment, such as subway stations and / or ticket machines. Although described as a direct view example, embodiments of the present invention are equally applicable to augmented reality applications that rely on indirect views.

[0055] Buildings 102, 104, 106, and 108 may be viewed as objects in the user's physical environment. In addition, a portion or segment of a building may be viewed as an object. Further examples of objects include cars and advertising stands. Augmented reality applications may be configured to provide information elements that are visually placed on an object. By this is meant that the information elements are rendered by a display mechanism so that the information elements appear to be attached to or associated with the object when rendered to the user, such as similar to how a poster may be attached to a wall or shelf. When information elements are visually placed on an object, the information they contain may be presented to the user in a manner that feels more natural and less distracting to the user, because people are accustomed to seeing information about an object, rather than information that appears to simply float in the air and is not associated with any real-life object. Objects that visually place information elements may be referred to as placed objects.

[0056] The augmented reality application may be configured to determine at least one placement object in order to present an information element to the user. Instead of presenting information about the placement object to the user, the augmented reality application may be configured to present information targeted at the user to the user. To facilitate this, the augmented reality application may be configured to select a placement object for information in a manner that maximizes the visibility of the information element to the user. It is also possible to select, for example, from a set of information elements to be presented to the user based on at least one of the characteristics, complexity, and urgency of the placement object. For example, in response to determining that the placement object will be visible to the user for a longer period of time, an information element including a longer message may be selected for the user, and in response to determining that the placement object will be visible to the user in a shorter period of time, an information element including a shorter message may be selected.

[0057] For example, information elements may be assigned to users who are part of a group, or they may be assigned to individual users. Examples of information elements assigned to users include traffic announcements, personal messages, targeted advertisements, general public announcements, and device-related status messages. Information elements may also be assigned to users via their location, for example, information elements such as public safety announcements may be assigned to all persons in a given geographic location. An example of such an announcement is a tsunami warning.

[0058] In order to maximize the visibility or prominence of the information element when placed on the object, the placement object can be selected based on the user's movement context. The movement context can include a movement state. Typically, the movement context can include a data structure that includes at least one of the following: the user's location, the direction the user is moving, the speed the user is moving, the navigation route followed by the user, the places the user has been in the past, and the objects the user has looked at in the past. The direction of the user's gaze can be obtained by the augmented reality application via a gaze tracking function, which can be included in an augmented reality device used by the user and associated with the augmented reality application.

[0059] The navigation route followed by the user may include a navigation route that is active in the user's navigation application. For example, the user may have a smartphone or other device running a navigation application, and the device running the augmented reality application may be the same device, or another device of the user associated with the device running the navigation application. For example, such an association may occur via a subscription such as a Google, Here, or digital ecosystem account. Alternatively, for example, the navigation route may include the route of a public transportation vehicle that the user has boarded.

[0060] The user's location may be expressed in a mobile context as an address, a geographic location, or by using another appropriate expression. The direction of movement may be expressed as a direction vector, which may also include a magnitude reflecting the speed at which the user is moving. Optionally, the direction vector may be normalized and a separate number may be used to indicate the speed (if any). In the case where the user uses a navigation application, an augmented reality application may be obtained based on navigation application information that discloses whether the user is on a driving segment of the navigation route or on a sidewalk. The augmented reality application may use this information to predict the user's speed.

[0061] When an AR application knows the user’s mobile context, it can predict where the user is likely to look. Figure 1 In the example of FIG. 1 , a possible field of view for a user at point 130 is schematically illustrated by ray 132 and ray 134. Ray 134 is limited by the corner of building 106, while ray 132 is limited by the width of the field of view. The user will not be able to see everything within the field defined by rays 132 and 134, but the user is likely to see at least some things within the field, especially objects that are disposed near the center of the field of view for a longer period of time. Figure 1 , the augmented reality application may select the corner of the building 104, represented by 104A, as the placement object for the information element to be displayed to the user in the augmented reality view, because the corner 104A is in the field of view, where the user will stay for a long time when crossing the route 120, and it is near the center of the field of view. As a further reason for using the corner 104A, the augmented reality view may assess that the traffic light may make the user stay for a period of time before he can cross the street 140, so that he will face the corner 104A while waiting for the traffic light to turn green.

[0062] Typically, objects (e.g., real-life objects) may be ranked based on their predicted visibility, and at least one placement object may be selected based at least in part on the ranking. The selected placement object may then be used to display an augmented reality information object, such as an information object as described above. For example, the ranking may be based on assigning visibility scores to at least two objects. The visibility score of an object may be defined in any of a number of possible ways, for example, the visibility score may be the sum of a visibility time score and a visibility centrality score, wherein the visibility time score indicates the length of time that the object is predicted to be in the user's field of view, and the visibility centrality score may indicate the predicted proximity of the object to the center of the field of view. In some embodiments, information elements about the placement of the placement object are selected based at least in part on how long the placement object is predicted to stay in the user's field of view. In detail, information elements with more information may be visually placed on placement objects that are predicted to stay in the user's field of view for a longer time, so as to give the user time to digest the information. Information elements with less information, such as shorter messages, may be visually placed on objects that are predicted to stay in the field of view for a shorter time. The reason for this is that it is acceptable for a placed object to remain in view for a shorter time because users are able to grasp the information content of such information elements more quickly.

[0063] In some embodiments, the visibility score is the integration of the predicted visibility centrality scores over time. Integration in this manner may provide the following advantages: it may effectively capture the amount of attention that a user may assign to an object.

[0064] In addition to predicting the visibility of an object based on the user's current movement, the augmented reality application can, optionally or additionally, grade at least two objects based on where the user has been in the past. For example, due to the fact that the user spends a lot of time in his office, the walls of the user's office can be graded as having high visibility and can therefore be selected as placement objects. Gaze tracking can be used to grade objects in places where the user spends a lot of time so that objects that the user has a particular viewing habit can be selected as placement objects. In order to achieve this grading, the augmented reality application can compile historical gaze tracking information. Historical movement, position, and / or gaze tracking information can be included in the mobile context. In some embodiments, the mobile context only includes historical movement, position, and / or gaze tracking information. By history is meant that something has already happened, but it is not meant to imply that a long time has passed since then.

[0065] In some embodiments, the augmented reality application may be configured to implement grading based at least in part on object type. Objects may be associated with an object type data structure including information characterizing real-life objects in the augmented reality application. For example, a car manufacturer may prefer to transmit information elements to users via cars manufactured by itself. In such a case, at least some cars will be associated with an object type data structure including manufacturer data entries in the augmented reality application, and the augmented reality information elements provided by the manufacturer may have the following requirements: it can be transmitted to the user only on such placed objects, wherein such placed objects are indicated in the associated object type data structure as being manufactured by the same manufacturer. Thus, the user can be reminded of the due maintenance of his car by an augmented reality information element that is visually placed on cars of the same manufacture. As another example, some real-life objects have an associated object type in the augmented reality application, which indicates that the object may not be occluded by the augmented reality information element. For example, in the case where real-life objects are used to transmit real-life information related to safety, visually placing augmented reality information elements on these objects may endanger the safety of users and / or others. Examples of such objects include traffic lights and medical equipment in hospitals. For example, when a user tends to look at traffic lights often, the traffic lights may otherwise be highly ranked as frequently looked at objects and appear like promising placement objects for augmented reality information elements. To enable use of the object type data structure, the augmented reality application may be configured to associate real life objects with the object type data structure. Such association may be facilitated, for example, by user location and / or some form of image recognition software.

[0066] In some embodiments, the augmented reality application may be configured to implement a grading based at least in part on an object's attention-worthy value. An object may be associated with an attention-worthy value, an attention-worthy value may be received from an external source in the augmented reality application, and / or the augmented reality application may be configured to determine at least one attention-worthy value. For example, where an object is the only representative of its object type, its attention-worthy value may be higher. Such an example is where there is only one car on a street with a large number of pedestrians and cyclists. The car, being the only representative of the object type car, may be assigned a higher attention-worthy value.

[0067] In some embodiments, object types may be classified based on how often the user looks at them. Thus, the augmented reality application may be enabled to select objects as placement objects based on the types of objects that the user is interested in. For example, if the augmented reality application finds that the user frequently looks at guitars or sports utility vehicles (SUVs), these object types may be selected as future placement objects. To prevent irritation to the user, in this case, the augmented reality information elements may be visually placed near these objects, rather than visually placed on these objects, so as not to annoy the user by blocking the real life objects that the user is interested in viewing.

[0068] In some embodiments, in response to the user being determined by the augmented reality application to be looking at something near the information element and not looking at the information element completely, the placement of the information element can be adjusted in real time. In this case, the information element can be moved so that it is visually closer to where the user is looking.

[0069] In some embodiments, when ranking objects based on their visibility, information about the shape of the objects may be considered. Such information may describe the three-dimensional shape of the objects. For example, the shapes of buildings may be used to determine whether they block objects behind them. For example, the number of floors in a building may affect the view occlusion characteristics of the building.

[0070] In some embodiments, augmented reality applications can be used to assist in the shooting of objects that appear only in the user's field of view for a short time. Such objects may be missed due to the time spent focusing and adjusting other settings of the camera. For example, it can be determined that a car is an object that moves quickly past the user. If the user wants to take a picture of the car, the camera will need to first find the correct focal length and other settings, and then take a photo. This may cause the user to miss a photo opportunity. The augmented reality application can have an estimate of the trajectory of the car and / or the user, and it can pre-adjust the focal length of the camera to the distance between the trajectory of the user and the trajectory of the car. Thus, take pictures faster. In the case of multiple objects around the user, the system can pre-adjust the focal length or other camera settings based on the object of the least amount of time in the user's field of view. For example, an object can be identified as potentially interesting to the user based on past gaze tracking information. For example, a camera may be included in a device associated with a device running an augmented reality application, or the augmented reality application may be run on the same device that also includes a camera. In some embodiments, the augmented reality application may be further configured to trigger the camera at the correct time when the object of interest becomes visible (e.g., not blocked by a passing car).

[0071] Figure 2 Example use cases in accordance with at least some embodiments of the present invention are shown. Figure 2An augmented reality view is shown in which streets 200 and 210 and building 230 are real life objects, that is, their presence is irrelevant to the functionality of the augmented reality application. Augmented reality information element 240 is visually placed on building 230, in other words, building 230 is a placed object for information element 240. Building 230 can be ranked based on its visibility before selecting it as a placed object.

[0072] Figure 3 An example apparatus capable of supporting at least some embodiments of the present invention is shown. Device 300 is shown, which may include, for example, a device configured to run an augmented reality application. Thus, device 300 may include, for example, a computing unit of a smart phone, a pair of augmented reality glasses, or another suitable device. Included in device 300 is a processor 310, which may include, for example, a single-core or multi-core processor, wherein a single-core processor includes one processing core and a multi-core processor includes more than one processing core. Processor 310 may include more than one processor. The processing core may include, for example, a Cortex-A8 processing core manufactured by ARM Holdings or a Steamroller processing core produced by Advanced Micro Devices Corporation. Processor 310 may include at least one Qualcomm Snapdragon and / or Intel Atom processor. Processor 310 may include at least one application-specific integrated circuit ASIC. Processor 310 may include at least one field programmable gate array FPGA. Processor 310 may be a component for implementing the method steps in device 300. Processor 310 may be configured at least in part by computer instructions to implement actions.

[0073] Device 300 may include memory 320. Memory 320 may include random access memory and / or permanent memory. Memory 320 may include at least one RAM chip. For example, memory 320 may include solid-state, magnetic memory, optical memory and / or holographic memory. Memory 320 may be at least partially accessible to processor 310. Memory 320 may be a component for storing information. Memory 320 may include computer instructions that processor 310 is configured to execute. When computer instructions configured to cause processor 310 to perform specific actions are stored in memory 320, and device 300 as a whole is configured to run under the guidance of processor 310 using computer instructions from memory 320, it can be considered that processor 310 and / or at least one of its processing cores are configured to implement the specific actions. Memory 320 may be at least partially included in processor 310. Memory 320 may be at least partially external to device 300 but accessible to device 300.

[0074] The device 300 may include a transmitter 330. The device 300 may include a receiver 340. The transmitter 330 and the receiver 340 may be configured to transmit and receive information, respectively, according to at least one cellular or non-cellular standard. The transmitter 330 may include more than one transmitter. The receiver 340 may include more than one receiver. For example, the transmitter 330 and / or the receiver 340 may be configured to operate according to the Global System for Mobile Communications GSM, Wideband Code Division Multiple Access WCDMA, Long Term Evolution LTE, IS-95, Wireless Local Area Network WLAN, Ethernet, and / or Worldwide Interoperability for Microwave Access WiMAX standards.

[0075] The device 300 may include a near field communication NFC transceiver 350. The NFC transceiver 350 may support at least one NFC technology, such as NFC, Bluetooth, Wibree, or similar technology.

[0076] The device 300 may include a user interface UI 360. The UI 360 may include at least one of a display, a keyboard, a touch screen, a vibrator arranged to signal the user by vibrating the device 300, a speaker, and a megaphone. The user can operate the device 300 via the UI 360, for example, to configure augmented reality settings.

[0077] The device 300 may include a user identity module 370 or be arranged to accept a user identity module 370. The user identity module 370 may include, for example, a subscriber identity module SIM, a card installable in the device 300. The user identity module 370 may include information identifying a subscription of a user of the device 300. The user identity module 370 may include cryptographic information that may be used to verify the identity of the user of the device 300 and / or facilitate encryption of communication information and billing of the user of the device 300 for communications enabled via the device 300.

[0078] The processor 310 may be provided with a transmitter arranged to output information from the processor 310 to other devices included in the device 300 via electrical conductors internal to the device 300. Such a transmitter may include a serial bus transmitter arranged to output information to the memory 320 for storage therein, for example, via at least one electrical conductor. In addition to the serial bus, the transmitter may optionally include a parallel bus transmitter. Similarly, the processor 310 may include a receiver arranged to receive information from other devices included in the device 300 into the processor 310 via electrical conductors internal to the device 300. Such a receiver may include a serial bus receiver arranged to receive information from the receiver 340, for example, via at least one electrical conductor, for processing in the processor 310. In addition to the serial bus, the receiver may optionally include a parallel bus receiver.

[0079] The device 300 may further include Figure 3 300. For example, where the device 300 comprises a smartphone, it may include at least one digital camera. Some devices 300 may include a rear camera and a front camera, wherein the rear camera may be intended for digital photography and the front camera is used for video telephony. The device 300 may include a fingerprint sensor arranged to at least partially authenticate a user of the device 300. In some embodiments, the device 300 lacks at least one of the above devices. For example, some devices 300 may lack an NFC transceiver 350 and / or a user identity module 370.

[0080] The processor 310, the memory 320, the transmitter 330, the receiver 340, the NFC transceiver 350, the UI 360 and / or the user identity module 370 may be interconnected in a variety of different ways through electrical conductors inside the device 300. For example, each of the aforementioned devices may be individually connected to a main bus inside the device 300 to allow the devices to exchange information. However, as will be appreciated by those skilled in the art, this is only an example, and depending on the embodiment, various ways of interconnecting at least two of the aforementioned devices may be selected without departing from the scope of the present invention.

[0081] Figure 4 Signaling according to at least some embodiments of the present invention is shown. From left to right on the vertical axis, the source SRC of the augmented reality information element, the augmented reality application AR-APP, the historical database HIST, the navigation application NAVI and the display mechanism DISP are deployed. Some embodiments of the present invention may lack the historical database HIST and / or the navigation application NAVI. For example, the source may include a computing device such as a public safety agency, an advertiser or a user.

[0082] In phase 410, the source SRC provides the information element to the augmented reality application AR-APP for display to the user. Optionally, the source may specify criteria related to the type of object in the communication of phase 410. In detail, the criteria may limit the type of object on which the information element can be placed. The augmented reality application may then consult at least one of the historical database HIST and the navigation application NAVI. In the case of consulting the historical database, phases 420 and 430 are implemented. In the case of consulting the navigation application, phases 440 and 450 are implemented.

[0083] In stage 420, the application requests information from the historical database about the object viewed by the user and / or objects that were nearby in the past. In stage 430, the information requested in stage 420 is responsively provided to the augmented reality application. The requested information may form part of the mobile context.

[0084] In stage 440, the application requests information about the route along which the user is moving from the navigation application. In stage 450, the information requested in stage 440 is responsively provided to the augmented reality application. The requested information may form part of the movement context.

[0085] In stage 460, the augmented reality application ranks at least two objects based at least in part on the visibility of the objects to the user. The objects may include real life objects as described above. The grading of stage 460 may take into account at least one of the following: the optional criteria received in stage 410, the user's location, the user's mobile state, any information received in stage 430 (if this stage exists), and any information received in stage 450 (if this stage exists). For example, the above may be combined with Figure 1 The information element is placed on the information element in a first position and a second position is placed on the information element in a second position. The information element is placed on the information element in a second position and a second position is placed on the information element in a second position. ...

[0086] Figure 5 is a first flow chart of a first method according to at least some embodiments of the present invention. For example, the stages of the method shown may be implemented in a device running an augmented reality application. Figure 3 An example of such a device is shown in .

[0087] Stage 510 includes obtaining a user's movement context. Stage 520 includes ranking at least two objects based on their visibility to the user based at least in part on the movement context. Finally, stage 530 includes determining at least one of the at least two objects as a placement object for an augmented reality information element based at least in part on the ranking.

[0088] Figure 6 is a second flow chart of a second method according to at least some embodiments of the present invention.The stages of the method shown may be implemented, for example, in a smartphone or a control device configured to control functions of a smartphone.

[0089] Stage 610 includes identifying an object of interest. Stage 620 includes predicting at least one of a movement path, a movement speed, and a movement pace of the object of interest. Finally, stage 630 includes adjusting at least one camera setting based on the movement path of the object of interest and based on the movement context of the user.

[0090] It should be understood that the disclosed embodiments of the present invention are not limited to the specific structures, processing steps or materials disclosed herein, but extend to their equivalents as will be recognized by those of ordinary skill in the relevant art. It should also be understood that the terminology employed herein is used for the purpose of describing only specific embodiments and is not intended to be limiting.

[0091] References throughout this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the phrases "in one embodiment" or "in an embodiment" appearing in various places throughout this specification do not necessarily all refer to the same embodiment.

[0092] As used herein, for convenience, multiple items, structural elements, constituent elements and / or materials may appear in a public list. However, these lists should be interpreted as each member of the list being individually identified as a separate and unique member. Thus, in the absence of contrary instructions, each member of such a list should not be interpreted as the de facto equivalent of any other member of the same list based solely on their appearance in a public group. In addition, various embodiments and examples of the present invention may refer to alternatives with its various components here. It should be understood that such embodiments, examples and alternatives should not be interpreted as de facto equivalents to each other, but should be considered as separate and autonomous representations of the present invention.

[0093] In addition, the described features, structures or characteristics may be combined in any suitable or technically feasible manner in one or more embodiments. In the following description, a variety of specific details are provided, such as examples of length, width, shape, etc., in order to provide a thorough understanding of embodiments of the present invention. However, those skilled in the relevant art will recognize that the present invention may be implemented without one or more specific details, or with the use of other methods, components, materials, etc. In other cases, known structures, materials or operations are not shown or described in detail to avoid obscuring various aspects of the present invention.

[0094] Although the foregoing examples illustrate the principles of the present invention in one or more specific applications, it will be apparent to those skilled in the art that various modifications in form, use, and implementation details may be made without inventiveness and without departing from the principles and concepts of the present invention. Therefore, it is intended that the present invention be limited only by the claims set forth below.

Claims

1. An apparatus comprising at least one processing core and at least one memory including computer program code, wherein the at least one memory and the computer program code together with the at least one processing core cause the apparatus to at least: Obtain the user's mobile context; Based at least in part on the movement context, ranking at least two objects based on their visibility to the user; determining, based at least in part on the ranking, at least one of the at least two objects as a placement object for an augmented reality information element; as well as The augmented reality information element visually placed on the placement object is provided in an augmented reality view.

2. The device according to claim 1, wherein: The mobile context further comprises a data structure, the data structure comprising at least one of: the location of the user; a navigation route followed by said user; the direction in which the user is moving; the speed at which the user is moving; the user's past locations; or The object that the user has looked at in the past.

3. The apparatus of claim 2, wherein the navigation route is a navigation route active in a navigation application of the user.

4. The device according to claim 1, wherein: The at least one memory and the computer program, together with the at least one processing core, further cause the apparatus to: determine whether the user is looking at the augmented reality information element.

5. The device according to claim 1, wherein: The user's movement context enables prediction of where the user is likely to be looking.

6. The device according to claim 4, wherein: The at least one memory and the computer program code, together with the at least one processing core, further cause the apparatus to: determine to notify a source of the augmented reality information element that the user is looking at the augmented reality information element.

7. The device according to claim 4, wherein: The at least one memory and the computer program code, together with the at least one processing core, further cause the apparatus to: determine a length of time that the user gazes at the augmented reality information element.

8. The apparatus of claim 6, wherein the source of the augmented reality information element comprises a public safety agency or a computing device of the user.

9. The device according to claim 1, wherein: The at least one memory and the computer program code, together with the at least one processing core, further cause the apparatus to: determining that the user is viewing an object near the augmented reality information element; and moving the augmented reality information element to be visually closer to a location where the user is looking.

10. The device according to claim 1, wherein: The at least one memory and the computer program code, together with the at least one processing core, further cause the apparatus to: associating the at least two objects with an object type data structure; and The at least two objects are ranked based at least in part on the object type data structure. 11 . The apparatus of claim 10 , wherein an object type associated with at least one other object of the at least two objects indicates that the at least one other object should not be used as the placement object.

12. The device according to claim 11, wherein The at least one further object comprises a traffic light.

13. A method comprising: Obtain the user's mobile context; Based at least in part on the movement context, ranking at least two objects based on their visibility to the user; determining, based at least in part on the ranking, at least one of the at least two objects as a placement object for an augmented reality information element; as well as The augmented reality information element visually placed on the placement object is provided in an augmented reality view.

14. The method according to claim 13, wherein: The mobile context further comprises a data structure, the data structure comprising at least one of: the location of the user; a navigation route followed by said user; the direction in which the user is moving; the speed at which the user is moving; the user's past locations; or The object that the user has looked at in the past. The method of claim 14 , wherein the navigation route is a navigation route active in a navigation application of the user.

16. The method according to claim 13, further comprising: Determining whether the user is looking at the augmented reality information element.

17. The method according to claim 13, wherein: The user's movement context enables prediction of where the user is likely to be looking.

18. The method according to claim 16, further comprising: Determining notifies a source of the augmented reality information element that the user is looking at the augmented reality information element.

19. The method according to claim 16, further comprising: Determine a time duration that the user gazes at the augmented reality information element.

20. The method of claim 18, wherein the source of the augmented reality information element comprises a public safety agency or a computing device of the user.

21. The method of claim 13, further comprising: determining that the user is viewing an object near the augmented reality information element; as well as The augmented reality information element is moved to be visually closer to the location where the user is looking.

22. The method of claim 13, further comprising: associating the at least two objects with an object type data structure; as well as The at least two objects are ranked based at least in part on an object type data structure associated with the at least two objects.

23. The method of claim 22, wherein an object type data structure associated with at least one other object of the at least two objects indicates that the at least one other object cannot be used as the placement object.

24. The method of claim 23, wherein the at least one further object comprises a traffic light.

25. A non-transitory computer readable medium having stored thereon a set of computer readable instructions that, when executed by at least one processor, cause an apparatus to at least: Obtain the user's mobile context; Based at least in part on the movement context, ranking at least two objects based on their visibility to the user; determining, based at least in part on the ranking, at least one of the at least two objects as a placement object for an augmented reality information element; as well as The augmented reality information element visually placed on the placement object is provided in an augmented reality view.

26. A method comprising: identifying at least two objects in a user's field of view; predicting at least one movement path of at least one of the at least two objects; Determine the user's mobile context; estimating, for each of the at least two objects, a time that the object will remain in the field of view of the user based on the movement path and the movement context of the user; selecting, from among the at least two objects, a first object estimated to remain in the user's field of view for the longest time; determining the first object as a placement object for an augmented reality information element; as well as The augmented reality information element visually placed on the placement object is provided in an augmented reality view.

27. The method according to claim 26, wherein: The mobile context further comprises a data structure, the data structure comprising at least one of: the location of the user; a navigation route followed by said user; the direction in which the user is moving; the speed at which the user is moving; the user's past locations; or The object that the user has looked at in the past.

28. The method of claim 27, wherein the navigation route is a navigation route active in a navigation application of the user.

29. The method of claim 26, further comprising: Determining whether the user is looking at the augmented reality information element.

30. The method of claim 26, wherein: The user's movement context enables prediction of where the user is likely to be looking.

31. The method of claim 29, further comprising: Determining notifies a source of the augmented reality information element that the user is looking at the augmented reality information element.

32. The method of claim 29, further comprising: Determine a time duration that the user gazes at the augmented reality information element.

33. The method of claim 31, wherein the source of the augmented reality information element comprises a public safety agency or a computing device of the user.

34. The method of claim 26, further comprising: determining that the user is viewing an object near the augmented reality information element; as well as The augmented reality information element is moved to be visually closer to the location where the user is looking.

35. The method of claim 26, further comprising: associating the at least two objects with an object type data structure; as well as The at least two objects are ranked based at least in part on an object type data structure associated with the at least two objects.

36. The method of claim 35, wherein an object type data structure associated with at least one other object of the at least two objects indicates that the at least one other object cannot be used as the placement object.

37. A method according to claim 36, wherein the at least one further object comprises a traffic light.

38. An apparatus comprising at least one processing core and at least one memory comprising computer program code, the at least one memory and the computer program code, together with the at least one processing core, causing the apparatus to at least: identifying at least two objects in a user's field of view; predicting at least one movement path of at least one of the at least two objects; Determine the user's mobile context; estimating, for each of the at least two objects, a time that the object will remain in the field of view of the user based on the movement path and the movement context of the user; selecting, from among the at least two objects, a first object estimated to remain in the user's field of view for the longest time; determining the first object as a placement object for an augmented reality information element; as well as The augmented reality information element visually placed on the placement object is provided in an augmented reality view.

39. A non-transitory computer readable medium having stored thereon a set of computer readable instructions that, when executed by at least one processor, cause an apparatus to at least: identifying at least two objects in a user's field of view; predicting at least one movement path of at least one of the at least two objects; Determine the user's mobile context; estimating, for each of the at least two objects, a time that the object will remain in the field of view of the user based on the movement path and the movement context of the user; selecting, from among the at least two objects, a first object estimated to remain in the user's field of view for the longest time; determining the first object as a placement object for an augmented reality information element; as well as The augmented reality information element visually placed on the placement object is provided in an augmented reality view.

Images