Integrating external messages in an augmented reality environment

Abstract

A method in a computing device for providing an external message in an augmented reality environment, comprising: receiving an external message at the device, the external message including metadata related to a desired presentation of the external message in the augmented reality environment; analyzing the message and determining parameters for integration defined in the metadata; and integrating the message into the augmented reality environment based on a comparison of characteristics of the augmented reality environment to the metadata related to the desired presentation of the external message in the augmented reality environment; and rendering the augmented reality environment with the external message integrated into the augmented reality environment.

Description

[Technical field]

[0001] The present invention relates to a method and system for integrating messages, such as advertisements, into augmented reality content in a manner that does not disrupt the augmented reality scene. [Background technology]

[0002] It has become common to present messages, such as advertisements and announcements, in the form of banners, videos, notifications, or in-play content in the form of photos, videos, or logos integrated into the presentation in an app. Mobile app developers connect their apps to ad networks, such as Google AdMob, Unity Ads, etc., so that messages provided by the ad networks can be downloaded to the device and presented within the app. This is done by integrating ad modules into the mobile app using software development kits (SDKs) provided by the ad network operators. These modules contain the application programming interfaces (APIs) necessary to communicate with the ad networks to receive messages and forward the received messages to the mobile app for presentation together with the native app content.

[0003] App developers do not need to be concerned with ad delivery decisions, i.e., decisions made by the network regarding which messages to deliver to a particular instance of a given app at a given time. Instead, the ad module can provide available contextual information such as the type of host app, the location of the device, demographic information about the user, etc. (possibly taking into account user preferences, legal restrictions, etc.), and the ad network can select among messages that are intended to be presented in the context defined by the provided contextual information.

[0004] When this type of external message is presented in an app for so-called augmented reality (AR), the external message is still presented in the same format as in a traditional 2D app screen. AR can be simply defined as the presentation of virtual content together with (e.g. superimposed on) a real-world environment, for example in goggles or glasses, a head-up display, or on the screen of a mobile phone that also displays a video presentation of the environment on the other side. If a traditional 2D message is presented as a banner on a screen or display that displays a real 3D scene (which for these purposes includes a 2D video of the 3D scene in which the device is present), the result is that the immersive experience of the AR scene is interrupted. This situation is of course even worse when the presentation of the AR is interrupted and the whole scene is temporarily replaced by the presentation of a 2D advertisement.

[0005] Therefore, there is a need for a method and system that can integrate messages into AR content in a way that is not intrusive, but rather feels natural to the user. In other words, the message should feel to the user as a natural part of the AR scene as part of the augmented reality presentation. Summary of the Invention [Problem to be solved by the invention]

[0006] SUMMARY OF THE PRESENT EMBODIMENT The present invention addresses the above-mentioned needs by providing a method in a computing device for providing external messages in an augmented reality environment. [Means for solving the problem]

[0007] The method includes receiving an external message including metadata related to a desired presentation of the external message in the augmented reality environment, integrating the external message into the augmented reality environment based on a comparison of characteristics of the augmented reality environment to the metadata related to the desired presentation of the external message in the augmented reality environment, and rendering the augmented reality environment including the external message integrated into the augmented reality environment. The metadata related to the presentation of the external message in the augmented reality environment includes technical criteria related to the integration of the external message into the augmented reality environment.

[0008] The metadata regarding the desired presentation of the external message is evaluated for completeness regarding integration, and if it is determined that the metadata is incomplete, for example, does not specify how the message should be placed or rendered in the AR scene with the desired details, it is determined whether the available metadata is sufficient for integration. If the available metadata is sufficient for integration, the message is integrated into the augmented reality environment based on the available information, even if the message does not provide an optimal user experience. If the available metadata is insufficient for integration, a process of surface detection is performed on the augmented reality environment, and based on the results, integration of the external message is performed by placing the message in relation to the detected surfaces in the augmented reality environment.

[0009] The technical criteria for integration may specify at least one of: an absolute location of the external message in the augmented reality environment, a location of the external message in the augmented reality environment relative to another object in the augmented reality environment, that the external message should be placed on a surface within the augmented reality environment, and that the external message should be placed on or near a particular object or type of object in the augmented reality environment.

[0010] Some embodiments of the invention include transmitting the context information to a remote message server, where receiving the external messages may include receiving a set of messages selected based on a comparison of the transmitted context information to selection criteria associated with each external message, and adding the received messages to local storage.

[0011] At the device, an external message to be integrated into the augmented reality environment may be selected from a set of messages based on prioritization according to a predefined scoring system for prioritizing the messages.

[0012] The metadata regarding the presentation of an external message in the augmented reality environment may, in some embodiments, include selection criteria specifying at least one of information regarding the loaded augmented reality content, features detected in the augmented reality environment, device location, and user demographics.

[0013] Messages not originally intended for presentation in an AR environment can be adapted for such presentation by a server configured to determine whether the message is pure text, and if so, convert the text to an image, determine whether the message is an image, and if so, generate and embed metadata describing how the image should be integrated into the augmented reality environment, determine whether the message is a video, and if so, generate and embed metadata describing how the image should be integrated into the augmented reality environment, and store or transmit the message as an external augmented reality message.

[0014] The metadata describing how the image should be integrated into the augmented reality environment includes technical criteria that specify at least one of: an absolute position of the external message in the augmented reality environment, a position of the external message in the augmented reality environment relative to another object in the augmented reality environment, that the external message should be placed on a surface within the augmented reality environment, and that the external message should be placed on or near a particular object or type of object in the augmented reality environment.

[0015] Some embodiments may further include embedding metadata regarding a selection of external messages for integration into the augmented reality environment including selection criteria specifying at least one of information regarding the loaded augmented reality content, features detected in the augmented reality environment, device location, and user demographics.

[0016] In another aspect of the invention, a device is configured to present an augmented reality environment and includes an augmented reality app module configured to receive external messages, a camera module configured to receive image information about a local scene from a device camera, and an augmented reality content integration module configured to create the augmented reality environment by receiving and integrating images from the camera module, primary augmented reality content from a local memory, and external messages from the augmented reality app module.

[0017] The augmented reality integration module is configured to integrate the external messages into the augmented reality environment based on metadata associated with each message, the metadata including technical criteria for integration of the external messages into the augmented reality environment. The augmented reality application module is configured to evaluate the metadata for the presentation of the external messages for completeness for integration. If the metadata is determined to be incomplete, the augmented reality application module is further configured to determine whether the available metadata is sufficient for integration.

[0018] If the available metadata is sufficient for integration, integrating the external message into the augmented reality environment based on the available information;

[0019] Integrating external messages into the augmented reality environment based on a process of surface detection (507) in the augmented reality environment and integrating external messages related to the detected surfaces in the augmented reality environment if the available metadata is insufficient for integration. It is configured as follows.

[0020] The technical criteria for integration may specify at least one of: an absolute location of the external message in the augmented reality environment, a location of the external message in the augmented reality environment relative to another object in the augmented reality environment, that the external message should be placed on a surface within the augmented reality environment, and that the external message should be placed on or near a particular object or type of object in the augmented reality environment.

[0021] The device may be further configured to receive primary augmented reality content from an augmented reality content service and external messages from a message service. [Brief description of the drawings]

[0022] Next, the present invention will be described with reference to the drawings.

[0023] [Figure 1] 1 is an example of an augmented reality scene with integrated external messages. [Diagram 2] 1 is a system that may be configured to operate in accordance with the present invention. [Diagram 3] FIG. 2 is a block diagram illustrating a device according to one embodiment of the present invention. [Figure 4] 4 is a flowchart of a process according to an embodiment of the present invention carried out on a device. [Diagram 5]1 is a flowchart of a process executed on a device when selecting and consolidating external messages according to one embodiment of the present invention. [Figure 6] 1 is a flowchart of a process according to an embodiment of the present invention executed on a server that prepares an external message for distribution and integration into an augmented reality environment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Augmented reality apps are quite sophisticated in terms of content presentation compared to traditional 2D apps. Some AR apps also provide 2D content, such as tags, grid lines, instructions, measurements, etc., that are superimposed on the 2D or 3D view, but 3D AR content is becoming more and more prevalent. To seamlessly integrate 3D AR content into a scene, the AR content should not only be positioned accurately in the view, but also in a way that preserves the 3D aspects of the elements introduced. In the following, 3D elements include elements that are rendered in 2D on a display together with a 2D video rendering of the real environment, but in a way that preserves the 3D nature of the elements, for example in perspective, texture, shadows, and other aspects.

[0025] Such 3D AR content is typically experienced by activating a camera on a device such as a mobile phone, detecting the environment (e.g., flat surfaces, or objects via image recognition, etc.), and placing the AR content within the scene according to the features of the detected environment. For example, a user can place a 3D dancing character as an AR element in the user's living room and enjoy the experience. Alternatively, a user can hold the phone's camera in front of their face and see AR glasses superimposed on a representation of their face, with the glasses positioned based on facial recognition software.

[0026] If such a presentation is interrupted by an in-app message such as an advertisement, for example when a user is viewing a dancing 3D character and an in-app advertising video commercial interrupts, the user may have to continue holding the phone still throughout the entire video presentation to ensure that the 3D AR content can continue correctly when the presentation ends. If the user moves the phone during the presentation of a 2D video advertisement while the camera is temporarily stopped, environmental data may be corrupted. In addition to this technical disruption, suffering an interruption while experiencing 3D AR content and having to watch a 2D video is inconvenient and perhaps annoying for most users.

[0027] The present invention provides a method and system for integrating AR messages, referred to herein as external AR messages, into other AR content in a manner that is not intrusive but rather allows users to experience the external AR messages in an AR format that is in harmony with the main AR experience, and also allows advertising providers and others to generate external AR messages that can be integrated in an easy manner. Instead of experiencing a 2D video advertisement that occupies the entire screen or a 2D banner that is simply superimposed on a portion of a 3D AR presentation, the user experiences the advertisement as a secondary content in AR in addition to the main AR content. For example, FIG. 1 shows an exemplary AR scene in which a user is viewing an augmented environment 101 in which a real person 102 is dancing together with a 3D dancing Santa Claus 103 content (the main AR content). At the same time, a soft drink advertisement is presented to the user as a 3D board 104 that displays information about the soft drink as secondary content to the side of the main content 103. In this way, the user experiences the external AR message together with the main AR content 103 in the AR environment 101 in a non-intrusive manner, facilitating the user experience.

[0028] The main module provided by the present invention is an AR app module that is integrated into an AR app by an app developer. The AR app may be developed on a computer 201 that includes the tools necessary for app development and may further include a software development kit (SDK) consistent with the present invention. The SDK may include tools as well as software modules or libraries that allow the integration of the AR app module into the main AR app. After the main AR app is developed, it may be provided to a user, for example via an app store, and the user may install the AR app on a device such as a mobile phone 202. The main AR app may now be able to access the main AR content from the server 203, which is integrated into the AR environment, for example Santa Claus 103 is integrated into the AR environment 101 of FIG. 1.

[0029] As already mentioned, the AR app module is integrated into the primary AR app and is therefore installed on the device 202 together with the primary AR app. In parallel with the integration of the primary AR content of the primary AR app into the AR environment, the AR app module can contact the external AR message repository 204 to request an external AR message. Upon receiving the external AR message, the AR app module can forward the external AR message via the API to the primary AR app, which can then integrate the external AR message into the AR environment, such as integrating the soft drink information board 104 into the AR environment 101 of FIG. 1.

[0030] The various devices and computers shown in FIG. 2 can be configured to communicate with one another via a computer network 205, which may be or may include a wide area network such as the Internet.

[0031] By providing the necessary APIs and requirements for formatting and placement of external AR messages, the present invention enables external AR messages, such as advertisements and announcements, to be easily created, distributed, and displayed to users of any mobile app that integrates an AR app module.

[0032] It should be noted that this solution is not limited to AR apps that only play predefined AR content, such as 3D characters or 2D AR filters. Embodiments of the present invention can also provide external AP messages to AR games and other interactive environments. Furthermore, external AR messages are not limited to static presentations, but may include active elements, thus enabling requests for interaction and action.

[0033] An embodiment of the present invention may be configured to process a variety of AR formats, some of which represent rich content developed with AR in mind. Examples of such content include 3D AR characters performing actions in the AR environment, such as mascots, logos, cartoon characters, as well as animated or static 3D AR objects such as hamburgers, soft drink bottles, cars, etc. The 3D object may be a 3D board with the appearance of an advertising billboard, a TV set, etc. The board can receive and display 2D video messages, thus providing integration into the 3D AR environment with the board as the rationale for the presence of the presentation for external messages originally developed for 2D media. The size, position, and viewpoint of the 2D presentation are determined by the relative position of the board to the observer in the scene.

[0034] Other content may be developed for 2D, but may be placed within the 3D AR environment rather than being able to occupy the entire display during presentation. For example, a 2D video may be placed as a floating image next to another AR object, such as a soft drink presentation 104 next to Santa Claus 103 in FIG. 1. This is similar to the billboard described above, except that there is no presentation of a board or screen that provides the rationale for the presence of the 2D image. Instead, the image appears to be floating in the air, but still has a defined position and orientation, and is viewed from an angle determined by its position relative to the user.

[0035] 2D AR images or filters, whether animated or static, can be used especially when the AR app uses image recognition to introduce AR elements. For example, if the AR app uses face recognition to display a filter such as AR sunglasses, the AR app module can flash an additional sunglasses accessory as a 2D image in the AR.

[0036] The AR app module integrated into the main AR app allows the main AR app to present external AR messages to the user along with the main AR content. Figure 3 shows in block diagram form how an embodiment of the present invention can be configured and operated. A main AR app 301 installed on the device 202 includes several modules, one of which is an AR app module 302. The main AR app 301 retrieves the main AR content from a service 303 running on the server 203, while the AR app module 302 retrieves external AR messages from a message service 304 running on an external AR message repository 204.

[0037] Additionally, the primary AR app 301 may include a camera module 305 configured to access, control, and capture images from a camera that is part of the device 202. Images from the camera are provided by the camera module 305 to the AR integration module, which also receives primary AR content provided by the AR content service 303, as well as external AR messages provided via an API from the AR app module 302. Communication with the services 303, 304 may be via the network 205.

[0038] The environment and AR content may be integrated into a presentation of the AR environment in an AR content integrator 306, from which it may be provided to an AR renderer 307, which may be configured to control the display on the device 202.

[0039] The AR app module 302 includes an endpoint that the developer of the primary AR app can use to provide information and a callback that indicates that the AR presentation has started along with information about how the user will experience the AR content. This may be, for example, information about whether the primary AR app uses surface detection, image recognition, etc. In this way, when the user accesses the AR content, the AR app module knows what means and format the external AR message should have to be correctly displayed with the primary AR content. For example, if surface detection is used to show the primary AR content, such as a dancing 3D Santa Claus, the AR app module uses the same surface detection information to position the 3D external AR message in a manner that is not intrusive to the user.

[0040] Reference is now made to FIG. 4, a flow chart illustrating the interaction between the main AR app 301, the AR app module 302, and the external AR message service 304. In a first step 401, the main AR app 301 is initialized, which includes loading the AR app module 302. Additionally, the main AR app 301, in some embodiments, can provide the AR app module 302 with information regarding any AR content loaded from the AR content service 303, or any particular features detected or recognized in the image provided by the camera module 305. Such information can, of course, change during the execution of the app, for example, because the user points the camera in a new direction or requests new content from the AR content service 303. In that case, in at least some embodiments of the invention, updated information can be provided to the AR app module 302.

[0041] In a next step 402, the AR app module 302 loads a set of default external AR messages already present on the device 202. This ensures that the AR app module 302 can provide external AR messages even in the absence of a connection to the external AR message service 304.

[0042] Additionally, the AR app module 302 transmits context information to the external AR message service 304 in step 403. This context information may vary in different embodiments, but may include the location of the device, demographic information about the user, message preferences set by the user, the user's user ID (which may be used to access a user profile already available to the external AR message service 304), information about the primary AR app 301, and information about AR content loaded by the primary AR app 301 or features or objects detected in an image obtained from the device's camera.

[0043] In step 404, the AR message service compares the received context information with selection criteria defined for the various external AR messages stored in the database 204 and selects a set of messages having criteria satisfied by the context information. This set of messages can then be sent to the AR app module 302. Upon receiving the selected external AR message, the AR app module 302 adds the received external AR message to a default message and any other messages that may have been previously downloaded and stored locally (step 405), making all locally stored external AR messages available for integration in the AR environment.

[0044] Although the flowchart of FIG. 4 shows a step-by-step process in which each step follows another for practical reasons, it will be understood that an embodiment of the invention executed on a physical device does not necessarily operate in this exact manner. For example, the step of sending context information to the external AR message service 304 may be executed regardless of whether the default set of messages has finished loading. Similarly, the context information may be resent if it changes during the execution of the app 301, resulting in a new message being selected by the external AR message service 304 and sent to the AR app module 302. Thus, the various steps may be implemented as routines that are called when certain events occur, and these routines are then executed until they can be completed.

[0045] Now, referring to FIG. 5, a process of selecting a particular external AR message from locally available messages and integrating the message into the AR environment will be described. As mentioned above, the external AR messages selected by the external AR message service 304 are selected because their criteria are met by the context information. However, some embodiments may include a method of prioritizing the available messages. This means that in a first step 501, the AR app module selects one of the available external AR messages from the local storage, and this can be done based on prioritization. The prioritization can be based on a scoring system related to how well the context information matches the selection criteria of each external AR message. For example, being related to features detected in the scene can be given a higher score than the device location, which can be given a higher score than the user's demographics. Various ways of prioritizing messages are known in the art and will not be discussed further, as it is just a matter of choosing criteria and a scoring system based on perceived importance (or willingness to pay for exposure).

[0046] However, some embodiments may also implement other selection criteria, for example by prioritizing sophisticated 3D AR messages over 2D video or banner presentations, or by prioritizing (or strictly selecting) content based on how it is integrated into the AR environment, for example based on surface detection or image recognition.

[0047] This means that the selection made in step 501 may include criteria provided as metadata with the external AR message, the type of the external AR message, local context information and AR environment information provided by the primary AR app 301, and metadata or data type information provided with the AR content loaded from the AR content service 303. Thus, various embodiments of the present invention may implement different functions with respect to the API. If the primary AR app 301 is configured to be able to load AR content from the AR content service 303, the AR app module may be configured to provide an endpoint that can provide information about the loaded AR content to the AR app module. This information may be forwarded to the external AR message service 304 as described above, or may be used locally to select one of several available external AR messages. For example, if the user launches an AR tour guide of the Eiffel Tower in Paris, this metadata may be forwarded to the AR app module 302 (or, in some embodiments, may be included in the AR app module 302 when integrated into the primary AR app 301 using the SDK), which forwards it to the external AR service 304. Based on this, the external AR message service 304 can select a set of messages related to restaurants, cafeterias, and bars near the Eiffel Tower, from which the AR app module 302 can select an external AR message including a 3D presentation of a sandwich promoting a famous sandwich shop near the Eiffel Tower.

[0048] It should be noted that some embodiments may not implement local storage of a set of messages. In such embodiments, no selection may be made by the AR app module 302. Instead, the external AR message service 304 selects just one message and sends this message to the device. This means that some steps described as being performed by the AR app module 302 are instead performed by the external AR message service 304.

[0049] In some situations, necessary metadata or context information may be missing. This is the case, for example, if the developer of the primary AR app 301 did not include this information when configuring the primary AR app 301, or if metadata is missing from the loaded AR content. In this context, all necessary information means all information, including metadata, that allows the AR app module 302 to select messages with selection criteria that are met by the context information, and that otherwise allows the AR app module 302 to conclude that the primary AR app 301 can correctly integrate the external AR message into the AR environment. This means that the message format, method (such as surface detection or image recognition), and any other necessary information (which may vary depending on the embodiment, content, and context) are compatible with the primary AR app 301 and any loaded content.

[0050] If the AR app module 302 concludes in step 502 that this is the case (i.e., at least one matching message is available), the process proceeds to step 503, where a message that meets all requirements is selected. The process then proceeds to step 504, where the selected external AR message is forwarded to the primary AR app 301.

[0051] If it is determined in step 502 that not all necessary information is available, the process instead proceeds to step 505, where a message that meets the available criteria is selected. This means that it is determined in step 502 that some criteria may not be met, i.e. that there is no available message for which all selection criteria are met by the context information and all technical information is available and compatible. Some information is missing, in which case it may not be known whether the external AR message matches the primary AR app 301 and other loaded AR content, or it may be determined that some criteria cannot be met. The latter may be the case that the context has changed, so that there is no available message for which all selection criteria are met by the current context, or there may be some technical incompatibility between the primary AR app 301 or other AR content loaded by the primary AR app 301 and the available external AR message.

[0052] Since there are several reasons why the external AR message selected in step 505 may not be a perfect match with respect to all criteria, in step 506 it is determined whether the selected message can still be processed by the primary AR app 301. This means that the available metadata describing the selected external AR message is sufficient for the AR app module to conclude that the AR content integrator module 306 can place the selected external AR message within the AR environment such that the AR renderer 307 can render the scene together with the external AR message in a satisfactory manner.

[0053] This determination is made by comparing the available metadata of the selected AR message with available information about the capabilities and requirements of the primary AR app, given its context, including other loaded AR content.

[0054] If, in step 506, it is determined that the primary AR app 301 can process the selected external AR message, the process proceeds to step 504 where the message is forwarded to the primary AR app 301.

[0055] However, if in step 506 it is determined that the primary AR app 301 cannot process the selected message, the process proceeds to step 507. In step 507, the AR app module 302 performs its surface detection functions (or, in some embodiments, its image recognition functions) to generate the metadata necessary to integrate the message into the AR environment. This metadata may include positioning information, and possibly other information related to the rendering.

[0056] In step 508, the generated metadata is embedded or otherwise associated with the selected external AR message to generate a message that can be processed by the primary AR app 301. The process then proceeds to step 504, where the modified message is forwarded to the AR content integrator 306 of the primary AR app 301.

[0057] When the process reaches step 509, the AR content integration module 306 has received an external AR message that can be integrated into the AR environment, and the integration is performed. The updated AR environment is then forwarded to the AR rendering module 307 and rendered on the display of the device 202.

[0058] The above description does not go into detail on how surface detection or image recognition is performed, since these are techniques well known in the art. Similarly, it is well known how to integrate AR elements into an AR environment, provided that all necessary parameters are available. In most situations, AR elements are developed to be integrated into an AR environment by software with already known capabilities. The present invention addresses the situation where an external AR message is created independently of the software that integrates and renders it, and the ability to perform this integration can be distributed to an unlimited number of different apps that can be highly diverse in capabilities and purposes. The present invention allows different apps to make their capabilities known to an integration module that acts partly as a front-end for an ecosystem of external AR content so that it can select content that suits it, and partly as a middleware that can modify AR content that does not suit it perfectly to suit it.

[0059] In this context, "fully compatible" means that the main AR app 301 can integrate and render the external message. The relevance of the external AR message to the main AR content may vary, as well as the extent to which it is experienced by the user as a natural part of the AR environment.

[0060] As already mentioned, the external AR message service 304 selects the external AR message from the database 204. This database is typically a repository of such developed AR messages and includes the necessary metadata related to the selection criteria but also to technical criteria for integration into the AR environment. However, the database 204 may further include messages developed for presentation in a traditional 2D environment, or the AR message service 304 may communicate with other databases that include such messages. Such messages may be, for example, 2D images or 2D videos. Such external messages may also be processed by the present invention. In particular, the 2D messages may be modified by an AR creation module that may be part of the AR message service 304. The AR creation module may be implemented on the computer 201 that is capable of creating external AR message content from other media types.

[0061] The AR creation module is a tool that executes the method of converting media content into AR content, and in particular into external AR messages that can be managed and presented according to the aspects of the invention described above. The module is a combination of software and hardware configured to execute the method of taking media content as input and providing AR content as output.

[0062] Some embodiments of the AR creation module are configured to receive traditional 2D messages, e.g., text, images, or videos, from a repository of messages, such as advertisements and announcements, and convert them into a format prepared for integration into the AR environment as an external AR message. Such a module may be integrated into or operate in conjunction with the AR message service 304 and the database 204. Thus, the module may be configured to receive traditional 2D messages, convert them, and store them in the database so that they can be selected and presented to a user as described above.

[0063] Regardless of whether the AR creation module is configured to operate as an automated converter that converts non-AR content into content prepared for AR integration in a server environment, or whether it is provided as part of a tool on a computer 201 operated by a designer, the specific steps performed are essentially the same. FIG. 6 shows a flow chart illustrating how this method can be performed. Again, the order of steps shown in the figures and described below has been selected to facilitate the description of certain features, but it should be understood that an embodiment operating on an actual device may, for example, perform steps in parallel and include decision points that are not simply binary. Similarly, terms such as "first," "second," and "next" should be understood to be intended to distinguish between different functions and not to indicate a strict order.

[0064] The embodiment shown in Figure 6 begins upon receipt of a message. In a first step 601, it is determined whether the message is text or contains text. If so, the process proceeds to step 602 where the entire message is converted into an image. If the message is only text, this can be done by virtually rendering the text in memory and converting the rendered text into an image format, e.g. jpg or png. If the message is an image that also contains text, the text can be converted, for example, into a layer within the image, and the image can then be flattened into a single layer.

[0065] If the message does not include text, processing proceeds from step 601 to step 603, which is also the next step after step 602. In step 603, it is determined whether the message is an image. If it is determined that the message is an image, which may have been converted to an image in step 602 or was provided as an image when initially received by the AR content creation module, the process proceeds to step 604. In step 604, the necessary metadata is created. The metadata may include selection criteria, but may also include additional information related to AR integration. For example, the metadata may indicate that the image should be placed on a surface detected in the AR environment using surface detection, and may specify size, dimensions, resolution, display duration, and possibly other information. After the information required for AR integration is added, the image may be stored in the database 204 as an external AR message in step 608.

[0066] If in step 603 it is determined that the message is not an image, processing continues to step 605 where it is determined whether the message is a video. If so, processing continues to step 606 where video metadata is generated and embedded in a process similar to that for images, except that the metadata may include other parameters.

[0067] Optionally, metadata may be embedded in the message and the message may be stored in database 204 in step 608 .

[0068] Finally, if it is determined that the message is not a video, processing can proceed to step 607, where the message is discarded since it cannot be converted into an external AR message.

[0069] The above description does not mention additional media, such as audio or haptic. Of course, the present invention may be modified to handle content that includes such additional information together with images or videos. Also, while the ability to handle text, images, and videos, possibly in combination with audio or haptic, may be sufficient for the conversion of traditional 2D messages, the AR creation module installed as a tool on a computer operated by a designer may include additional functionality, such as the ability to add AR-related metadata to 3D objects.

[0070] Finally, the AR creation module can include certain additional features, such as natural language processing and image recognition. Such features may be included as metadata describing the content, used to generate additional information from the message to modify selection criteria, or to generate 3D AR content for a richer and more natural user experience. For example, if the message is an image of a running shoe, an image recognition algorithm can identify it as such and automatically generate a 3D shoe model. In this manner, the ecosystem can convert at least some 2D or text ads into 3D ads, improving the user experience.

Claims

1. A method for a computing device that provides external messages in an augmented reality environment, The steps include receiving an external message which includes metadata relating to a desired presentation of the external message in the augmented reality environment, A step of integrating the external message into the augmented reality environment based on a comparison between the characteristics of the augmented reality environment and the metadata relating to the desired presentation of the external message in the augmented reality environment, A step of rendering the augmented reality environment, including the external message integrated into the augmented reality environment. Includes, The metadata relating to the presentation of the external message in the augmented reality environment includes technical criteria relating to the integration of the external message into the augmented reality environment. The metadata relating to the presentation of the external message is evaluated for completeness with respect to integration (502), and if the metadata is determined to be incomplete, it is determined whether the available metadata is sufficient for integration. If the available metadata is sufficient for integration, the external message is integrated into the augmented reality environment based on the available information. A method wherein, if the available metadata is insufficient for integration, the external message is integrated into the augmented reality environment based on a process of surface detection (507) in the augmented reality environment and integration of the external message related to the detected surface in the augmented reality environment.

2. The method according to claim 1, wherein the technical criteria relating to the integration specify at least one of the following: the absolute position of the external message in the augmented reality environment; the position of the external message in the augmented reality environment relative to another object in the augmented reality environment; that the external message should be placed on or near a particular object or a particular type of object in the augmented reality environment.

3. The further step includes sending context information to a remote message server (403), The method according to claim 1 or 2, wherein the step of receiving the external messages includes receiving a set of messages selected based on a comparison of the transmitted context information with selection criteria associated with each external message (405), and adding the received messages to local storage.

4. The method according to claim 3, wherein the external message is selected from a set of messages based on prioritization according to a predetermined scoring system for prioritizing messages (501).

5. The method according to claim 1 or 2, wherein the metadata relating to the presentation of the external message in the augmented reality environment includes a selection criterion specifying at least one of the following: information about the loaded augmented reality content, features detected in the augmented reality environment, device location, and user demographics.

6. A method in a server for generating external messages that can be integrated into an augmented reality environment, The steps include determining whether the message is pure text (601), and if it is pure text, converting the text into an image (602), The steps include determining whether the message is an image (603), and if it is an image, generating and embedding metadata that describes how the image should be integrated into the augmented reality environment (604), The steps include determining whether the message is a video (605), and if it is a video, generating and embedding metadata that describes how the video should be integrated into the augmented reality environment (606), The steps (608) include storing or transmitting the aforementioned message as an external augmented reality message and Methods that include...

7. The method according to claim 6, wherein the metadata describing how the image should be integrated into an augmented reality environment includes a technical criterion specifying at least one of the following: the absolute position of the external message in the augmented reality environment; the position of the external message in the augmented reality environment relative to another object in the augmented reality environment; that the external message should be placed on or near a particular object or a particular type of object in the augmented reality environment.

8. The method according to claim 6 or 7, further comprising the step of embedding metadata relating to the selection of the external message for integration into the augmented reality environment, which includes a selection criterion specifying at least one of the following: information about loaded augmented reality content, features detected in the augmented reality environment, device location, and user demographics.

9. A device (202) comprising: an augmented reality application module (302) configured to present an augmented reality environment and to receive external messages; a camera module (305) configured to receive image information about a local scene from a device camera; and an augmented reality content integration module (306) configured to create an augmented reality environment by receiving and integrating images from the camera module (305), main augmented reality content from local memory, and external messages from the augmented reality application module (302), wherein The augmented reality integration module (306) is configured to integrate external messages into the augmented reality environment based on metadata associated with each message, which includes technical criteria for integrating the external messages into the augmented reality environment. The augmented reality application module (302) evaluates the metadata relating to the presentation of the external message for completeness with respect to integration (502), and if the metadata is determined to be incomplete, it determines whether the available metadata is sufficient for integration. If the available metadata is sufficient for integration, the external message is integrated into the augmented reality environment based on the available information. If the available metadata is insufficient for integration, the external messages are integrated into the augmented reality environment based on the process of surface detection (507) in the augmented reality environment and the integration of the external messages related to the detected surfaces in the augmented reality environment. Device (202) is configured as follows.

10. The device (202) according to claim 9, wherein the technical criteria relating to the integration specify at least one of the following: the absolute position of the external message in the augmented reality environment; the position of the external message in the augmented reality environment relative to another object in the augmented reality environment; that the external message should be placed on or near a particular object or a particular type of object in the augmented reality environment.

11. The device (202) according to claim 9 or 10, further configured to receive primary augmented reality content from an augmented reality content service (303) and external messages from a messaging service (304).

Images