Integration of external messages in augmented reality environment

Abstract

A method in a computing device for providing external messages in an augmented reality environment is provided. An external message is received in a device, the external message including metadata related to a desired presentation of the external message in an augmented reality environment. The device analyzes the message and determines parameters defined in the metadata for integration. Messages are integrated into the augmented reality environment based on a comparison of characteristics of the augmented reality environment to metadata related to desired presentation of external messages in the augmented reality environment. The augmented reality environment is then rendered with the external messages integrated into the augmented reality environment.

Description

Technical Field

[0001] The present invention relates to methods and systems for integrating messages, such as advertisements, into augmented reality content in a manner that does not interrupt the augmented reality scene. Background Art

[0002] It is common practice to present messages such as advertisements, announcements, etc. in the form of banners, videos, notifications, and real-time content in the form of pictures, videos or logos integrated into the application presentation. The developer of the mobile application connects its application to an advertising network, such as Google AdMob, Unity Ads, etc., so that the messages provided by the advertising network can be downloaded to the device and presented within the application. This is achieved by integrating the advertising module into the mobile application using a software development kit (SDK) provided by the operator of the advertising network. These modules include the necessary application programming interface (API) to communicate with the advertising network to receive messages and forward the received messages to the mobile application, so that the message is presented together with the original application content.

[0003] Application developers do not have to consider ad delivery decisions, i.e., decisions made by the network about which messages to deliver to a particular instance of a given application at a given time. Instead, the ad module can provide contextual information that can be utilized, such as the type of host application, the location of the device, demographic information related to the user, etc. - perhaps taking into account user configuration, legal restrictions, etc. - and the ad network can select among messages that are intended to be presented in the context defined by the provided contextual information.

[0004] When this type of external message is presented in a so-called augmented reality (AR) application, the external message is still presented in the same format as a traditional 2D application screen. AR can be simply defined as presenting virtual content together with (e.g., superimposed on) a real-world environment, for example, on goggles or glasses, a heads-up display, or on the screen of a mobile phone that also displays a video presentation of the environment on the other side of the phone. When a traditional 2D message is presented as a banner on a screen or display that displays a real 3D scene (for these purposes, including a 2D video of the 3D scene of the device), the result is an interruption of the immersive experience of the AR scene. Of course, the situation is even worse if the AR presentation is interrupted and the entire scene is temporarily replaced by the presentation of a 2D advertisement.

[0005] Therefore, there is a need for methods and systems that can integrate messages into AR content not in a disruptive manner, but in a way that the user experiences naturally. In other words, as part of the augmented reality presentation, the user should experience that the message naturally belongs to the AR scene. Summary of the invention

[0006] The present invention solves the above needs by providing a method for providing an external message in an augmented reality environment in a computing device. The method includes: receiving an external message, the external message including metadata related to a desired presentation of the external message in the augmented reality environment; integrating the external message into the augmented reality environment based on a comparison of characteristics of the augmented reality environment and the metadata related to the desired presentation of the external message in the augmented reality environment; and rendering the augmented reality environment using the external message integrated into the augmented reality environment. The metadata related to the presentation of the external message in the augmented reality environment includes a technical standard for integrating the external message into the augmented reality environment.

[0007] Metadata related to the desired presentation of the external message is evaluated for completeness with respect to integration, and if it is determined that the metadata is incomplete, e.g., the message does not specify in desired detail how the message should be positioned or rendered in the AR scene, determining whether the available metadata as is is sufficient for integration. If the available metadata is sufficient for integration, the message is integrated into the augmented reality environment based on the available information, even though the message may not provide an optimal user experience. If the available metadata is insufficient for integration, a surface detection process is performed on the augmented reality environment, and integration of the external message is performed based on the detection results by positioning the message in conjunction with the detected surfaces in the augmented reality environment.

[0008] The technical criteria for integration may specify at least one of: the absolute position of the external message in the augmented reality environment, the position of the external message in the augmented reality environment relative to another object in the augmented reality environment, the external message should be positioned on a surface in the augmented reality environment, and the external message should be positioned on or near a specific object or type of object in the augmented reality environment.

[0009] Some embodiments of the invention include sending context information to a remote message server.The step of receiving external messages may then include receiving a set of messages that have been selected based on a comparison of the sent context information with selection criteria associated with the respective external messages, and adding the received messages to a local memory.

[0010] In the device, the external message to be integrated into the augmented reality environment may be selected from the group of messages based on a priority ranking according to a predetermined scoring system for message priority ranking.

[0011] In some implementations, metadata associated with presentation of external messages in an augmented reality environment may include selection criteria specifying at least one of information about loaded augmented reality content, features detected in the augmented reality environment, device location, and user demographics.

[0012] A message that was not originally intended to be presented in an AR environment can be adapted for presentation by a server that is configured to: determine whether the message is plain text, and if so, convert the text into an image; determine whether the message is an image, and if so, generate and embed metadata describing how the image should be integrated into the augmented reality environment; determine whether the message is a video, and if so, generate and embed metadata describing how the image should be integrated into the augmented reality environment; and store or send the message as an external augmented reality message.

[0013] The metadata describing how the image should be integrated into the augmented reality environment includes technical standards that specify at least one of: an absolute position of the external message in the augmented reality environment, a position of the external message in the augmented reality environment relative to another object in the augmented reality environment, the external message should be positioned on a surface in the augmented reality environment, and the external message should be positioned on or near a specific object or type of object in the augmented reality environment.

[0014] Some embodiments may also include embedding metadata related to selection of an external message for integration in an augmented reality environment, the metadata including selection criteria specifying at least one of information about loaded augmented reality content, features detected in the augmented reality environment, device location, and user demographics.

[0015] In another aspect of the present invention, a device is configured to present an augmented reality environment and includes: an augmented reality application module configured to receive external messages; a camera module configured to receive image information related to a local scene from a device camera; and an augmented reality content integration module configured to create an augmented reality environment by receiving and integrating images from the camera module, main augmented reality content from a local memory, and external messages from the augmented reality application module.

[0016] The augmented reality integration module is configured to integrate the external message into the augmented reality environment based on metadata associated with the corresponding message and including technical standards for integrating the external message into the augmented reality environment. The augmented reality application module is configured to evaluate the completeness of the metadata associated with the presentation of the external message in terms of integration. The augmented reality application module is further configured to, if it is determined that the metadata is incomplete, determine whether the available metadata is sufficient for integration, and

[0017] - if the available metadata is sufficient for integration, integrating the external message into the augmented reality environment based on the available information, and

[0018] - If the available metadata is insufficient for integration, integrating the external message into the augmented reality environment based on a process of surface detection in the augmented reality environment (507) and integration of the external message in conjunction with the detected surface in the augmented reality environment.

[0019] The technical criteria for integration may specify at least one of: the absolute position of the external message in the augmented reality environment, the position of the external message in the augmented reality environment relative to another object in the augmented reality environment, the external message should be positioned on a surface in the augmented reality environment, and the external message should be positioned on or near a specific object or type of object in the augmented reality environment.

[0020] The device may be configured to receive primary augmented reality content from an augmented reality content service and to receive external messages from a messaging service. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention will now be described with reference to the accompanying drawings, in which:

[0022] Figure 1 is an example of an augmented reality scenario with integrated external messaging;

[0023] Figure 2 is a system that can be configured to operate according to the present invention;

[0024] Figure 3 is a block diagram showing an apparatus according to an embodiment of the present invention;

[0025] Figure 4 is a flow chart of a process performed on a device according to an embodiment of the present invention;

[0026] Figure 5 is a flowchart of a process performed on a device when selecting and integrating an external message according to an embodiment of the present invention; and

[0027] Figure 6 is a flow chart of a process performed on a server for preparing an external message for distribution to and integration into an augmented reality environment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In terms of content presentation, augmented reality applications are very complex compared to traditional 2D applications. Some AR applications also provide 2D content, such as labels, grid lines, directions, measurements, etc. superimposed on 2D or 3D views, but 3D AR content is becoming more and more popular. In order to seamlessly integrate 3D AR content into the scene, the AR content should not only be correctly positioned in the view, but also positioned in a way that maintains the 3D aspects of the introduced elements. In the following, 3D elements will include elements rendered in 2D on the display together with a 2D video rendering of the real environment, but rendered in a way that maintains the 3D properties of the elements in terms of perspective, texture, shadows, and other aspects.

[0029] Such 3D AR content is typically experienced by activating a camera on a device such as a mobile phone, detecting the environment (e.g., a simple surface or an object of interest identified by image recognition, etc.) and placing the AR content in the scene based on the detected environmental features. For example, a user may place a 3D dancing character as an AR element in the user's living room and enjoy the experience. Alternatively, a user may place a mobile phone camera in front of her and see AR glasses superimposed on a representation of her face, the glasses being located based on facial recognition software.

[0030] If this presentation is interrupted by an in-app message such as an advertisement, for example, when a promotional video ad within an app interrupts the user while the user is watching a dancing 3D character, the user may have to hold the phone still during the entire video presentation to ensure that the 3D AR content can continue correctly when the presentation ends. If the user moves the phone during the presentation of a 2D video ad while the camera is temporarily deactivated, there is a chance that the environmental data will become corrupted. In addition to this technical interruption, it is inconvenient and potentially annoying for most users to be interrupted while experiencing 3D AR content and have to watch a 2D video.

[0031] The present invention provides methods and systems for integrating AR messages (referred to herein as external AR messages) into other AR content in a non-interruptive manner, but rather allows users to experience external AR messages in an AR format consistent with the primary AR experience, and thereby also allows advertisers and others to create external AR messages that allow themselves to be integrated in an easy manner. Instead of experiencing a 2D video ad that takes up the entire screen or a 2D banner that just covers part of a 3D AR presentation, the user will experience the ad as AR auxiliary content outside of the primary AR content. For example, Figure 1An exemplary AR scene of an augmented environment 101 is shown in which a user is watching a real person 102 dancing with a 3D dancing Santa Claus 103 content (primary AR content). At the same time, an advertisement for soft drinks is presented to the user as a 3D board 104 that displays information related to the soft drinks as auxiliary content on one side of the primary content 103. Thus, the user experiences the external AR message together with the primary AR content 103 in the AR environment 101 in a non-intrusive manner, making the user experience smooth.

[0032] The main module provided by the present invention is an AR application module that is integrated into the AR application by the application developer. The AR application can be developed on a computer 201, which includes the necessary tools for application development and may also include a software development kit (SDK) consistent with the present invention. The SDK may include tools and software modules or libraries that allow the AR application module to be integrated into the main AR application. After the main AR application has been developed, it can be provided to the user, for example, through an application store, and the user can install the AR application on a device such as a mobile phone 202. The main AR application is now able to access the main AR content from the server 203 and integrate the content into the AR environment, for example, integrating Santa Claus 103 into Figure 1 In the AR environment 101.

[0033] As already mentioned, the AR application module is integrated into the main AR application and is therefore installed on the device 202 together with the main AR application. In parallel with the main AR application integrating the main AR content into the AR environment, the AR application module may contact the external AR message repository 204 to request external AR messages. When an external AR message is received, the external AR message may be forwarded by the AR application module to the main AR application via the API, and the main AR application may integrate the external AR message into the AR environment, for example, integrating the soft drink information board 104 into the AR environment. Figure 1 In the AR environment 101.

[0034] Figure 2 The various devices and computers shown may be configured to communicate with each other via a computer network 205, which may be or include a wide area network such as the Internet.

[0035] By providing the necessary APIs and adjusting the requirements for the format and placement of external AR messages, the present invention allows external AR messages such as advertisements and announcements to be easily created, distributed, and displayed to users of any mobile application that has integrated an AR application module.

[0036] It should be noted that this solution is not limited to AR applications that only play predefined AR content (e.g., 3D characters or 2D AR filters). Embodiments of the present invention can also provide external AP messages to AR games and other interactive environments. In addition, external AR messages are not limited to static presentations, but can include active elements and thus allow interaction and invocation of actions.

[0037] Embodiments of the present invention may be configured to handle a variety of AR formats, some of which represent rich content developed with AR in mind. Examples of such content include 3D AR characters (e.g., mascots, logos, cartoon characters, etc.) that perform actions in an AR environment, and dynamic or static 3D AR objects (e.g., hamburgers, soft drink bottles, cars, etc.). The 3D object may also be a 3D board with the appearance of a billboard, a television, etc. The board can receive and display 2D video messages, thereby integrating external messages originally developed for 2D media into a 3D AR environment, in conjunction with the board as the basis for presenting presence. The size, position, and viewing angle of the 2D presentation will be determined by the position of the board in the scene relative to the viewer.

[0038] Other content may be developed for 2D but placed within a 3D AR environment rather than being allowed to take over the entire display when presented. For example, a 2D video could be positioned as a floating image next to another AR object, such as Figure 1 Soft drinks presentation 104 next to Santa Claus 103. This is similar to the billboard described above, except there is no representation of a board or screen providing a rationale for the presence of a 2D image. Instead, the image appears to float in mid-air, but still has a defined position and orientation, and is viewed from an angle determined by the position relative to the user.

[0039] When an AR application uses image recognition to introduce AR elements, 2D AR images or filters, whether dynamic or static, can be used in particular. For example, if an AR application uses face recognition to display filters such as AR sunglasses, the AR application module can display additional sunglasses accessories in AR as 2D images.

[0040] By integrating the AR application module into the main AR application, the main AR application can present the external AR message to the user together with the main AR content. Figure 3 A block diagram showing how embodiments of the present invention may be configured and operated. A main AR application 301 installed on a device 202 includes multiple modules, one of which is an AR application module 302. The main AR application 301 obtains main AR content from a service 303 running on a server 203, while the AR application module 302 obtains external AR messages from a message service 304 running on an external AR message repository 204.

[0041] The main AR application 301 may also include a camera module 305 configured to access, control and acquire images from a camera that is part of the device 202. The images from the camera are provided by the camera module 305 to the AR integration module, which also receives the main AR content that has been provided by the AR content service 303 and the external AR messages provided through the API from the AR application module 302. Communication with the services 303, 304 can be carried out through the network 205.

[0042] The environment and AR content are integrated into an AR environment representation in an AR content integrator 306 , from which the environment and AR content may be provided to an AR renderer 307 , which may be configured to control a display on the device 202 .

[0043] The AR application module 302 includes endpoints that the developer of the primary AR application can use to provide information and callbacks indicating that AR presentation has been initiated and information about how the user will experience the AR content. For example, this can be information about whether the primary AR application will use surface detection, image recognition, etc. In this way, when the user accesses the AR content, the AR application module will know what mode and format the external AR messages should have in order to be displayed correctly with the primary AR content. For example, if surface detection will be used to display the primary AR content (such as a dancing 3D Santa Claus), the AR application module will use the same surface detection information to place 3D external AR messages for the user in a non-intrusive manner.

[0044] Reference now Figure 4 , Figure 4 4 is a flow chart showing the interaction between the main AR application 301, the AR application module 302, and the external AR message service 304. In a first step 401, the main AR application 301 is initialized, and this includes loading the AR application module 302. In some embodiments, the main AR application 301 may also provide the AR application module 302 with information about any AR content loaded from the AR content service 303 or any specific features detected or recognized in the image provided by the camera module 305. Of course, such information may change while the application is running, for example because the user points the camera in a new direction or requests new content from the AR content service 303. In this case, in at least some embodiments of the present invention, the updated information may be provided to the AR application module 302.

[0045] In the next step 402 , the AR application module 302 loads a set of default external AR messages that already exist on the device 202 . This ensures that the AR application module 302 can provide external AR messages even without a connection to the external AR message service 304 .

[0046] In step 403, the AR application module 302 will also send context information to the external AR messaging service 304. This context information may vary in different implementations, but may include the location of the device, demographic information about the user, messaging preferences configured by the user, the user ID of the user (which may be used to access the user profile that the external AR messaging service 304 has made available), information about the primary AR application 301, and information about AR content loaded by the primary AR application 301 or features or objects detected in images acquired from the device camera.

[0047] In step 404, the AR message service compares the received context information with the selection criteria defined for various external AR messages stored in the database 204 of the AR message service, and selects a set of messages having the criteria satisfied by the context information. This set of messages may then be sent to the AR application module 302. The AR application module 302, upon receiving the selected external AR message, adds the received external AR message to the default message and any other locally stored messages that may have been previously downloaded 405, and enables all locally stored external AR messages to be obtained from the integration in the AR environment.

[0048] It should be understood that although Figure 4 The flowchart in shows a step-by-step process of each step one after another for practical reasons, and an embodiment of the present invention running on a physical device does not necessarily operate in such a strict manner. For example, the step of sending context information to the external AR messaging service 304 can be performed regardless of whether the set of default messages has been loaded. Similarly, if the context information changes while the application 301 is running, the context information can be retransmitted, which may result in the external AR messaging service 304 selecting a new message and sending it to the AR application module 302. In this way, the various steps can be implemented as routines that are called when specific events occur, and then these routines are executed until these routines can be completed.

[0049] Now go to Figure 5, a description of the process of selecting a specific external AR message from the local available messages and integrating the message into the AR environment will be given. As described above, the external AR message selected by the external AR message service 304 is selected because the context information meets the criteria of the external AR message. However, some embodiments may include a method for prioritizing the available messages. This shows that in the first step 501, the AR application module selects one of the external AR messages that can be obtained from the local memory, and this can be done based on priority ranking. Prioritization can be based on a scoring system that is associated with the degree of match between the context information and the selection criteria of each external AR message. For example, association with features detected in the scene can give a higher score than device location, and device location can give a higher score than user demographics. Various methods of prioritizing messages are known in the art and will not be discussed further because it is simply a matter of selecting criteria and scoring systems based on perceived importance (or willingness to pay for exposure).

[0050] However, some embodiments may also implement other selection criteria, for example by prioritizing complex 3D AR messages over 2D video or banner presentations, or by prioritizing (or strictly selecting) content based on how it is integrated into the AR environment, for example, based on surface detection or image recognition.

[0051] This illustrates that the selection made in step 501 may include the standard provided as metadata with the external AR message, the type of the external AR message, the local context information and AR environment information provided by the primary AR application 301, and the metadata or data type information provided with the AR content loaded from the AR content service 303. Therefore, various embodiments of the present invention may implement different functions with an API. If the primary AR application 301 is configured to be able to load AR content from the AR content service 303, the AR application module may be configured to provide an endpoint at which information about the loaded AR content can be provided to the AR application module. As described above, the information may be forwarded to the external AR message service 304 and may also be used locally to select one of a plurality of available external AR messages. For example, if a user activates an AR tour guide of the Eiffel Tower in Paris, the metadata may be transmitted to the AR application module 302 (or in some embodiments, the metadata may have been included in the AR application module 302 when the metadata is integrated into the primary AR application 301 using an SDK), and the AR application module 302 will in turn forward the metadata to the external AR service 304. Based on this, the external AR message service 304 may select a group of messages related to restaurants, cafeterias, and bars near the Eiffel Tower, and the AR application module 302 may select an external AR message including a 3D representation of a sandwich promoting a famous sandwich shop near the Eiffel Tower from these messages.

[0052] It should be noted that some embodiments may not implement local storage of a set of messages. In such embodiments, the AR application module 302 may not make a selection. Instead, the external AR message service 304 simply selects a message and sends it to the device. This illustrates that some steps described as being performed by the AR application module 302 are instead performed by the external AR message service 304.

[0053] In some cases, required metadata or contextual information may be missing. This may be the case, for example, if the developer of the primary AR application 301 neglected to include this information when configuring the primary AR application 301, or if metadata is missing from the loaded AR content. In this context, all required information refers to all information that enables the AR application module 302 to select a message having selection criteria satisfied by the contextual information, and all information also includes metadata that enables the AR application module 302 to infer that the primary AR application 301 will be able to correctly integrate the external AR message into the AR environment. This indicates that the message format, method (e.g., surface detection or image recognition), and any other required information (which may vary depending on implementation, content, and context) are compatible with the primary AR application 301 and any loaded content.

[0054] If the AR application module 302 concludes in step 502 that this is the case (i.e., at least one compatible message is available), the process moves to step 503 of selecting a message that meets all requirements. The process then moves to step 504 of forwarding the selected external AR message to the primary AR application 301.

[0055] If it is determined in step 502 that all required information is not available, the process moves to step 505 of selecting a message that meets the available criteria. This means that it is determined in step 502 that some criteria may not be met, that is, there is no available message for which all selection criteria are met by the context information and all technical information is available and compatible. Or some information is missing, in which case it may not be known whether the external AR message is compatible with the primary AR application 301 and other loaded AR content, or it may be determined that some criteria cannot be met. The latter may be because the context has changed so that no available message has all selection criteria met by the current context, or there may be some technical incompatibility between the primary AR application 301 or other AR content loaded by the primary AR application 301 and the available external AR message.

[0056] Since there may be a variety of reasons why the external AR message selected in step 505 does not fully match all criteria, it is determined whether the primary AR application 301 is still able to process the selected message in step 506. This means that the available metadata describing the selected external AR message is sufficient for the AR application module to infer that the AR content integrator module 306 will be able to position the selected external AR message in the AR environment so that the AR renderer 307 can render the scene with the external AR message in a satisfactory manner.

[0057] This determination is made by comparing the available metadata of the selected AR message with available information about the functionality and requirements of the primary AR application, given a context that includes other loaded AR content.

[0058] If it is determined in step 506 that the primary AR application 301 is able to process the selected external AR message, the process moves to step 504 of forwarding the message to the primary AR application 301 .

[0059] However, if it is determined in step 506 that the primary AR application 301 will not be able to process the selected message, the process moves to step 507. In step 507, the AR application module 302 runs its own surface detection function (or, in some embodiments, its own image recognition function) to generate the necessary metadata for integrating the message into the AR environment. The metadata may include positioning information and may also include other information related to rendering.

[0060] In step 508, the generated metadata is embedded into or associated with the selected external AR message to create a message that the primary AR application 301 will be able to process. The process then moves to step 504 where the modified message is forwarded to the AR content integrator 306 in the primary AR application 301.

[0061] When the process reaches step 509, the AR content integration module 306 has received an external AR message that should be integrated into the AR environment and performs the integration. The updated AR environment is then forwarded to the AR rendering module 307, where the updated AR environment is rendered to the display of the device 202.

[0062] The above description does not involve details about how to perform surface detection or image recognition, as these are well-known techniques in the art. Similarly, if all necessary parameters are available, it is well known how to integrate AR elements into an AR environment. In most cases, AR elements are developed so that they can be integrated into an AR environment through software with known functions. The present invention solves such a situation, in which the external AR message is created independently of the software that will integrate and render the external AR message, and the ability to perform such integration can be assigned to an unlimited number of different applications, which may differ greatly in function and purpose. The present invention enables various applications to make their functions known to an integration module, which is partially used as a front end for an external AR content ecosystem so that compatible content can be selected, and partially used as a middleware capable of modifying incompletely compatible AR content so that the AR content becomes fully compatible.

[0063] In this context, full compatibility indicates that the primary AR application 301 is able to integrate and render external messages. The relevance of the external AR messages to the primary AR content may vary, and the degree to which the user experiences the external AR messages as a natural part of the AR environment may also vary.

[0064] As already mentioned, the external AR message service 304 selects the external AR message from the database 204. The database is typically a repository of AR messages that have been developed in this way, and the repository includes the required metadata, which is not only related to the selection criteria, but also to the technical criteria for integration into the AR environment. However, the database 204 may also include messages that have been developed for presentation in a traditional 2D environment, or the AR message service 304 may also communicate with other databases including messages that have been developed for presentation in a traditional 2D environment. Such messages may, for example, be 2D images or 2D videos. Such external messages may also be processed by the present invention. Specifically, the 2D message may be modified by an AR creator module, which may be part of the AR message service 304. The AR creator module may also be implemented on a computer 201, in which external AR message content may be created from other media types.

[0065] The AR creator module is a tool that executes a method of converting media content into AR content (and specifically into external AR messages that can be managed and presented according to aspects of the present invention described above). The module is a combination of software and hardware that is configured to execute a method of receiving media content as input and providing AR content as output.

[0066] Some embodiments of the AR creator module are configured to accept traditional 2D messages, e.g., text, images, or videos, from a repository of messages such as advertisements and announcements, and convert these 2D messages into a format ready for integration into the AR environment as external AR messages. Such a module may be integrated into the AR message service 304 and database 204, or operated in association with the AR message service and database. The module may then be configured to receive traditional 2D messages, convert these 2D messages, and store these 2D messages in a database so that these 2D messages can be selected and provided to the user as described above.

[0067] Whether the AR creator module is configured to operate in a server environment as an automatic converter of non-AR content to content ready for AR integration, or is provided as part of a tool on computer 201 operated by a designer, the specific steps performed are essentially the same. Figure 6 A flow chart is shown that illustrates how the method is performed. Again, it should be understood that the order of steps shown in the figures and described below is selected to facilitate explanation of certain features, but implementations operating on actual devices may perform steps in parallel, include decision points that are not simply binary, etc. Similarly, terms like first, second, and next should be understood to be intended to distinguish different functions rather than to indicate a strict order.

[0068] When a message is received, Figure 6 Implementation shown. In a first step 601, it is determined whether the message is text or contains text. If this is the case, the process moves to step 602 of converting the entire message into an image. If the message is text only, this can be done by virtually rendering the text in a memory and converting the rendered text into an image format (e.g., jpg or png). If the message is an image that also includes text, the text can, for example, be converted into a layer in the image, and the image can then be merged into one layer.

[0069] If the message does not contain text, the process proceeds from step 601 to step 603, which is also the next step after step 602. In step 603, it is determined whether the message is an image. If it is determined that the message is an image (which would be the case if the message was converted into an image in step 602, or if it was transmitted as an image when first received by the AR content creation module), the process moves to step 604. In step 604, the necessary metadata is created. The metadata may include selection criteria and may also include additional information related to AR integration. For example, metadata may indicate that the image should be positioned on a surface detected in the AR environment using surface detection, which metadata may specify size, dimensions, resolution, display duration, and possibly other information. After adding the information required for AR integration, the image may be stored in database 204 as an external AR message in step 608.

[0070] If it is determined in step 603 that the message is not an image, the process moves to step 605 to determine whether the message is a video. If it is determined that the message is a video, the process moves to step 606, which generates and embeds video metadata in a process similar to that for images, except that the metadata may include other parameters.

[0071] If necessary, metadata has been embedded in the message, and the message can be stored in the database 204 in step 608.

[0072] Finally, if it is determined that the message is not a video, the process may move to step 607 of discarding the message because the message cannot be converted into an external AR message.

[0073] The above description does not mention additional media, such as audio or haptics. Of course, the present invention can be modified to be able to process content including images or videos as well as such additional information. Furthermore, while the ability to process text, images, and video (perhaps in combination with audio or haptics) may be sufficient for converting traditional 2D messages, an AR creation module installed as a tool on a computer operated by a designer may include additional functionality, such as the ability to add AR-related metadata to 3D objects.

[0074] Finally, the AR creator module may include certain additional features, such as natural language processing and image recognition. These features may be used to generate additional information from the message, or included as metadata describing the content to modify the selection criteria, or to generate 3D AR content for a richer, more natural user experience. For example, if the message is an image of a running shoe, an image recognition algorithm may identify it as such, and a 3D shoe model may be automatically generated. In this way, the ecosystem can convert at least some 2D or text ads into 3D ads and enhance the user experience.

Claims

1. A method for providing an external message in an augmented reality environment in a computing device, comprising: receiving the external message, the external message comprising metadata related to a desired presentation of the external message in the augmented reality environment; integrating the external message into the augmented reality environment based on a comparison of the characteristics of the augmented reality environment and the metadata associated with a desired presentation of the external message in the augmented reality environment; as well as rendering the augmented reality environment using the external message integrated into the augmented reality environment; in: The metadata related to the presentation of the external message in the augmented reality environment includes a technical standard for integrating the external message into the augmented reality environment; and evaluating the completeness of the metadata associated with the presentation of the external message with respect to the integration (502), and if it is determined that the metadata is incomplete, determining whether the available metadata is sufficient for integration, and - if the available metadata is sufficient for integration, integrating the external message into the augmented reality environment based on the available information, and - If the available metadata is not sufficient for integration, integrating the external message into the augmented reality environment based on a process of surface detection in the augmented reality environment (507) and integration of the external message in combination with the detected surface in the augmented reality environment.

2. The method according to claim 1, wherein: The technical standard for integration specifies at least one of: the absolute position of the external message in the augmented reality environment, the position of the external message in the augmented reality environment relative to another object in the augmented reality environment, the external message should be positioned on a surface in the augmented reality environment, and the external message should be positioned on or near a specific object or type of object in the augmented reality environment.

3. The method according to claim 1 or 2, further comprising sending context information to a remote message server (403), and wherein: The step of receiving the external messages includes receiving a set of messages that have been selected based on a comparison of the sent context information with selection criteria associated with the respective external messages (405), and adding the received messages to a local memory.

4. The method according to claim 3, wherein: The external message is selected from the group of messages based on a priority ranking according to a predetermined scoring system for message prioritization (501).

5. A method according to any one of the preceding claims, wherein: The metadata associated with presentation of the external message in the augmented reality environment includes selection criteria specifying at least one of information about loaded augmented reality content, features detected in the augmented reality environment, device location, and user demographics.

6. A method for generating an external message in a server that can be integrated into an augmented reality environment, comprising: determining whether a message is plain text (601), and if the message is plain text, converting the text into an image (602); determining whether the message is an image (603), and if the message is an image, generating and embedding metadata describing how the image should be integrated into an augmented reality environment (604); determining whether the message is a video (605), and if the message is a video, generating and embedding metadata describing how the image should be integrated into an augmented reality environment (606); as well as The message is stored or sent as an external augmented reality message (608).

7. The method according to claim 6, wherein: The metadata describing how the image should be integrated into the augmented reality environment includes technical standards that specify at least one of: the absolute position of the external message in the augmented reality environment, the position of the external message in the augmented reality environment relative to another object in the augmented reality environment, the external message should be positioned on a surface in the augmented reality environment, and the external message should be positioned on or near a specific object or type of object in the augmented reality environment.

8. The method according to claim 6 or 7 also includes embedding metadata related to the selection of the external message for integration in the augmented reality environment, the metadata including selection criteria, the selection criteria specifying at least one of information about loaded augmented reality content, features detected in the augmented reality environment, device location and user demographics.

9. A device (202) configured to present an augmented reality environment, and comprising: An augmented reality application module (302), configured to receive an external message; a camera module (305) configured to receive image information related to a local scene from a device camera; and an augmented reality content integration module (306) configured to create an augmented reality environment by receiving and integrating images from the camera module (305), primary augmented reality content from a local memory, and external messages from the augmented reality application module (302); in: The augmented reality integration module (306) is configured to integrate the external message into the augmented reality environment based on metadata, the metadata being associated with the corresponding message and including technical standards for integrating the external message into the augmented reality environment; and The augmented reality application module (302) is configured to evaluate the completeness of the metadata associated with the presentation of the external message with respect to integration (502), and if it is determined that the metadata is incomplete, determine whether the available metadata is sufficient for integration, and - if the available metadata is sufficient for integration, integrating the external message into the augmented reality environment based on the available information, and - If the available metadata is not sufficient for integration, integrating the external message into the augmented reality environment based on a process of surface detection in the augmented reality environment (507) and integration of the external message in combination with the detected surface in the augmented reality environment.

10. The device (202) according to claim 9, wherein The technical standard for integration specifies at least one of: the absolute position of the external message in the augmented reality environment, the position of the external message in the augmented reality environment relative to another object in the augmented reality environment, the external message should be positioned on a surface in the augmented reality environment, and the external message should be positioned on or near a specific object or type of object in the augmented reality environment.

11. The device (202) according to claim 9 or 10, further configured to receive primary augmented reality content from an augmented reality content service (303), and to receive external messages from a messaging service (304).

Images