Computer-implemented method, information processing system, and computer program (spatiotemporal relationship-based MR content placement)

Abstract

To provide an approach for analyzing a visual scene corresponding to a location of a user in a combined reality environment including a set of visible objects in a physical world.SOLUTION: The present approach determines one set of spatial relations and one set of temporal relations between a set of visible objects to generate an advertisement based on the determined one set of spatial relations and one set of temporal relations. The present approach overlays an advertisement in a combined reality environment on at least one of the one set of visible objects.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] Virtual reality (VR) is an artificial, computer-generated simulation of a real-world environment that immerses the user in the virtual world, making them feel as if they are actually experiencing the simulated reality. Some virtual reality applications include entertainment and education. Users may capture their experience using multiple cameras aligned to capture a full 360-degree view. [Background technology]

[0002] Today's virtual reality systems use virtual reality headsets or multiple projection environments to generate realistic images, sounds, and other sensations that simulate a user's presence in a virtual environment. A person using a virtual reality device can see, move through, and interact with virtual features or items in a virtual world. Summary of the Invention [Problem to be solved by the invention]

[0003] Mixed reality (MR), also known as hybrid reality, merges the real and virtual worlds to create new environments and visualizations where physical and digital objects coexist and interact in real time. Mixed reality does not operate exclusively in the physical or virtual world, but is a hybrid of real and virtual reality, encompassing both augmented reality and augmented virtual reality. [Means for solving the problem]

[0004] According to one embodiment of the present disclosure, a technique is provided for analyzing a visual scene corresponding to a user's position in a mixed reality environment including a set of visible objects in the physical world, determining a set of spatial relationships and a set of temporal relationships between the set of visible objects, and generating an advertisement based on the determined set of spatial relationships and the set of temporal relationships. The technique overlays the advertisement in the mixed reality environment over at least one of the set of visible objects.

[0005] The foregoing is a summary and, as such, necessarily contains simplifications, generalizations, and omissions of detail; accordingly, those skilled in the art will appreciate that the summary is merely illustrative and is not intended to be in any way limiting. Other aspects, inventive features, and advantages of the present disclosure, as defined solely by the claims, will become apparent in the non-limiting detailed description set forth below. [Brief explanation of the drawings]

[0006] The present disclosure may be better understood, and its numerous objects, features, and advantages made apparent to those skilled in the art by referencing the accompanying drawings. [Figure 1] 1 is a block diagram of a data processing system in which the methods described herein may be implemented. [Figure 2] An extension of the information handling system environment shown in FIG. 1 is provided to illustrate that the methods described herein may be performed on a wide variety of information handling systems operating in a networked environment. [Figure 3] FIG. 1 is an exemplary diagram of placing advertisements on objects in a mixed reality environment based on spatiotemporal analysis. [Figure 4] 1 is an exemplary diagram illustrating a spatiotemporal relationship-based (STRB) ad generation system that identifies spatiotemporal relationships between objects in a scene and builds a spatial relationship graph and temporal relationships. [Figure 5] FIG. 10 is an exemplary diagram illustrating the STRB ad generation system that creates a combined embedding for mapping objects to ads. [Figure 6] 1 is an exemplary flowchart illustrating steps taken to evaluate a user's mixed reality environment and optimize advertisement rendering in the mixed reality environment based on spatiotemporal analysis. [Figure 7] 1 is an exemplary flowchart illustrating steps taken to build spatial and temporal relationship graphs and generate object metadata. [Figure 8] FIG. 1 is an exemplary diagram showing steps taken to create an optimized mixed reality advertisement. DETAILED DESCRIPTION OF THE INVENTION

[0007] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, when used herein, the terms "comprises" or "comprising," or combinations thereof, specify the presence of stated features, integers, steps, operations, elements, or components, or combinations thereof, but are understood not to exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof, or combinations thereof.

[0008] In the following claims, the corresponding structure, material, acts, and equivalents of any means-plus-function or step-plus-function element are intended to include any structure, material, or act for performing the function as specifically claimed in combination with other claimed elements. The description of the present disclosure has been presented for purposes of illustration and description and is not intended to be exhaustive or to limit the disclosure to the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present disclosure. The embodiments have been chosen and described to best explain the principles and practical application of the disclosure and to enable others skilled in the art to understand the disclosure in various embodiments with various modifications to suit the particular use contemplated.

[0009] The present invention may be a system, method, or computer program product, or combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium having computer-readable program instructions for causing a processor to carry out aspects of the present invention.

[0010] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media may also include the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in grooves with instructions recorded on them, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.

[0011] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0012] The computer-readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or may be source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk® or C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions to personalize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the present invention.

[0013] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations or block diagrams or combinations thereof, and combinations of blocks in the flowchart illustrations or block diagrams or combinations thereof, can be implemented by computer-readable program instructions.

[0014] These computer-readable program instructions can be provided to a computer processor or other programmable data processing apparatus to make a machine, such that the instructions, executed by the computer processor or other programmable data processing apparatus, create means for implementing the functions / acts specified in the blocks of the flowcharts or block diagrams, or both. These computer-readable program instructions can also be stored on a computer-readable storage medium that can cause a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium having stored thereon comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in the blocks of the flowcharts or block diagrams, or combination thereof.

[0015] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device and cause the computer, other programmable apparatus, or other device to perform a series of operational steps to produce a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram blocks.

[0016] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, comprising one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be performed as a single step, or may be executed simultaneously, substantially simultaneously, partially, or fully in a time-overlapping manner, depending on the functionality involved, or the blocks may even be executed in the reverse order. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may embody a combination of dedicated hardware and computer instructions. The following detailed description generally follows the summary of the disclosure as set forth above and, where appropriate, further explains and expands upon definitions of various aspects and embodiments of the disclosure.

[0017] 1 illustrates an information handling system 100, a simplified example of a computer system capable of performing the computing operations described herein. Information handling system 100 includes one or more processors 110 coupled to a processor interface bus 112. Processor interface bus 112 connects processor 110 to a northbridge 115, also known as a memory controller hub (MCH). Northbridge 115 connects to system memory 120 and provides a means for processor 110 to access the system memory. A graphics controller 125 also connects to northbridge 115. In one embodiment, a peripheral component interconnect (PCI) express bus 118 connects northbridge 115 to graphics controller 125. Graphics controller 125 connects to a display device 130, such as a computer monitor.

[0018] Northbridge 115 and Southbridge 135 connect to each other using bus 119. In some embodiments, the bus is a Direct Media Interface (DMI) bus, which transfers data in each direction between Northbridge 115 and Southbridge 135 at high speeds. In some embodiments, a PCI bus connects the Northbridge and Southbridge. Southbridge 135 is a chip that implements capabilities, also known as an Input / Output (I / O) Controller Hub (ICH), typically operating at slower speeds than those provided by the Northbridge. Southbridge 135 typically provides various buses used to connect various components. These buses include, for example, PCI and PCI Express buses, ISA buses, System Management Buses (SMBus or SMB), or Low Pin Count (LPC) buses, or combinations thereof. The LPC bus often connects low-bandwidth devices such as boot ROM 196 and "legacy" I / O devices (using "super I / O" chips). The "legacy" I / O devices (198) may include, for example, serial and parallel ports, a keyboard, a mouse, or a floppy disk controller, or a combination thereof. Other components often included in the southbridge 135 include a direct memory access (DMA) controller, a programmable interrupt controller (PIC), and a storage device controller that connects the southbridge 135 to a non-volatile storage device 185, such as a hard disk drive, using bus 184.

[0019] Express card 155 is a slot that connects hot-pluggable devices to the information handling system. Express card 155 connects to southbridge 135 using both a Universal Serial Bus (USB) and a PCI Express bus, thereby supporting both PCI Express and USB connections. Southbridge 135 includes a USB controller 140 that provides USB connectivity to devices that connect to USB. These devices include a webcam (camera) 150, an infrared (IR) receiver 148, a keyboard and trackpad 144, and a Bluetooth® device 146 that provides a wireless personal area network (PAN). USB controller 140 also provides USB connectivity to many other types of USB-connected devices 142, such as a mouse, a removable non-volatile storage device 145, a modem, a network card, an integrated services digital network (ISDN) connector, a fax machine, a printer, a USB hub, and many other types of USB-connected devices. While removable non-volatile storage device 145 is shown as a USB-connected device, removable non-volatile storage device 145 can be connected using a different interface, such as a Firewire® interface.

[0020] Wireless local area network (LAN) devices 175 connect to Southbridge 135 via PCI bus or PCI Express bus 172. LAN devices 175 typically implement one of the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards for wireless modulation techniques, all of which use the same protocol for wireless communication between information handling system 100 and another computer system or device. Optical storage devices 190 connect to Southbridge 135 using Serial Analog Telephone Adapter (ATA) (SATA) bus 188. Serial ATA adapters and devices communicate over a high-speed serial link. The Serial ATA bus also connects Southbridge 135 to other forms of storage devices, such as hard disk drives. Audio circuitry 160, such as a sound card, connects to Southbridge 135 via bus 158. Audio circuitry 160 also provides functionality associated with audio hardware, such as an audio line-in and optical digital audio input port 162, an optical digital output and headphone jack 164, an internal speaker 166, and an internal microphone 168. The Ethernet controller 170 connects to the southbridge 135 using a bus such as a PCI bus or a PCI Express bus. The Ethernet controller 170 connects the information handling system 100 to computer networks such as local area networks (LANs), the Internet, and other public and private computer networks.

[0021] 1 illustrates one information handling system, information handling systems may take many forms. For example, an information handling system may take the form of a desktop, server, portable, laptop, notebook, or other form factor computer or data processing system. Additionally, an information handling system may take other form factors, such as a personal digital assistant (PDA), a gaming device, an automated teller machine (ATM), a mobile phone device, a communications device, or other device that includes a processor and memory.

[0022] FIG. 2 provides an expansion of the information handling system environment shown in FIG. 1 to illustrate that the methods described herein can be performed on a wide variety of information handling systems operating in a networked environment. Types of information handling systems range from small handheld devices, such as handheld computer / cell phone 210, to large mainframe systems, such as mainframe computer 270. Examples of handheld computers 210 include personal entertainment devices, such as personal digital assistants (PDAs), Moving Picture Experts Group Layer-3 audio (MP3) players, portable televisions, and compact disc players. Other examples of information handling systems include pen or tablet computers 220, laptop or notebook computers 230, workstations 240, personal computer systems 250, and servers 260. Other types of information handling systems not individually shown in FIG. 2 are represented by information handling system 280. As shown, various information handling systems can be networked together using computer network 200. Types of computer networks that can be used to interconnect various information handling systems include local area networks (LANs), wireless local area networks (WLANs), the Internet, public switched telephone networks (PSTNs), other wireless networks, and any other network topologies that can be used to interconnect information handling systems. Many information handling systems include a non-volatile data store, such as a hard drive or non-volatile memory, or both. The embodiment of the information handling system shown in FIG. 2 includes a separate non-volatile data store (more specifically, server 260 utilizes non-volatile data store 265, mainframe computer 270 utilizes non-volatile data store 275, and information handling system 280 utilizes non-volatile data store 285). The non-volatile data store can be a component external to the various information handling systems or can be a component internal to one of the information handling systems.Additionally, removable non-volatile storage device 145 can be shared between two or more information handling systems using various techniques, such as connecting removable non-volatile storage device 145 to a USB port or other connector on the information handling systems.

[0023] As mentioned above, mixed reality integrates the real world with the virtual world. To take advantage of the introduction of mixed reality technology, advertising platforms are finding ways to insert advertisements into mixed reality environments. A challenge discovered with existing platforms is that although they have the intelligence to place advertisement content / notification content related to physical objects in a mixed reality environment, they are unable to learn the spatiotemporal relationships between multiple objects and intelligently place advertisements in the mixed reality environment. Furthermore, existing platforms are unable to adjust and optimize mixed reality advertisement / content rendering parameters, such as the shape, size, color, or viewing angle of the advertisement, or a combination thereof.

[0024] 3-8 illustrate techniques that may be implemented on an information processing system for placing mixed reality advertisements on objects within a mixed reality environment based on spatiotemporal relationships between the objects to improve advertisement visibility and increase advertisement rendering space.

[0025] As described herein, a temporal relationship relates to a relationship in time, and a spatial relationship relates to a relationship in space. For example, when a user watches television, the user also sees a remote control in the same time frame (temporal relationship). Therefore, the STRB advertisement generation system 330 identifies the temporal relationship and overlays advertisements on top of the remote control when the user watches television. Furthermore, while the user watches television, the user also sees other objects near the television, such as an entertainment center (spatial relationship). Therefore, the STRB advertisement generation system 330 identifies the spatial relationship and overlays advertisements on top of the surrounding objects when the user watches television.

[0026] 3 is an exemplary diagram of placing advertisements on objects in a mixed reality environment based on spatiotemporal analysis. A spatiotemporal relationship-based (STRB) advertisement generation system 330 provides an approach to placing mixed reality advertisements based on spatiotemporal relationships between objects captured in the mixed reality environment 300. In this embodiment, the STRB advertisement generation system 330 analyzes a visual scene to identify significant objects and their relationships to other objects.

[0027] A user 305 wears a mixed reality device 310 and enters a mixed reality environment 300. The mixed reality environment 300 includes a primary object A, a primary object B, and a secondary object C, each of which is an object in the physical world (e.g., a television, an entertainment center, a remote control, etc.). As the user 305 moves around the mixed reality environment 300, or as the device 310 captures user interactions 315, gaze and body movements 320, and scenes 325, these are provided to a spatiotemporal relationship-based advertisement generation system 330.

[0028] The spatiotemporal relationship graph generator 345 analyzes user interactions 315, gaze and body movements 320, and the scene 325 to generate a spatial relationship graph and a temporal relationship graph. The spatial relationship graph indicates the spatial relationships between objects, and the temporal relationship graph indicates the temporal relationships between objects (see FIG. 4 and corresponding text for further details). The output of the spatiotemporal relationship graph generator 345 is provided to the object instance metadata 340.

[0029] Additionally, a recognition signal analyzer 335 evaluates user interactions 315 and gaze and body movements 320, and the output of the recognition signal analyzer 335 is provided to an object instance metadata computation 340. The object instance metadata computation 340 computes object instance metadata that captures various aspects of visibility details of the mixed reality environment 300 by learning the spatiotemporal relationships between objects. In one embodiment, the object instance metadata includes object shape, object size, object color, relationships with neighboring objects, partial occlusion score, a histogram representing the user depth from the object, user interaction information with the object and neighboring objects, a user visibility score for the object, or temporal relationships between objects, or a combination thereof.

[0030] The output of the object instance metadata operation 340 is provided to a user interaction embedding operation 350. The personalization parameters 355 and user engagement 360 are also provided to the user interaction embedding operation 350. The user interaction embedding operation 350, in one embodiment, computes node embeddings by analyzing local structure in the graph using a graph2vec approach. In one embodiment, the user interaction embedding operation 350 computes a user interaction embedding for each of the objects based on aspects such as object visibility scores, objects with similar spatial and temporal embeddings, the personalization parameters 355 of the user 305, and the distance and viewing angle of the user 305 with respect to the object.

[0031] The output of user interaction embedding computation 350 is provided to advertisement content matching 365, which searches for advertisement content in advertisement store 370 based on the user interaction embedding. The output of advertisement content matching 365 is provided to mixed reality advertisement rendering 395, which renders a mixed reality advertisement based on the viewability score (see FIG. 8 and corresponding text for further details). Figure 3 shows mixed reality advertisements X and Y being overlaid on primary object A and secondary object C, respectively.

[0032] In one embodiment, the STRB ad generation system 330 learns ad rendering parameters to improve ad visibility and maximize ad rendering space. In this embodiment, the STRB ad generation system 330 estimates ad rendering parameters along with viewability scores based on user 305 interactions by solving a joint optimization that improves ad visibility and increases ad rendering space.

[0033] In one embodiment, the STRB ad generation system 330 recalculates ad rendering parameters based on relative changes to a set of objects in the mixed reality environment 300. Ad rendering parameters include scale, position, viewing angle, transparency level, color, light, and shading (see FIG. 8 and corresponding text for further details). The STRB ad generation system 330 identifies the best possible animation for ad content by analyzing the visual scene and the user's 305 interactions in the mixed reality environment 300.

[0034] FIG. 4 is an exemplary diagram illustrating a STRB ad generation system 330 that identifies spatiotemporal relationships between objects in a scene and builds a spatial relationship graph and temporal relationships.

[0035] The spatiotemporal relationship graph generator 345 extracts objects from the scene 325 to generate a spatial relationship graph 410 and a temporal relationship graph 430. The spatial relationship graph 410 maps neighboring objects within a certain range that form a relationship as part of the spatial relationship graph 410. For example, when the user 305 watches television, the user 305 also sees objects near the television. Therefore, the STRB advertisement generation system 330 overlays rendered content / advertisements on top of the objects around the television.

[0036] The temporal relationship graph 430 identifies temporal relationships by learning second order inferences from user interactions 315, or gaze and body movements 320, or both. For example, when a user 305 watches television, the STRB advertisement generation system 330 detects that the user 305 also has a television remote control in view. Therefore, the STRB advertisement generation system 330 overlays rendered content / advertisements on top of the television remote control.

[0037] 5 is an exemplary diagram illustrating the STRB advertisement generation system 330 creating a combined embedding to map an object to an advertisement. In one embodiment, when the STRB advertisement generation system 330 performs the user interaction embedding operation 350, the STRB advertisement generation system 330 analyzes the user's 305 current location in the mixed reality environment 300 and its relationship to primary object A, primary object B, and secondary object C.

[0038] The STRB ad generation system 330 then learns a joint embedding (joint embedding 500) that captures the relationships between primary object A, primary object B, and secondary object C and the ad content representations in ads X, Y, and Z, capturing user interactions in the mixed reality environment 300. In this embodiment, the STRB ad generation system 330 trains a deep learning model by collecting a set of user interactions aligned with mixed reality ad content that captures positive and negative mixed reality ads.

[0039] 6 is an exemplary flowchart illustrating steps taken to evaluate a user's mixed reality environment and optimize advertisement rendering in the mixed reality environment based on spatiotemporal analysis. The processing of FIG. 6 begins at 600, whereupon, at step 610, the process detects a user 305 entering the mixed reality environment 300. At step 620, the process captures a scene in the mixed reality environment 300, and at step 630, the process extracts objects from the captured scene and performs semantic segmentation on the scene to achieve scene understanding.

[0040] In predetermined process 640, the process performs spatiotemporal graph construction and object instance metadata generation (see FIG. 7 and corresponding text for process details). The process determines whether there are any ad vectors and co-entities found in the user's 305 view of the mixed reality environment 300 (decision 650).

[0041] In one embodiment, the decision 650 allows the STRB ad generation system 330 to know when and where to emit AR content for alternative items. By checking the ad vector and co-entity in the view, the STRB ad generation system 330 determines whether and what to display. For example, the ad vector may be anywhere the STRB ad generation system 330 wants to display an ad for pretzels, and the co-entity is a can of soda. In this example, if the can of soda (co-entity) is in the view, the vector for displaying the ad may be on or adjacent to the counter. In another example, a dishwasher may be the co-entity, and the ad vector may be an open dishwasher using liquid detergent. In this example, the ad vector corresponds to a particular slot on the dishwasher for inserting a detergent pack, showing how inserting a detergent pack is easier than filling the slot with liquid.

[0042] If there are advertising vectors and co-entities found within the user's 305 view of the mixed reality environment 300, decision 650 branches to the "Yes" branch, whereupon, at predetermined process 660, the process creates a mixed reality advertisement optimized for the mixed reality environment 300 (see FIG. 8 and corresponding text for processing details). At step 670, the process renders the optimized advertisement on or near objects, which may include primary and secondary objects (co-entities). On the other hand, if there are no advertising vectors and co-entities found within the user's 305 view of the mixed reality environment 300, decision 650 branches to the "No" branch, bypassing steps 660 and 670.

[0043] The process determines whether to continue (decision 680). If the process should continue, decision 680 branches to the "yes" branch and loops back to capture more scenes from the user's 305 view of the mixed reality environment 300. This loop continues until the process should terminate, at which point decision 680 branches to the "no" branch, which exits the loop. Processing of Figure 6 then ends at 695.

[0044] 7 is an exemplary flowchart showing the steps taken to build spatial and temporal relationship graphs and generate object metadata. The processing of FIG. 7 begins at step 700, whereupon at step 710 the process estimates relative positions between extracted objects based on semantic segmentation. At step 720, the process identifies primary objects, such as televisions.

[0045] At step 730, the process builds a spatial relationship graph 410 that identifies secondary objects within a specified distance range to the primary object. At step 740, the process builds a temporal relationship graph 430 that describes the temporal relationships between the user's 305 interactions with one or more primary and secondary objects within a threshold time period.

[0046] At step 750, the process identifies relative locations and relationships between the primary and secondary objects based on the spatial relationship graph 410 and the temporal relationship graph 430. At step 760, the process generates object instance metadata based on the identified relative locations and relationships between the primary and secondary objects (e.g., between a television and a remote control). The process of Figure 7 then returns to the calling routine (see Figure 6) at 795.

[0047] 8 is an exemplary diagram illustrating the steps taken to create an optimized mixed reality advertisement. The processing of FIG. 8 begins at step 800, whereupon at step 820 the process queries the advertisement database store 370 to obtain advertisement content representations to render when the user 305 is near multiple visible objects that are within a certain allowed distance threshold of each other.

[0048] At step 840, the process computes a learned joint embedding that captures the relationship between the visible objects and the advertising content representations, as described herein. At step 860, the process calculates a viewability score by solving a joint optimization configured to improve the visibility of the advertising content and increase the advertising rendering space within the mixed reality environment.

[0049] At step 880, the process computes rendering parameters for the representation of the advertising content within the mixed reality environment based on the user's 305 interactions and the viewability score computed as a function of the combined embedding operation. The processing of Figure 8 then returns to the calling routine (see Figure 6) at 895.

[0050] In one embodiment, the STRB ad generation system 330 identifies a set of ad content by analyzing user interactions, where each of the set of ad content is linked with a set of objects present in the visual scene based on their spatiotemporal relationships. The STRB ad generation system 330 then learns rendering parameters for the ad content by solving a joint optimization using constraints such as: i) identifying the position and size of the ad content i based on the user's implicit feedback; ii) not overlapping the identified ad content in the mixed reality space; iii) identifying a viewing angle for each of the ad content to maximize the ad visual interpretation of the ad content; iv) identifying a binary flag for each of the ad content to support animation; and v) the sum of the content rendering space is within a threshold. As a result, the joint optimization is solved using linear programming by maximizing the content rendering space and maximizing the ad content viewability score.

[0051] While particular embodiments of the present disclosure have been shown and described, it will be apparent to those skilled in the art that, based on the teachings herein, changes and modifications may be made without departing from the present disclosure and its broader aspects. Accordingly, the appended claims are intended to encompass within their scope all such changes and modifications as fall within the true spirit and scope of the present disclosure. It is to be further understood that the present disclosure is defined solely by the appended claims. Where a specific number of introduced claim elements is intended, such intention will be expressly recited in the claims; those skilled in the art will understand that, in the absence of such recitation, no such limitation exists. As a non-limiting example, and to aid in understanding, the following appended claims include the use of the introductory phrases "at least one" and "one or more" to introduce claim elements. However, the use of such phrases should not be construed as suggesting that the introduction of a claim element by the indefinite article "a" or "an" limits any particular claim containing such introduced claim element to a disclosure containing only one such element, even if the same claim also includes the introductory phrase "one or more" or "at least one" and an indefinite article such as "a" or "an," and the same applies to the use of definite articles in a claim.

Claims

1. A method of creating a mixed reality environment, comprising: a computer system analyzing a visual scene corresponding to a user's position within a mixed reality environment, the visual scene including a set of visible objects in the physical world; the computer system determining a set of spatial relationships and a set of temporal relationships between the set of visible objects; generating an advertisement based on the set of spatial relationships and the set of temporal relationships between the set of visible objects; the computer system overlaying the advertisement over at least one of the set of visible objects in the mixed reality environment; A computer-implemented method comprising:

2. The computer system constructs a spatial graph describing the set of spatial relationships, the spatial graph being based on a set of distances in the mixed reality environment between the set of visible objects, including one or more primary objects and one or more secondary objects; the computer system constructing a temporal graph describing the set of temporal relationships, the temporal graph being based on a time frame during which the one or more primary objects and the one or more secondary objects are visible to the user; generating the advertisement for overlaying on one of the one or more secondary objects based on the spatial graph and the temporal graph, The computer-implemented method of claim 1 further comprising:

3. The computer system generating a set of object instance metadata for each of the set of visible objects based on the set of spatial relationships and the set of temporal relationships; determining, based on the set of object instance metadata, that at least two of the set of visible objects are within a first predetermined distance from each other; The computer-implemented method of claim 2 further comprising:

4. The computer system detecting that the user is within a second predetermined distance from the at least two visible objects; generating the advertisement in response to the computer system determining that the at least two visible objects are within a first predetermined distance from one another and detecting that the user is within the second predetermined distance from the at least two visible objects; The computer-implemented method of claim 3 further comprising:

5. The computer system capturing a set of user interactions within the mixed reality environment; the computer system selecting a set of advertising content based on the captured set of user interactions; the computer system computing a binding embedding based on the set of object instance metadata indicating a relationship between the set of visible objects and the set of advertising content; calculating a viewability score in response to solving a joint optimization including a set of constraints to improve visibility of the set of advertising content in the mixed reality environment; the computer system computing a set of rendering parameters for the advertisement within the mixed reality environment based on the combined embeddings and the viewability score; the computer system applying the set of rendering parameters to the set of advertisement content to generate the advertisement; The computer-implemented method of claim 3 further comprising:

6. The computer system solving the joint optimization to increase advertising rendering space within the mixed reality environment and improve the visibility of the set of advertising content within the mixed reality environment. The computer-implemented method of claim 5 further comprising:

7. The computer system recalculating the set of rendering parameters based on one or more relative changes to the set of visible objects, wherein the one or more relative changes are selected from the group consisting of an optimized scale, a user position, and a viewing angle of the advertisement relative to the user. The computer-implemented method of claim 5 further comprising:

8. one or more processors; a memory coupled to at least one of the one or more processors; a set of computer program instructions stored in said memory, an operation of analyzing a visual scene corresponding to a user's position within a mixed reality environment, the visual scene including a set of visible objects in a physical world; determining a set of spatial relationships and a set of temporal relationships between said set of visible objects; generating an advertisement based on the set of spatial relationships and the set of temporal relationships between the set of visible objects; overlaying the advertisement over at least one of the set of visible objects in the mixed reality environment; a set of computer program instructions that are executable by at least one of the one or more processors to execute An information processing system comprising:

9. the one or more processors: constructing a spatial graph describing the set of spatial relationships, the spatial graph being based on a set of distances in the mixed reality environment between the set of visible objects, the set including one or more primary objects and one or more secondary objects; constructing a temporal graph describing the set of temporal relationships, the temporal graph being based on a time frame during which the one or more primary objects and the one or more secondary objects are visible to the user; generating the advertisement for overlaying on one of the one or more secondary objects based on the spatial graph and the temporal graph; 10. The information handling system of claim 8, further comprising:

10. the one or more processors: generating a set of object instance metadata for each of the set of visible objects based on the set of spatial relationships and the set of temporal relationships; determining, based on the set of object instance metadata, that at least two of the set of visible objects are within a first predetermined distance from each other; 10. The information handling system of claim 9, further comprising:

11. the one or more processors: detecting that the user is within a second predetermined distance from the at least two visible objects; generating the advertisement in response to determining that the at least two visible objects are within a first predetermined distance from one another and detecting that the user is within the second predetermined distance from the at least two visible objects; 11. The information handling system of claim 10, further comprising:

12. the one or more processors: capturing a set of user interactions within the mixed reality environment; selecting a set of advertising content based on the captured set of user interactions; computing a combined embedding based on the set of object instance metadata indicating a relationship between the set of visible objects and the set of advertising content; calculating a viewability score in response to solving a joint optimization including a set of constraints to improve visibility of the set of advertising content in the mixed reality environment; computing a set of rendering parameters for the advertisement within the mixed reality environment based on the combined embedding and the viewability score; applying the set of rendering parameters to the set of advertisement content to generate the advertisement; 11. The information handling system of claim 10, further comprising:

13. the one or more processors: solving the joint optimization to increase advertising rendering space within the mixed reality environment and improve the visibility of the set of advertising content within the mixed reality environment.

13. The information handling system of claim 12, further comprising:

14. the one or more processors: recalculating the set of rendering parameters based on one or more relative changes to the set of visible objects, the one or more relative changes selected from the group consisting of an optimized scale, a user position, and a viewing angle of the advertisement relative to the user; 13. The information handling system of claim 12, further comprising:

15. A computer program, when executed by an information processing system, causing the information processing system to: an operation of analyzing a visual scene corresponding to a user's position within a mixed reality environment, the visual scene including a set of visible objects in a physical world; determining a set of spatial relationships and a set of temporal relationships between said set of visible objects; generating an advertisement based on the set of spatial relationships and the set of temporal relationships between the set of visible objects; overlaying the advertisement over at least one of the set of visible objects in the mixed reality environment; A computer program comprising computer program code for causing the computer to perform operations including:

16. The information processing system, constructing a spatial graph describing the set of spatial relationships, the spatial graph being based on a set of distances in the mixed reality environment between the set of visible objects, the set including one or more primary objects and one or more secondary objects; constructing a temporal graph describing the set of temporal relationships, the temporal graph being based on a time frame during which the one or more primary objects and the one or more secondary objects are visible to the user; generating the advertisement for overlaying on one of the one or more secondary objects based on the spatial graph and the temporal graph; 16. The computer program of claim 15, further comprising:

17. The information processing system, generating a set of object instance metadata for each of the set of visible objects based on the set of spatial relationships and the set of temporal relationships; determining, based on the set of object instance metadata, that at least two of the set of visible objects are within a first predetermined distance from each other; 17. The computer program of claim 16, further comprising:

18. The information processing system, detecting that the user is within a second predetermined distance from the at least two visible objects; generating the advertisement in response to determining that the at least two visible objects are within a first predetermined distance from one another and detecting that the user is within the second predetermined distance from the at least two visible objects; 20. The computer program of claim 17, further comprising:

19. The information processing system, capturing a set of user interactions within the mixed reality environment; selecting a set of advertising content based on the captured set of user interactions; computing a combined embedding based on the set of object instance metadata indicating a relationship between the set of visible objects and the set of advertising content; calculating a viewability score in response to solving a joint optimization including a set of constraints to improve visibility of the set of advertising content in the mixed reality environment; computing a set of rendering parameters for the advertisement within the mixed reality environment based on the combined embedding and the viewability score; applying the set of rendering parameters to the set of advertisement content to generate the advertisement; 20. The computer program of claim 17, further comprising:

20. The information processing system, solving the joint optimization to increase advertising rendering space within the mixed reality environment and improve the visibility of the set of advertising content within the mixed reality environment.

20. The computer program of claim 19, further comprising:

Images