Visibility of Objects in a Virtual Environment

Abstract

A method, system, and apparatus for determining the visibility of an object by a user of a virtual environment, including a computer program encoded on a computer storage medium, the method, system, and apparatus including: capturing a two-dimensional projection of an object presented in the virtual environment; determining that the two-dimensional projection of the object matches a reference version of the object based on a comparison of an average color of features of the reference version of the object and an average color of features in the two-dimensional projection of the object; and classifying the presentation of the object in the virtual environment based on whether the two-dimensional projection of the object matches the reference version of the object.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of priority based on U.S. Provisional Application No. 63 / 443,171, filed on February 3, 2023, which is hereby incorporated by reference in its entirety.

[0002] This specification relates to virtual computing environments.

Background Art

[0003] Virtual environments in game environments, metaverses, and multiverses can include content rendered across the virtual environment, but the content may not be visible for various reasons. For example, content rendered in a virtual environment may not be presented until the user adjusts the user's field of view to the location of the rendered content within the virtual environment. Also, content rendered within the virtual environment may be blocked, for example, by one or more other objects within the virtual environment, preventing the user from seeing the rendered content.

Summary of the Invention

[0004] This specification describes techniques for determining whether content presented in a computing environment within a user's field of view is blocked, impaired, or otherwise not considered visible.

[0005] These techniques generally include an object visibility pipeline for determining whether an object (e.g., publisher content) presented to a user interacting with a computing environment (e.g., a dynamic gaming environment, a two-dimensional, three-dimensional, augmented reality, virtual reality environment) is visible to the user. Based on the results of the object visibility pipeline, the rendered content can be classified as visible or not visible (e.g., occluded or otherwise considered invisible) in the computing environment.

[0006] Generally, one innovative aspect of the subject matter described herein can be embodied in a method that includes the actions of determining that the presentation of an object within a virtual environment meets a set of visibility conditions, capturing a two-dimensional projection of the object presented in the virtual environment, determining that the two-dimensional projection of the object matches a reference version of the object based on a comparison of the average color of the features of the reference version of the object and the average color of the features in the two-dimensional projection of the object, and classifying the presentation of the object within the virtual environment based on whether the two-dimensional projection of the object matches the reference version of the object. In response to determining that the two-dimensional projection of the object matches the reference version of the object, classify the presentation of the object within the virtual environment as a visible rendering of the object. In response to determining that the two-dimensional projection of the object does not match the reference version of the object, classify the presentation of the object within the virtual environment as a non-visible rendering of the object.

[0007] Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0008] The above and other embodiments can each include one or more of the following features, either alone or in combination. Specifically, one embodiment includes all of the following features in combination. In some embodiments, the set of visibility conditions includes verifying, from one or more processors, the rendering verification of objects in the virtual environment.

[0009] In some embodiments, the set of visibility conditions includes determining that the viewing angle of an object within the user's field of view in the virtual environment meets the criteria of a threshold angle with respect to the surface normal of the object in the virtual environment relative to the user's field of view.

[0010] In some embodiments, the set of visibility conditions includes determining that object pixels including the object include coordinates that coincide with the user's field of view in the virtual environment.

[0011] In some embodiments, the set of visibility conditions includes determining whether a transparency threshold is met for one or more features of an object within the user's field of view, for example, from the user's viewpoint. One or more features of the object can include features of at least one corner of the object and the central feature of the object.

[0012] In some embodiments, the set of visibility conditions includes verifying that the dimensionality of the object meets a threshold dimensionality. Verifying the dimensionality includes determining that the pixel ratio of object pixels to pixels on the screen meets a threshold, and determining a threshold number of object pixels including the pixels on the screen.

[0013] In some embodiments, determining that a set of visibility conditions is met includes determining that the average luminance of an object meets a luminance threshold. Determining that the average luminance of an object meets a luminance threshold may include calculating the average luminance of pixels that include the object, converting the average luminance to a representative value, and comparing the representative value to the value of the luminance threshold.

[0014] In some embodiments, classifying a presentation of an object in a virtual environment further includes incrementing a count of the object's visibility in response to classifying the presentation of the object in the virtual environment as a visible rendering of the object, determining that a number of a series of increments of the count of the object's visibility meets a visibility count threshold, and registering the presentation of the object.

[0015] In some embodiments, classifying a presentation of an object in a virtual environment further includes incrementing a count of the object's non-visibility in response to classifying the presentation of the object in the virtual environment as a non-visible rendering of the object, determining that a number of a series of increments of the count of the object's non-visibility meets a non-visibility count threshold, and providing an alert regarding the object's non-visibility.

[0016] In some embodiments, determining that a two-dimensional projection of an object matches a reference version of the object includes calculating a hash of the two-dimensional projection and comparing the hash of the two-dimensional projection to a hash of the reference version of the object. Calculating the hash of the two-dimensional projection and the hash of the reference version of the object can include calculating an average hash. Calculating an average hash can include calculating values of average colors of at least a portion of the two-dimensional projection, encoding each pixel of the two-dimensional projection based on whether the color value of the pixel is at least the value of the average color, creating a bit string based on the encoded pixels, and converting the bit string to a hexadecimal value.

[0017] In some embodiments, determining that a two-dimensional projection of an object matches a reference version of the object includes determining a difference between a hexadecimal value and a reference hexadecimal value representing the reference version of the object.

[0018] In some embodiments, determining that a two-dimensional projection of an object matches a reference version of the object includes identifying the positions of a set of edges in the reference version of the object, searching for the positions of the set of edges in the two-dimensional projection, and comparing the average color of the pixels at the positions of the edges in the two-dimensional projection with the average color of the pixels at the positions of the edges in the reference version of the object.

[0019] The subject matter described in this specification can be implemented to achieve one or more of the following advantages. An object visibility pipeline that includes a series of sequential checks for object visibility can reduce the processing resources required to determine visibility by having each visibility check verified before the system can proceed to the next visibility check. For example, if a visibility condition is not met, the system may not proceed to the next verification step of the set of visibility conditions, thereby reducing the computational requirements for verifying the object's visibility. The object visibility pipeline can be used as a lightweight object visibility process that can be executed by an edge device with computational and / or power limitations, such as a mobile device operating on a battery. In these situations, reducing the processing resources required to make a visibility determination reduces battery consumption, and thus increases the amount of time the device can operate on a single battery charge. Further, since the processing capabilities of user devices are limited, reducing the processing resources required to make a visibility determination prevents adverse effects by diverting processing resources from rendering and presenting a virtual three-dimensional environment. For example, using fewer resource-intensive visibility determinations, such as those described herein, helps prevent game glitches or delays that could both potentially render the game unplayable. The lightweight visibility determination process includes hash techniques or feature detection techniques (e.g., edge detection techniques), which are described in detail below. These two techniques enable the determination of visibility on a user device without interrupting or unduly affecting the rendering or presentation of a three-dimensional environment on the user device.

[0020] By verifying the visibility of objects, important feedback on the effectiveness of content embedded in a virtual environment, such as presentation registration, can be provided to the content publisher. For example, the presentation of embedded content in a three-dimensional virtual game or experience environment can be verified more efficiently and accurately. The object visibility pipeline can be integrated into dynamic game environments, such as three-dimensional VR / AR experiences, without substantially affecting the game environment. The user's field of view can constantly change in a dynamic game environment, so content may enter and exit the user's field of view, and in such situations, the pipeline can more accurately track the presentation of publisher content by the user. Furthermore, the object visibility pipeline includes checks to determine whether the characteristics of the presented objects are maintained in the environment, thus not only enabling the content publisher to verify the accurate presentation of content, but also allowing developers to use it to improve the environment during the development cycle.

[0021] Details of one or more embodiments of the subject matter of this specification are described in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

Brief Description of the Drawings

[0022]

Figure 1A

Figure 1B

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

DETAILED DESCRIPTION OF THE INVENTION

[0023] Like reference symbols and designations in the various drawings refer to like elements.

[0024] In some cases, the content can be presented to a user in a virtual environment or an augmented reality environment (e.g., a two - dimensional environment or a three - dimensional environment) in a non - destructive manner without interrupting the user's immersive experience while providing means for presenting the content (e.g., third - party publisher content) to the user within the computing environment. The presented content can be, for example, a two - dimensional or three - dimensional object rendered within the computing environment. The user's field of view within the computing environment can change over time, for example, as the user moves relative to the computing environment. Further, the computing environment can include one or more other objects rendered within the computing environment within the user's field of view, potentially obscuring (e.g., partially obscuring) the presented content. The visibility of the content to the user within the user's field of view can be determined using an object visibility pipeline that can determine the visibility of the presented content by the user over a measured period.

[0025] In some embodiments, by tracking the visibility of the presented content over a period of time, a classification of the presented content that is perceptible to a user within a computing environment is triggered, which may be referred to as an effective presentation, e.g., an effective impression. For the presentation of the presented content, additional metrics related to the presented content, such as content exposure time, interaction of the user with the content, etc., may be additionally tracked.

[0026] As described in more detail below, whether the presentation of the content is classified as being perceptible by the user may depend on "unscripted" actions within a virtual environment (e.g., a two-dimensional environment, a three-dimensional environment, or other types of immersive experiences). For example, if a user crashes a car in a racing game and the smoke from the fire makes it difficult to perceive a set of content presented on a building wall (or an advertisement board in a three-dimensional environment), the set of content may be considered imperceptible even though the smoke is not completely opaque and is defined to have an attribute (e.g., a physical collider or other attribute) that allows the smoke to be directly identified as an obstructive one with respect to the visibility of the object. Due to the dynamic nature of the three-dimensional environment (e.g., changing dynamically in response to the user's actions) and the fact that scene objects with undefined detectable attributes can significantly impair the perceptibility of a set of content, techniques may be used to determine the level of degradation of the presentation of a set of content by scene objects (e.g., smoke, clouds, spirits, etc.). As described in more detail below, these techniques can include one or more of hash techniques or feature detection / matching techniques such as edge detection, which can be used to quantify the level of degradation of the presentation of a set of content relative to a reference version of the set of content. It should be noted that the techniques described herein are described for use in a three-dimensional environment, but these or similar techniques can also be implemented in a two-dimensional environment.

[0027] FIG. 1A is a block diagram of an exemplary framework 100 in which third - party content is delivered for presentation using virtual two - dimensional or three - dimensional objects in a virtual environment such as a virtual reality environment. The virtual environment can be a two - dimensional (2D) environment, a three - dimensional (3D) environment, or other types of immersive interactive environments including augmented reality and virtual reality environments. The exemplary framework 100 includes a network 102 such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 102 connects an application server 104, a user device 106, a third - party content server 108, and a third - party content delivery system 110 (also referred to as a content delivery system). The exemplary framework 100 can include many different application servers 104, user devices 106, and third - party content servers 108.

[0028] The user device 106 is an electronic device that can request and receive resources (e.g., virtual environment applications) via the network 102. Exemplary user devices 106 include personal computers, mobile communication devices, and other devices that can send and receive data via the network 102. The user device 106 typically includes a user application such as a web browser to facilitate the sending and receiving of data via the network 102, but native applications executed by the user device 106 can also facilitate the sending and receiving of data via the network 102.

[0029] A resource (e.g., a virtual environment application or a definition file of a virtual environment) is a resource for rendering a virtual environment that may include text, images, videos, or other media types on the user device 106. Examples of resources include virtual reality applications, video games, mixed reality applications, augmented reality applications, and definitions of virtual environments that can be displayed in any of these types of applications. A resource may include data that defines one or more virtual environments and virtual objects within the virtual environment. A resource can include data that defines virtual objects such as two-dimensional objects or three-dimensional objects for presentation within the virtual environment. A resource can be provided to the user device 106 by the application server 104. For example, the application server 104 may include a server that hosts a publisher's website. In this example, the user device 106 can initiate a request for a given resource, and the application server 104 that hosts the given resource can respond to the request by sending the resource to the user device 106. In some embodiments, the application server can provide one or more definition files to the user device 106. The definition file includes data representing a virtual environment that can be processed by an application installed on the user device 106 for rendering the virtual environment.

[0030] In some situations, a given resource can include third - party tags or third - party scripts that reference the third - party content delivery system 110. In these situations, when the given resource is processed by the user device 106, the third - party tags or third - party scripts are executed by the user device 106. Execution of the third - party tags or third - party scripts causes the user device 106 to be configured to generate a request 112 for third - party content (e.g., content that is not defined within the resource but is obtained from a third - party and inserted into the resource) to be sent to the third - party content delivery system 110 via the network 102. For example, the third - party tags or third - party scripts may enable the user device 106 to generate a packetized data request that includes header and payload data. The request 112 can include data such as the name (or network location) of the server for which the third - party content is requested, the name (or network location) of the device making the request (e.g., the user device 106), and / or information that the third - party content delivery system 110 can use to select the third - party content to be provided in response to the request. The request 112 is sent by the user device 106 via the network 102 (e.g., a telecommunications network) to the server of the third - party content delivery system 110.

[0031] Requirement 112 can include data specifying a resource, data specifying characteristics of a virtual object (e.g., a two-dimensional or three-dimensional virtual object) on which third-party content is presented, and data specifying characteristics of the virtual environment in which the virtual object is generated. For example, data specifying the shape or geometry of a virtual object (e.g., a two-dimensional or three-dimensional virtual object) on which third-party content is presented, the size of the virtual object (e.g., length, width, height, and / or volume), the location of the virtual object within the virtual environment, the number of eligible surfaces of the virtual object on which third-party content can be received, descriptive keywords related to the virtual environment, and / or data specifying the media type eligible for presentation in the virtual object can be provided to the content delivery system 110.

[0032] Requirement 112 can also include data related to other information, such as information provided by the user, geographical information indicating the state or region where the request is submitted, or other information providing context about the environment in which the third-party content is displayed. Requirement 112 can also provide data specifying characteristics of the user device 106, such as information identifying the model of the user device 106, the selection capabilities of the device 106 (e.g., whether hand-based controls for selecting virtual objects are available, whether controls on the headset itself that allow the user to tap and select items rendered in the virtual reality environment are available), the configuration of the user device 106, the type of electronic display (e.g., a touch screen of a smartphone, tablet, game device, or a head-mounted display of the VR device 106). Requirement 112 can be transmitted, for example, via a packetized network, and Requirement 112 itself can be formatted as packetized data having a header and payload data. The header can specify the destination of the packet, and the payload data can include any of the above information.

[0033] In response to receiving request 112 and / or using the information included in request 112, third-party content delivery system 110 selects third-party content to be presented on or near a virtual object within a virtual environment.

[0034] In some embodiments, the delivery parameters (e.g., selection criteria) for certain third-party content may include delivery keywords (e.g., resources or terms specified in request 112, etc.) that need to match for the third-party content to be eligible for presentation. The delivery parameters may also require that request 112 includes information specifying a particular geographic region (e.g., country or state) and / or information specifying that request 112 was originated from a particular type of user device 106 for the third-party content to be eligible for presentation. The delivery parameters can also specify the value-added and / or budget for delivering a particular third-party content.

[0035] The identification of eligible third-party content can be segmented into a plurality of tasks 117a - 117c, and then the tasks are assigned among the computing devices within a set of multiple computing devices 114. For example, different computing devices 114 within the set can each analyze a different portion of the third-party corpus database 116 to identify various third-party content having distribution parameters that match the information included in request 112. In some embodiments, each given computing device 114 within the set can analyze different data dimensions (or sets of dimensions) and return the results (Res1 - Res3) 118a - 118c of the analysis to the third-party content delivery system 110. For example, the results 118a - 118c provided by each of the computing devices within the set can identify a subset of third-party content that is eligible for delivery in response to the request and / or a subset of third-party content having particular delivery parameters or attributes.

[0036] The third - party content delivery system 110 aggregates the results 118a - 118c received from a set of multiple computing devices 114 and uses the information associated with the aggregation result to select one or more instances of third - party content provided in response to the request 112. For example, the third - party content delivery system 110 can select a set of acquired third - party content based on the results of one or more content evaluation processes, as discussed in more detail below. Next, the third - party content delivery system 110 can generate and transmit, via the network 102, reply data 120 (e.g., digital data representing a reply) that enables the user device 106 to integrate the set of acquired third - party content into a virtual environment for presentation to a qualified virtual object within the virtual environment, for example.

[0037] In some embodiments, the user device 106 executes the instructions included in the reply data 120, whereby the user device 106 is configured to and enabled to obtain the set of acquired third - party content from one or more third - party content servers. For example, the instructions within the reply data 120 can include a network location (e.g., a Uniform Resource Locator (URL)) and a script for the user device 106 to send a third - party request (3PR) 121 to the third - party content server 108 to obtain a given acquired third - party content from the third - party content server 108. In response to this request, the third - party content server 108 transmits third - party data (TP data) 122 to the user device 106 to incorporate the given acquired third - party content into the virtual environment and cause the user device 106 to present it.

[0038] Figure 1B represents a block diagram of an exemplary client computing system 150 configured to render a virtual environment, e.g., a two - dimensional or three - dimensional virtual environment, showing third - party content specified by the content delivery system 152. In some embodiments, the client computing system 150 is a user device, e.g., the user device 106 from FIG. 1. The content delivery system 152 can be configured as the third - party content delivery system 110 from FIG. 1, the third - party content server 108 from FIG. 1, or can include aspects of both the server 108 and the system 110. The content delivery system 152 can generally be implemented as a system of one or more computers at one or more locations. The client computing system 150 communicates with the content delivery system 152 via a network (e.g., the Internet, a local area network, a wireless broadband network). Although not shown in FIG. 1B, the client computing system 150 can communicate with other systems in addition to the content delivery system 152 for various purposes. For example, the client computing system 150 can communicate with a server of an online application store or a developer server to obtain virtual reality, augmented reality, and / or mixed reality applications that enable the system 150 to render a virtual environment. Similarly, the client computing system 150 can communicate with a server of an online application store or a developer server to obtain a definition file for a virtual environment, e.g., for an immersive virtual reality game.

[0039] The client computing system 150 can be any of a variety of computing systems and / or game devices configured to render a virtual environment incorporating third-party content and be renderable. In some examples, the client computing system 150 is configured to present a virtual reality type of virtual environment that a user views via a head-mounted display. In other examples, the client computing system 150 is configured to present other types of virtual environments, such as an augmented reality environment, a mixed reality environment, or a conventional two-dimensional screen environment. The system 150 may be integrated within one device or may include a plurality of separately connected components located at one or more locations. In some embodiments, the client computing system 150 includes a display 154, a memory subsystem 156, a virtual environment rendering engine 158, an input handler 160, a content manager 162, and a network interface 164.

[0040] The display 154 is an electronic display configured to visually present a virtual environment to a user. The display 154 can take various forms for different types of systems. For example, the display 154 can include a head-mounted display, a mobile device, a tablet, a television, or a display of a game console, or other displays through which a user can view a virtual environment. As shown in FIG. 1B, the display 154 can be, for example, a head-mounted display or a display (e.g., a screen) of a mobile device. For example, in a virtual reality system, the display 154 can be a head-mounted display in which the display screen of the display 154 is fixed at a position a few inches in front of the user's eyes. In a VR system, the display 154 can provide a stereoscopic presentation of a virtual environment, such as a three-dimensional virtual environment. When the user views the stereoscopic presentation of the virtual environment through a set of lenses, the virtual environment appears to have depth, so that the user experiences a sense of immersion in the virtual environment. In some embodiments, the screen is an integral component of the head-mounted display. In other embodiments, a smartphone or other mobile unit is removably fixed to the head unit to form a head-mounted display that uses the screen of the mobile unit as the screen of the head-mounted display. The display 154 can be, for example, a liquid crystal display (LCD), an organic light emitting diode display (OLED), or an active matrix OLED (AMOLED) display.

[0041] Memory subsystem 156 includes one or more storage devices that store data characterizing a virtual environment. The virtual environment is a virtual environment that can be rendered in three dimensions. Examples of virtual environments include 3D games and video environments (e.g., live or recorded event streams such as a 3D concert or a sports event stream). In some cases, a user of client computing system 150 can explore the virtual environment by moving their head to look around the environment (e.g., in a virtual reality system), by moving through the environment, by manipulating objects in the environment, or by combinations thereof. Other components of client computing system 150 can access memory subsystem 156 to read, write, or delete data from the storage devices.

[0042] In some embodiments, the data stored by memory subsystem 156 that characterizes the virtual environment includes declarations of third-party content. Third-party content, such as virtual objects, can be declared with respect to the virtual environment using any of a variety of suitable programming techniques. In some embodiments, a developer can insert tags, scripts, or executable code into a definition file(s) for the virtual environment, which when executed, instantiates objects in the virtual environment, e.g., two-dimensional or three-dimensional objects, according to any parameters specified herein.

[0043] The virtual environment rendering engine 158 is a subsystem of the client computing system 150. It reads the definition of the virtual environment from the memory subsystem 156 and renders a virtual environment for presentation to the user via the display 154, and is optionally configured to use one or more additional peripheral output devices (e.g., speakers, hand controllers, haptic feedback devices). The rendering engine 158 is configured to perform the operations described herein and can include one or more data processing devices (e.g., processors) that are executable. The data processing device may be dedicated to the rendering engine 158 or may be at least partially shared with other components of the client computing system 150. In some embodiments, the rendering engine 158 includes one or more graphics processing units (GPUs) that process the virtual environment definition file and render the presentation of the environment. For example, the rendering engine 158 of a virtual reality system can process one or more definition files of the virtual environment to generate a stereoscopic display of the virtual environment, which provides an immersive 3D experience to the user when viewed by the user through specially configured lenses.

[0044] Input handler 160 is a subsystem of client computing system 150 configured to monitor one or more input channels for user input received while the virtual environment is being rendered for the user. Input handler 160 is configured to perform the operations described herein and can include one or more data processing devices (e.g., processors) that are executable. Input handler 160 can detect various types of user input depending on the particular configuration of client computing system 150. For example, a basic virtual reality (VR) system can detect user input based on signals from one or more orientation sensors and motion sensors of a head-mounted display unit. The orientation sensors and motion sensors can include one or more accelerometers, compasses, gyroscopes, magnetometers, or combinations of such sensors. The orientation sensors and motion sensors can generate signals that indicate in real time the direction of the user's line of sight within the 3D VR environment, and these signals can be interpreted by input handler 160 to track the direction of the user's line of sight in real time. Tracking the direction of the user's line of sight in real time can be used, for example, to determine the user's field of view within the virtual environment and to define a viewport of the visible portion of the virtual environment by the user. Further, client computing system 150 can include one or more buttons or switches that can be actuated by the user to provide input to system 150, for example, in a hand-based controller or a head-mounted display. More advanced VR systems can provide additional user input channels such as motion tracking sensors located external to the head-mounted display that track the reference movement of the head-mounted display. Input handler 160 can interpret signals from the external motion sensors to determine the user's movement in six degrees of freedom, including, for example, rotation and translation.

[0045] In some embodiments, system 150 includes a content manager 162 for monitoring third-party content provided within a virtual environment. The content manager 162 can be a subsystem of the system 150 that manages content (e.g., virtual objects) displayed in the virtual environment. The content manager 162 can be implemented as one or more data processing devices (e.g., processors) at one or more locations programmed to perform the operations described herein. The data processing device can be dedicated to the content manager 162 or can be shared with one or more other components of the system 150. For example, the data processing device can include a central processing unit (CPU) and / or a graphics processing unit (GPU) of a client device.

[0046] In some embodiments, the content manager 162 receives information related to the user's current field of view from the input handler 160 through the display 154 and is configured to determine, as will be described in more detail with reference to FIGS. 3 and 4, that the presentation of an object within the virtual environment meets the visibility condition(s). The content manager 162 can further classify the presentation of the object, for example, as a visible rendering of the object or as a non-visible rendering of the object, and can be configured to provide information related to the classified presentation to the content delivery system 152. For example, the content manager 162 can provide the content delivery system 152 with confirmation of a user interaction with the presentation of the object.

[0047] Client computing system 150 sends messages to and receives messages from content delivery system 152. Content delivery system 152 may be implemented as one or more computers (e.g., data processing devices) at one or more locations. Generally, content delivery system 152 is configured to select third-party content to be displayed within a virtual environment on client computing system 150. Content delivery system 152 enables client computing system 150 to utilize the selected third-party content by sending the content to client system 150 via a network such as, for example, the Internet or a local area network. Content delivery system 152 can include one or more of front-end server 166, third-party content database 168, content selector 170, a data repository storing selection criteria 172, a second data repository 174 storing end-user account and profile information, and a third data repository 176 storing third-party content provider account and profile information.

[0048] The front - end server 166 is configured to receive and transmit information from the content delivery system 152. The front - end server 166 provides an interface for the content delivery system 152 to interact with other computers via a communication network (e.g., the Internet). For example, FIG. 1B shows a front - end server 166 communicating with a client computing system 150. The front - end server 166 receives requests for third - party content, performs initial processing of the received requests, transfers the information obtained from the requests to other appropriate components of the content delivery system 152, and transmits responses generated by the system 150 in response to the requests. In some embodiments, the front - end server 166 includes a network interface that provides an interconnection between the content delivery system 152 and one or more networks, which may be either public (e.g., the Internet) or private (e.g., a local area network). The network interface may include one or more network interface cards, which are configured to transmit and receive data, for example, via a packet - based network.

[0049] The content database 168 is a database that maintains an index of third - party content or another type of data repository. Also, the third - party content itself may be stored by the content database 168, by the content delivery system 152 but outside the content database 168, or in one or more other systems outside the content delivery system 152. Generally, the content database 168 identifies a set of third - party content that is available for the content delivery system 152 to return to a client system in response to requests for third - party content presented, for example, within a virtual environment.

[0050] Content selector 170 is a component of content delivery system 152 and selects third-party content obtained in response to a request, such as content to be displayed within a virtual environment. To determine the third-party content to be obtained, content selector 170 evaluates eligible third-party content items with respect to various selection criteria 172 related to the request. The selection criteria may include keywords or other context data specified in the request. In some embodiments, the selection criteria further include profile data indicating the interests and preferences of the end user of client system 150, profile data of third-party content providers, and information regarding the virtual environment in which the virtual object is presented. Selection criteria 172 may further include an attachment value presented by the third-party content provider indicating a price that the third-party content provider may pay for the third-party content selected and returned for display on or near the virtual object in response to the request. Content selector 170 applies selection criteria 172 to a given third-party content request and performs an evaluation process to select the third-party content to be obtained.

[0051] FIG. 2 is an exemplary operating environment 200 for object presentation within a virtual environment. A client computing system, such as client computing system 150, can render a virtual environment on a display, such as display 154. The virtual environment may be rendered by a rendering engine, such as virtual environment rendering engine 158 of client computing system 150. The virtual environment can include third-party content that can be rendered as two-dimensional or three-dimensional objects within the virtual environment, such as third-party content provided by content selector 170. As used herein, the field of view 202 within virtual environment 204 is defined from the perspective of user 206 within virtual environment 204. Sometimes, the field of view 202 of user 206 of virtual environment 204 can change as the user moves relative to the virtual environment. For example, when the user shifts their line of sight within the virtual environment or moves their virtual representation within the virtual environment. In some embodiments, when the user's field of view shifts within the virtual environment, an object 208 within virtual environment 204, such as publisher content, can move outside of the user's field of view 202. In other words, the viewport 210 corresponding to the display on the screen for user 206 can move relative to object 208, such that at least a portion of object 208 may move outside of viewport 210.

[0052] In some embodiments, one or more other objects, such as objects 212, 214, are rendered within the field of view of user 206 within virtual environment 204. One or more other objects 212, 214 can be positioned within the virtual environment such that at least one of objects 212, 214 at least partially obscures the user 206's view of object 208. For example, object 212 obscures a portion of object 208 from the view of user 206 within viewport 210.

[0053] In some embodiments, a content manager 162 of a system, such as a client computing system 150, can determine whether an object rendered within a virtual environment 204 meets visibility conditions.

[0054] FIG. 3 is a flowchart of an exemplary process 300 for determining the visibility of an object by a user of a virtual environment. For convenience, process 300 is described as being performed by one or more computer systems located at one or more locations and appropriately programmed in accordance with this specification. For example, the process described with reference to FIG. 3 can be performed by a client computing system 150.

[0055] At 302, the system determines that the presentation of an object within the virtual environment meets a set of visibility conditions. The set of visibility conditions can include one or more system checks of the visibility of the object within the user's field of view within the virtual environment (e.g., within a viewport). In some embodiments, the set of visibility conditions can include a set of sequential checks, where each visibility check is required by a set of rules that need to be verified before the system can perform the next visibility check. If the visibility conditions are not met, the system does not proceed to the next verification step of the set of visibility conditions, thereby saving processing resources that would otherwise be allocated to performing further visibility analysis. For example, the system can choose to end the process of determining that the presentation of the object meets the set of visibility conditions for each of the remaining visibility conditions of the set of visibility conditions.

[0056] In some embodiments, a series of validations of a set of visibility conditions can be ordered such that the computational requirements for each validation step, e.g., computational complexity, resource usage, power requirements, the period to complete the computation, etc., are in an increasing order. In other words, the visibility condition with the least computational load will be validated before the visibility condition with a greater computational load. By arranging the sequence of checks of the set of visibility conditions in the order of increasing computational complexity, especially when the visibility analysis ends based on the determination that the object is not visible (e.g., blocked or otherwise considered non-visible) at an early stage of the sequence, the computational requirements necessary for the execution of the set of visibility conditions can be reduced. In other words, if the visibility condition with low computational complexity indicates that the object is not visible, there is no need to execute the visibility condition with a higher level of computational complexity.

[0057] At 304, the system captures a two-dimensional projection of the object as presented in the virtual environment. Capturing the two-dimensional projection can include applying a projective transformation to the object within the virtual environment, e.g., using a homography. The two-dimensional projection can be resized to match the dimensions of the reference version of the object. For example, assume that the reference version of the object is the 8x8 version of the object. In this example, for the analysis to be performed, the two-dimensional projection can be scaled down to the 8×8 version.

[0058] At 306, the system determines that the two-dimensional projection of the object matches the reference version of the object based on a comparison of the average color of the features of the reference version of the object and the average color of the features in the two-dimensional projection of the object. In some embodiments, determining the match between the two-dimensional projection of the object and the average reference version includes determining that the two-dimensional projection of the object has a difference smaller than a specified (e.g., threshold) difference between the average color of the features of the two-dimensional projection of the object and the average color of the features of the reference version of the object.

[0059] In some embodiments, the system calculates the average hash of the two-dimensional projection and the average hash of the reference object, and determines the match between the two-dimensional projection of the object and the reference version of the object by comparing the average hash of the two-dimensional projection of the object with the average hash of the reference version of the object. The system can calculate the average hash of the two-dimensional projection of the object by calculating the average color value of the pixels included in at least a part of the two-dimensional projection of the object.

[0060] In some embodiments, when an object such as an image and / or an advertisement is presented in a virtual environment, the average hash operation can be performed within the area of the virtual environment occupied by the content and within the field of view presented to the user. The average hash operation can be performed on a 2D representation. The output of the average hash operation can be the actual hash value representing the presentation of the advertisement to the user in the virtual environment.

[0061] As part of the average hash operation, each pixel included in at least a part of the two-dimensional projection is encoded based on whether the color value of the pixel is at least the average color value. More specifically, if the color value of the pixel is above the average color of the evaluated part, the pixel is encoded with "1", otherwise, the pixel is encoded with "0". In some embodiments, the hash image is converted to grayscale before encoding, and the average grayscale pixel value is calculated using the grayscale values of all the pixels in the part of the two-dimensional projection of the image being evaluated. The system creates a bit string based on the encoded pixels and converts the bit string to a hexadecimal value.

[0062] Similarly, the system can calculate the average color value of the pixels included in the reference version of the object. Each of the pixels included in the reference version of the object can be encoded based on whether the color value of the pixel is at least the average color value, and the system can create a bit string based on the encoded pixels and convert the bit string into a hexadecimal value. The system can determine whether the two-dimensional projection of the object matches the reference version of the object by determining the difference between the hexadecimal value corresponding to the two-dimensional projection of the object and the hexadecimal value corresponding to the reference version of the object. The match can be verified based on the difference between the hexadecimal values being less than a threshold difference value. For example, if the reference hash value is hexadecimal B98C0 and the actual hash value is hexadecimal B98B0, the average hash analysis outputs a difference of hexadecimal 10 (i.e., B98C0 - B98B0 = 10). This difference between the actual hash value and the reference hash value is compared to a predetermined threshold to arrive at a visibility determination. For example, if the difference is greater than the threshold, the object is classified as invisible / imperceptible, but if the difference is less than the threshold, the object is classified as visible / perceptible.

[0063] In some embodiments, the system determines whether the two-dimensional projection of the object matches the reference version of the object by identifying the positions of a set of features within the reference version of the object. The features can include, for example, the edges and / or corners of the object. For example, the system determines that the two-dimensional projection of the object matches the reference version of the object by identifying the positions of a set of edges and / or corners of the reference version of the object. The features of the object can also include visual differences, such as text, transitions between light / dark colors, transitions between unique colors, or other visually distinguishable features of the object.

[0064] In some embodiments, determining the visibility of an object may depend on which of the features in the reference object are also found within the set of features detected in the actual object as presented in the virtual environment. For example, if the original version of the content has 15 reference corners, the visibility of the content presented in the computing environment may be based on how many of those reference corners / edges are detected in the presentation of the content within the field of view presented to the user. If only 5 of the reference corners / edges are found at the detected actual corners, the corner detection analysis outputs a difference of 10 (i.e., 15 - 5). This difference between the value of the actual corner and the value of the reference corner is compared to a predetermined threshold to arrive at a determination of visibility. For example, if the difference is greater than the threshold, the object is classified as non - visible, but if the difference is less than the threshold, the object is classified as visible.

[0065] The system can search for the positions of sets of features such as edges / corners in the two - dimensional projection of the object. For example, Harris corner detection can be used to search for and identify the features of the object. In some embodiments, the system can convert the image to grayscale and apply filtering to smooth the noise in the image. The system can use the Sobel operator to obtain the x - gradient and y - gradient values of each pixel in the image and calculate the corner / edge features considering an N×N (e.g., 3x3, 4x4, etc.) window surrounding each pixel. The calculated characteristics of the corner / edge can be a corner intensity function, such as a Harris value. Pixels having a Harris value greater than a specified threshold can be reliably identified as corners / edges of the object.

[0066] In some embodiments, the system can search for a threshold (e.g., sufficient) subset of the edges of the reference version of the object in the two-dimensional projection. To ensure that the edges detected in the actual presentation of the object are the same as the edges detected in the reference version of the object, the system compares the average color of the pixels at the positions of the edges in the two-dimensional projection with the average color of the pixels at the positions of the edges in the reference version of the object. The match can be verified based on a comparison between the average color of the corresponding pixels obtained from the two-dimensional projection and the reference object with respect to the threshold. For example, an expression of the difference in the average color of the corresponding pixels obtained from the two-dimensional projection and the reference object is calculated, and an empirically determined threshold is applied to the calculated difference to determine that it is a valid match. In some embodiments, other visual features identified near the reference edges can be used to distinguish the various reference corners and / or to ensure that the actual edges detected in the actual presentation of the object correspond to the reference corners.

[0067] At 308, the system classifies the presentation of an object within the virtual environment based on whether the two-dimensional projection of the object matches the reference version of the object. In some embodiments, the system classifies the presentation of the object as visible (310). The system increments the visibility count of the object (312). The system can repeat the process described with respect to 302 - 308. For example, the system can continue to periodically check the visibility of the object over a period of time according to 302 - 308. The duration of the periodic check can be, for example, about 1 second. During the period of the periodic check of the object's visibility, the system can determine that the object continues to meet the visibility conditions and is classified as visible. In some embodiments, the system determines that the number of a series of increments of the visibility count meets the threshold of the visibility count (314). When the system determines that the number of a series of increments meets the threshold of the visibility count, then the system registers the presentation of the object (316). Registering the presentation of the object can include providing confirmation to the publisher of the presented object.

[0068] In some embodiments, the system classifies the presentation of an object within the virtual environment as a non-visible rendering of the object (318). In some embodiments, the system determines that the number of a series of increments of the non-visibility count meets the threshold of the non-visibility count (320). The system can repeat the process described with reference to 302-308 periodically, for example, multiple times per second, and each non-visibility count is incremented when the system classifies the object as non-visible. In some embodiments, the system determines that the number of a series of increments of the non-visibility count meets the threshold of the non-visibility count (322). For example, the number can be a series of increments of the non-visibility count. In other examples, the non-visibility count can be non-sequential increments, for example, a cumulative non-visibility count measured over a period of time. In some embodiments, when the non-visibility count meets the threshold of the non-visibility count, the system can provide an alert regarding the non-visibility of the object (324). In some embodiments, the alert can be provided to the game engine to unload the object. For example, when the system determines that the number of times the object has been classified as non-visible has reached the threshold, the system can provide an alert to exchange the object with another different object.

[0069] In some embodiments, the set of visibility conditions includes a series of checks for the visibility of an object within the virtual environment by the user (e.g., as described with reference to 302 in FIG. 3). The set of visibility conditions can include a set of checks in a sequential order, and each visibility check needs to be verified before the system can perform the next visibility check. If the visibility conditions are not met, the system may not be able to proceed to the next verification step of the set of visibility conditions. FIG. 4 is a flowchart of an exemplary process 400 for determining that the presentation of an object meets a set of visibility conditions.

[0070] At 402, the system verifies the visibility of the object. In some embodiments, checking the visibility of the object includes receiving, from a processor (e.g., a GPU or a CPU), a rendering confirmation that the object is rendered within the computing environment. For example, a game engine can instruct the processor to render an object in a virtual environment. The rendered object may go outside the viewport that is visible to the user. For example, the rendered object may be within the computing environment but outside the user's field of view. In other examples, the rendered object may go outside the user's viewport but appear as a shadow within the user's field of view. The rendered object may be inside the viewport that is visible to the user. If the system determines that the object is not visible in the viewport, the system determines that the object does not meet the visibility condition(s).

[0071] At 404, the system determines that the viewing angle of the object meets the criteria of an angle threshold. The viewing angle can be determined as the angle from the surface normal of the object, such as the normal 216 in FIG. 2, to the user's viewpoint. For example, the object can be a two-dimensional display within a virtual computing environment, and the normal is defined from the two-dimensional surface. In some embodiments, the viewing angle is compared with the criteria of an angle threshold, such as a viewing angle threshold or a viewing angle range threshold, and a viewing angle that exceeds the viewing angle threshold or is outside the viewing angle range threshold is determined by the system to be non-visible. For example, the viewing angle can be set to be non-visible when it exceeds 55 degrees according to a standard, such as the IAB (Interactive Advertising Bureau) standard. In an example where the system determines that the viewing angle of the object does not meet the criteria of the angle threshold, the system determines that the object does not meet the visibility condition(s).

[0072] At 406, the system determines whether an object is fully or partially on the screen. In some embodiments, determining whether an object is on the screen, e.g., within the user's field of view, includes determining that the pixels included in the rendered object have respective coordinates that coincide with the user's field of view within the computing environment. For example, the system can map the world coordinates of the object to the coordinates of the viewport to determine whether the coordinates of the object are within the boundaries of the viewport. If the system determines that the object is not on the screen, the system determines that the object does not meet the visibility condition(s). If the system determines that the object is partially on the screen, the system can further determine that the portion of the object determined to be on the screen does not meet the minimum threshold of an object considered to be visible.

[0073] At 408, the system determines whether one or more features of an object within the user's field of view are blocked by one or more other objects located between the user's viewpoint and the object, and causes one or more other objects to block the view of the user's object. In some embodiments, as part of determining whether one or more other objects block the view of the object from the user, the system can determine whether one or more other objects meet a transparency threshold. For example, if one or more other objects are sufficiently transparent, as indicated by meeting the transparency threshold, they may not be considered to block the features of the object. In some embodiments, determining whether one or more other objects block the view of the user's object includes using ray casting techniques, as described in more detail, for example, with reference to FIG. 5.

[0074] As shown in the exemplary operating environment 500 of FIG. 5, an object 502 can be blocked by other objects 504 within the viewport 506 of a user 508 shown as a camera in FIG. 5, for example. In some embodiments, a raycast, such as raycast 510, can be initiated from the user 508's perspective to a point on the object 502. For example, the raycast can be initiated from the user's viewport to a point on the object, and the point on the object can be a corner and / or edge of the object, as well as a central feature of the object (e.g., a center point). If the raycast is blocked by other objects, the system can recursively initiate other raycasts from the other objects towards the point on the object. In some embodiments, the raycast can alternatively (or additionally) be initiated from a point on the object 502 towards the user 508's perspective. For example, the raycast can be initiated from a point on the object to the user's viewport, and the point on the object can be a corner and / or edge of the object, as well as a central feature of the object (e.g., a center point). If the raycast is blocked by other objects, the system can recursively initiate other raycasts from the other objects towards the user's viewport.

[0075] If the system determines that one or more other objects are located between the object and the user's viewport, the system can check the transparency values of the one or more obstructing objects. For example, the system can check the RGBA values and / or physical attributes of the one or more objects determined to be between the user's viewport and the object. Based on the transparency values of the one or more objects, the system can determine that the transparency threshold is met when the one or more objects are determined to be transparent. If the transparency threshold is met for one or more obstructing objects located between the user's viewpoint and the features of the object, the system can determine that the features are not obstructed. In some embodiments, the system can determine that the object is obstructed if at least one feature of the object, e.g., two or more features of the object, is obstructed. The features of the object can be, for example, points, regions, corners, edges, etc. of the object. For example, the system can determine that the object is obstructed if at least one corner or edge of the object is obstructed. In some embodiments, the system can determine that the object is obstructed if a feature located at the center of the object, e.g., the center point, is obstructed. If the system determines that one or more features of an object within the user's field of view are obstructed, the system determines that the object does not meet the visibility condition(s).

[0076] Referring to FIG. 4, at 410, the system verifies that the dimension of the object meets the threshold dimensionality. In some embodiments, the system verifies the threshold dimensionality by determining that the pixel ratio (e.g., percentage) of the pixels included in the object meets the threshold as compared to the pixels on the screen (e.g., the pixels of the viewport). For example, the threshold pixel percentage of the object pixels can be about 1.5% of the pixels on the screen, and an object having a number of object pixels less than about 1.5% of the number of pixels on the screen does not meet the threshold dimensionality. For example, the system can verify the dimension of the object using polygon-based calculations, in which case the area of the polygon is calculated using Equation (1) for (i) the area of the object on the screen (e.g., within the viewport) and (ii) the complete area of the object, using the vertices of the corners of the viewport. An example of object 600 viewed through viewport 602 of client device 604 is shown in FIG. 6. The system can normalize the viewport space. For example, the lower left corner is (0,0) and the upper right corner is (1,1). Equation (1) calculates the percentage of the area of the object with respect to the area of the viewport.

Number

[0077] In some embodiments, the system verifies the threshold dimensionality by determining the number of pixels of an object that are pixels on the screen. For example, the threshold number of object pixels may be such that at least about 50% of the object pixels are pixels on the screen. In other words, at least about half of the object pixels are rendered within the user's viewport. If the system determines that the dimensionality of the object does not meet the threshold dimensionality, the system determines that the object does not meet the visibility condition(s). For example, the system can verify the dimensionality of an object using a polygon-based calculation where the area of the polygon is calculated using Equation (1) where the coordinates of the object are constrained to the area of the object that is visible at the viewport boundary and on the screen. Dividing this value by the unconstrained area of the object gives the percentage of the object that is visible in the viewport.

[0078] At 412, the system determines that the average luminance of the object meets the luminance threshold. In some embodiments, the system determines the average luminance of the object by calculating the average luminance of the pixels included in the object and converting the value of the average luminance to a representative value. For example, the average luminance can be the average value, median value, mode value, or central tendency of the luminance of the pixels included in the object. Equation (2) is an example that can be used to calculate the luminance (e.g., lumens) per pixel.

Number

[0079] 414, the system determines that the two-dimensional projection of the object matches the reference version of the object based on the comparison between the average color of the features of the reference version of the object and the average color of the features in the two-dimensional projection of the object. For example, the reference version of the object can be obtained from, for example, the content delivery system 152 of the content database 168.

[0080] In some embodiments, the steps described by reference are executed by one or more processors. The processor(s) can be, for example, a GPU, a CPU, or a TPU. Sometimes, one or more of the calculations are selectively executed by the processor based on, for example, the calculation speed, the available processing power, the type of calculation, etc. In some embodiments, the process can be executed in parallel by two or more processors. In some embodiments, determining the pixel-based visibility condition and / or classifying the presentation of the object is executed by the GPU. In some embodiments, calculations that require less calculation than the threshold calculation are executed by the CPU, and calculations that require more calculation than the threshold calculation are executed by the GPU.

[0081] In some embodiments, the operations described with reference to FIGS. 3 and 4 are performed to achieve a sequentially increasing computational complexity. For example, the object visibility pipeline can start from step 302 in FIG. 3 and sequentially execute processes 402, 404, 406, 408, 410, and 412 in FIG. 4. Step 414 in FIG. 4 may follow, and the operation of step 414 will be described in further detail with reference to steps 304, 306, 308, and 310 / 318 in FIG. 3. Following the classification at step 310 / 318, the system can perform additional steps 312, 314, and 316, or steps 320, 322, and 324, respectively. Of course, the order of the operations performed can be changed.

[0082] In some embodiments, some or all of the operations described with reference to FIGS. 3 and 4 can be repeatedly executed to continuously check the visibility of an object while the object is being rendered in a virtual environment.

[0083] In some embodiments, the operations described with reference to FIGS. 3 and 4 can be utilized by a system to determine the object visibility of a plurality of (e.g., two or more) objects rendered within a virtual environment.

[0084] FIG. 7 is a block diagram of an exemplary computer system 700 that can be used to perform the operations described above. System 700 includes a processor 710, a memory 720, a storage device 730, and an input / output device 740. Each component 710, 720, 730, and 740 can be interconnected using, for example, a system bus 750. The processor 710 can process instructions for execution within the system 700. In one embodiment, the processor 710 is a single-threaded processor. In other embodiments, the processor 710 is a multi-threaded processor. The processor 710 can process instructions stored in the memory 720 or the storage device 730.

[0085] Memory 720 stores information within system 700. In one embodiment, memory 720 is a computer-readable medium. In one embodiment, memory 720 is a volatile memory unit. In other embodiments, memory 720 is a non-volatile memory unit.

[0086] Storage device 730 can provide mass storage for system 700. In one embodiment, storage device 730 is a computer-readable medium. In various different embodiments, storage device 730 can include, for example, a hard disk device, an optical disk device, a storage device shared over a network by multiple computing devices (e.g., a cloud storage device), or any other such mass storage device.

[0087] Input / output device 740 provides input / output operations for system 700. In one embodiment, input / output device 740 can include one or more of a network interface device, such as an Ethernet card, a serial communication device, such as and an RS-232 port, and / or a wireless interface device, such as an 802.11 card. In other embodiments, the input / output device can include a driver device configured to receive input data and transmit output data to an external device 760, such as a keyboard, a printer, a display device, etc. However, other embodiments, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc., can also be used.

[0088] Although an exemplary processing system has been described with reference to FIG. 7, embodiments of the subject matter and functional operations described herein can be implemented in other types of digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or combinations of one or more of them.

[0089] An electronic document may correspond to a file, but it is not necessarily required to correspond to a file. A document may be stored in part of a file that holds other documents, a single file dedicated to the document, or multiple related files.

[0090] Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions, encoded on a computer storage medium for execution by, or to control the operation of, a data processing apparatus. Alternatively, or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information to be sent to a suitable receiver device for execution by the data processing apparatus. A computer storage medium can be, or can include, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Further, a computer storage medium is not a propagated signal, but a computer storage medium can be the source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium can also be, or can include, one or more separate physical components or media (such as multiple CDs, disks, or other storage devices).

[0091] The operations described in this specification can be implemented as operations executed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0092] The term "data processing apparatus" encompasses, by way of example, any kind of apparatus, device, and machine for processing data, including programmable processors, computers, systems on a chip, or multiple ones thereof or combinations thereof. The apparatus can include special-purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). The apparatus can also include, in addition to hardware, code that creates an execution environment for a computer program of interest, such as processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or code that constitutes one or more combinations thereof. The apparatus and the execution environment can implement various different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0093] The data processing device can also take the form of a game device. A game device is a device that enables a user to participate in a game application. For example, a user can control one or more characters, avatars, or other rendered content displayed in the game application. A game device typically includes a computer processor, a hardware memory device, and a controller interface (rendered physically or visually on a display) that enables user control over the content rendered by the game application. The game device can store and execute the game application locally, or execute a game application (such as an online game application) that is at least partially stored and / or provided by a cloud server. Similarly, the game device can interface with a game server that executes the game application and "streams" the game application to the game device. The game device can be a tablet device, a mobile communication device, a computer, or another device that performs functions other than executing a game application.

[0094] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or in the form of a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may or may not correspond to a file in a file system. The program can be stored in a part of a file that holds other programs or data (e.g., one or more scripts stored in a document of a markup language), in a single file dedicated to the program of interest, or in multiple related files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers located at one location or distributed across multiple locations and interconnected by a communication network.

[0095] The processes and logic flows described herein can be executed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data to generate output. The processes and logic flows can also be executed by special-purpose logic circuits, such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits), and the apparatus can also be implemented as those special-purpose logic circuits.

[0096] Processors suitable for the execution of a computer program include, for example, both general and special purpose processors, as well as any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. Essential elements of a computer are a processor that performs actions in accordance with instructions, and one or more memory devices for storing the instructions and data. Generally, a computer also includes, or is coupled to operate for receiving data from, transferring data to, or both, one or more mass storage devices such as magnetic disks, magneto-optical disks, or optical disks. However, a computer does not require such devices. Additionally, a computer can be incorporated into other devices such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, global positioning system (GPS) receivers, or portable storage devices (such as universal serial bus (USB) flash drives). Storage devices suitable for storing computer program instructions and data include, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and all forms of non-volatile memory, media, and memory devices including CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, dedicated logic circuitry.

[0097] To interact with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and a pointing device, such as a mouse or trackball, by which the user can input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be acoustic, speech language, or tactile input. Further, the computer can interact with the user by sending and receiving documents to and from the devices used by the user, such as by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0098] Embodiments of the subject matter described herein can be implemented in a computing system that includes a back-end component, such as a data server, or a middleware component, such as an application server, or a front-end component, such as a graphical user interface or a web browser by which a user can interact with an embodiment of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, such as by a communication network. Examples of communication networks include local area networks ("LANs"), and wide area networks ("WANs"), Internet networks (such as the Internet), and peer-to-peer networks (such as ad hoc peer-to-peer networks).

[0099] A computing system can include a client and a server. The client and the server are generally far apart from each other and typically communicate through a communication network. The relationship between the client and the server is created by computer programs operating on respective computers and by the client and the server having a relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to the client device (for the purpose of, for example, displaying the data to a user interacting with the client device and receiving user input from the user). Data generated at the client device (e.g., as a result of user interaction) can be received at the server from the client device.

[0100] Although this specification contains many details of specific embodiments, these should not be construed as limiting the scope of any invention or of what may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. The particular features described in the context of individual embodiments herein can also be implemented in combination within a single embodiment. Conversely, the various features of the present invention described in the context of a single embodiment can be implemented separately, or in any suitable sub-combination, in a plurality of embodiments. Furthermore, where features are described as functioning in a particular combination and are initially claimed as such, one or more features from the claimed combination may in some cases be excised from the combination, and the claimed combination may be directed to a sub-combination or variation of a sub-combination.

[0101] Similarly, although operations are shown in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order or in a sequential order shown, or that all of the operations shown be performed, to obtain a desirable result. In certain circumstances, multitasking and parallel processing may be advantageous. Further, the separation of various system components in the above embodiments should not be understood as requiring such separation in all embodiments, and the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0102] Thus, particular embodiments of the invention have been described. Other embodiments are within the scope of the following claims. In some cases, different orders of performing the actions recited in the claims may still result in desirable outcomes. Further, the processes shown in the accompanying figures do not necessarily require being in the particular or sequential order shown to obtain a desirable result. In certain embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A method for determining the visibility of an object by a user in a virtual environment, comprising: determining that the presentation of the object in the virtual environment meets a set of visibility conditions; capturing a two-dimensional projection of the object presented in the virtual environment; determining that the two-dimensional projection of the object matches the reference version of the object based on a comparison of the average color of the features of the reference version of the object and the average color of the features in the two-dimensional projection of the object; classifying the presentation of the object in the virtual environment based on whether the two-dimensional projection of the object matches the reference version of the object, classifying the presentation of the object in the virtual environment as a visible rendering of the object in response to determining that the two-dimensional projection of the object matches the reference version of the object, and classifying the presentation of the object in the virtual environment as a non-visible rendering of the object in response to determining that the two-dimensional projection of the object does not match the reference version of the object, including classifying, including a method.

2. The set of visibility conditions includes: verifying, from one or more processors, the rendering confirmation of the object in the virtual environment, the method according to claim 1.

3. The set of visibility conditions includes: determining that the viewing angle of the object within the user's field of view in the virtual environment meets the criteria of a threshold angle with respect to the surface normal of the object in the virtual environment with respect to the user's field of view, the method according to claim 1 or 2.

4. The set of visibility conditions includes: determining that the object pixels including the object include coordinates that match the user's field of view in the virtual environment, the method according to any one of claims 1 to 3.

5. The set of visibility conditions includes: determining that one or more features of the object within the user's field of view are not blocked by one or more other objects, the method according to any one of claims 1 to 4.

6. Determining that the one or more features of the object within the user's field of view are not blocked by one or more other objects includes determining that a transparency threshold is met for the one or more other objects determined to be located between the user's field of view and the one or more features of the object, the method according to claim 5.

7. The one or more features of the object include at least one corner feature of the object and a center feature of the object, the method according to claim 5 or 6.

8. The set of visibility conditions is verifying that the dimensionality of the object meets a threshold dimensionality, determining that a pixel ratio between the object pixels and the pixels on the screen meets a threshold value, and determining a threshold number of object pixels including the pixels on the screen, including verifying, the method according to any one of claims 1 to 7.

9. The set of visibility conditions is determining that an average luminance of the object meets a luminance threshold, the method according to any one of claims 1 to 8.

10. Determining that the average luminance of the object meets the luminance threshold includes calculating an average luminance of the pixels including the object, converting the average luminance to a representative value, and comparing the representative value with the value of the luminance threshold, the method according to claim 9.

11. Classifying the presentation of the object in the virtual environment includes incrementing a count of the visibility of the object in response to classifying the presentation of the object in the virtual environment as a visible rendering of the object, determining that a series of increments of the count of the visibility of the object meets a visibility count threshold, and registering the presentation of the object, further including, the method according to any one of claims 1 to 10.

12. Classifying the presentation of the object in the virtual environment includes incrementing a count of the non-visibility of the object in response to classifying the presentation of the object in the virtual environment as a non-visible rendering of the object, Determining that the number of a series of increments of the non-visibility count satisfies a threshold of the non-visibility count; Further comprising providing an alert regarding the non-visibility of the object, the method according to any one of claims 1 to 11.

13. Determining that the two-dimensional projection of the object matches the reference version of the object comprises Calculating a hash of the two-dimensional projection, and Comparing the hash of the two-dimensional projection with the hash of the reference version of the object, the method according to any one of claims 1 to 12.

14. Calculating the hash of the two-dimensional projection and the hash of the reference version of the object is calculating an average hash, which Comprises calculating an average color value of at least a part of the two-dimensional projection, Encoding each pixel of the two-dimensional projection based on whether the color value of the pixel is at least the average color value, Creating a bit string based on the encoded pixels, and Converting the bit string into a hexadecimal value, the calculating method according to claim 13.

15. Determining that the two-dimensional projection of the object matches the reference version of the object comprises determining a difference between the hexadecimal value and a reference hexadecimal value representing the reference version of the object, the method according to claim 14.

16. Determining that the two-dimensional projection of the object matches the reference version of the object comprises Identifying positions of a set of edges in the reference version of the object, Searching for the positions of the set of edges in the two-dimensional projection, and Comparing an average color of pixels at the positions of the edges in the two-dimensional projection with an average color of pixels at the positions of the edges in the reference version of the object, the method according to any one of claims 1 to 12.

17. One or more non-transitory computer storage media encoded with computer program instructions, wherein when the instructions are executed by one or more computers, the one or more computers are caused to perform operations, and the operations are Determining that the presentation of an object in a virtual environment meets a set of visibility conditions, Capturing a two-dimensional projection of the object presented in the virtual environment, Based on a comparison between the average color of features of a reference version of the object and the average color of the features in the two-dimensional projection of the object, determining that the two-dimensional projection of the object matches the reference version of the object, Classifying the presentation of the object in the virtual environment based on whether the two-dimensional projection of the object matches the reference version of the object, In response to determining that the two-dimensional projection of the object matches the reference version of the object, classifying the presentation of the object in the virtual environment as a visible rendering of the object, and In response to determining that the two-dimensional projection of the object does not match the reference version of the object, classifying the presentation of the object in the virtual environment as a non-visible rendering of the object, one or more non-transitory computer storage media comprising.

18. A system comprising one or more computers and one or more storage devices storing instructions, the instructions when executed by the one or more computers cause the one or more computers to perform operations, the operations being, Determining that the presentation of an object in a virtual environment meets a set of visibility conditions, Capturing a two-dimensional projection of the object presented in the virtual environment, Based on a comparison between the average color of features of a reference version of the object and the average color of the features in the two-dimensional projection of the object, determining that the two-dimensional projection of the object matches the reference version of the object, Classifying the presentation of the object in the virtual environment based on whether the two-dimensional projection of the object matches the reference version of the object, In response to determining that the two-dimensional projection of the object matches the reference version of the object, classifying the presentation of the object within the virtual environment as a visually recognizable rendering of the object, and In response to determining that the two-dimensional projection of the object does not match the reference version of the object, classifying the presentation of the object within the virtual environment as a non-visually recognizable rendering of the object, including classifying, and a system including the same.

Images