Object visibility in virtual environment

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining visibility of a user to an object in a virtual environment, including: capturing a two-dimensional projection of an object as presented in the virtual environment; determining that the two-dimensional projection of the object matches the reference version of the object based on a comparison of the average color of the features of the reference version of the object and the average color of the features in the two-dimensional projection of the object; and classifying the presentation of the object in the virtual environment based on whether the two-dimensional projection of the object matches the reference version of the object.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application Serial No. 63 / 443,171, filed on February 3, 2023, the entirety of which is incorporated herein by reference. Technical Field

[0003] This specification relates to virtual computing environments. Background Art

[0004] Virtual environments in gaming environments, metaverses, and multiverses may include content rendered throughout the virtual environment, but the content may not be visible for a variety of reasons. For example, the content rendered in the virtual environment may not be presented until the user adjusts the user's field of view to the location of the rendered content in the virtual environment. The rendered content in the virtual environment may also be obscured from view by the user, for example, by one or more other objects in the virtual environment, thereby preventing the user from viewing the rendered content. Summary of the invention

[0005] This specification describes techniques for determining whether content presented within a user's field of view in a computing environment is obscured, corrupted, or otherwise not considered viewable.

[0006] These techniques generally relate to an object visibility pipeline for determining whether an object (e.g., publisher content) presented to a user interacting with a computing environment (e.g., a dynamic gaming environment, a two-dimensional, three-dimensional, augmented reality, virtual reality environment) is visible to the user. Based on the results of the object visibility pipeline, the rendered content may be classified as visible or not visible (e.g., occluded or otherwise deemed not visible) in the computing environment.

[0007] In general, one innovative aspect of the subject matter described in this specification can be embodied in a method comprising the following acts: determining that a presentation of an object within a virtual environment satisfies a set of visibility conditions; capturing a two-dimensional projection of the object as presented in the virtual environment; determining that the two-dimensional projection of the object matches a reference version of the object based on a comparison of an average color of features of a reference version of the object with an average color of features in the two-dimensional projection of the object; and classifying the presentation of the object in the virtual environment based on whether the two-dimensional projection of the object matches the reference version of the object. In response to determining that the two-dimensional projection of the object matches the reference version of the object, classifying the presentation of the object within the virtual environment as a visible rendering of the object. In response to determining that the two-dimensional projection of the object does not match the reference version of the object, classifying the presentation of the object within the virtual environment as a non-visible rendering of the object.

[0008] Other embodiments of this aspect include corresponding systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0009] The foregoing and other embodiments may each optionally include one or more of the following features, alone or in combination. In particular, one embodiment includes all of the following features in combination. In some implementations, a set of visibility conditions includes verifying rendering confirmation for objects within the virtual environment from one or more processors.

[0010] In some implementations, a set of visibility conditions includes determining that a viewing angle of an object in the virtual environment within a field of view of the user satisfies a threshold angle criterion regarding a surface normal of the object in the virtual environment relative to the field of view of the user.

[0011] In some implementations, a set of visibility conditions includes determining that object pixels comprising the object include coordinates that coincide with a field of view of a user within the virtual environment.

[0012] In some implementations, a set of visibility conditions includes determining whether one or more features of an object within the user's field of view (e.g., from the user's perspective) meet a transparency threshold. The one or more features of the object may include at least one corner feature of the object and a center feature of the object.

[0013] In some implementations, the set of visibility conditions includes verifying that the dimensionality of the object meets a threshold dimensionality. Verifying the dimensionality includes: determining that a pixel ratio of object pixels to on-screen pixels meets a threshold; and determining that the threshold number of object pixels includes on-screen pixels.

[0014] In some implementations, a set of visibility conditions includes determining that an average brightness of an object satisfies a threshold brightness. Determining that an average brightness of an object satisfies a threshold brightness may include:

[0015] An average brightness of pixels constituting an object is calculated; the average brightness is converted into a representative value; and the representative value is compared with a threshold brightness value.

[0016] In some implementations, classifying the presentation of an object in a virtual environment further comprises: in response to classifying the presentation of an object in the virtual environment as a visual rendering of the object, incrementing a visibility count of the object; determining that a sequential number of increments of the visibility count of the object satisfies a threshold visibility count; and registering the presentation of the object.

[0017] In some implementations, classifying the presentation of an object in a virtual environment further includes: in response to classifying the presentation of an object in the virtual environment as a non-visible rendering of the object, incrementing a non-visibility count of the object; determining that a sequential number of increments of the non-visibility count satisfies a threshold non-visibility count; and providing an alert regarding the non-visibility of the object.

[0018] In some implementations, determining that a two-dimensional projection of an object matches a reference version of the object includes: computing a hash of the two-dimensional projection; and comparing the hash of the two-dimensional projection to a hash of a reference version of the object. Computing the hash of the two-dimensional projection of the object and the hash of the reference version may include computing an average hash. Computing the average hash may include computing an average color value of at least a portion of the two-dimensional projection; encoding each pixel of the two-dimensional projection based on whether the color value of the pixel is at least the average color value; creating a bit string based on the encoded pixels; and converting the bit string to a hexadecimal value.

[0019] In some implementations, determining that the two-dimensional projection of the object matches a reference version of the object includes determining a difference between the hexadecimal value and a reference hexadecimal value representing the reference version of the object.

[0020] In some implementations, determining whether a two-dimensional projection of an object matches a reference version of the object includes: identifying locations of a set of edges in the reference version of the object; searching for locations of the set of edges in the two-dimensional projection; and comparing an average color of pixels at the locations of the edges in the two-dimensional projection with an average color of pixels at the locations of the edges in the reference version of the object.

[0021] The subject matter described in this specification can be implemented so as to achieve one or more of the following advantages. An object visibility pipeline comprising a series of sequential checks for object visibility, wherein each visibility check is verified before the next visibility check can be performed by the system, can reduce the processing resources required to determine visibility. For example, in the case where a visibility condition is not met, the system may not proceed to the next verification step of a set of visibility conditions, thereby reducing the computational requirements for verifying object visibility. The object visibility pipeline can be used as a lightweight object visibility process that can be performed by an edge device with, for example, computational and / or power constraints on a user device, such as a battery-powered mobile device. In these cases, reducing the processing resources required to make visibility determinations reduces battery consumption, and therefore, extends the amount of time that the device can operate on a single battery charge. In addition, since the user device has limited processing power, reducing the processing resources required to make visibility determinations prevents the negative impact of diverting processing resources from rendering and presenting a virtual three-dimensional environment. For example, using less resource-intensive visibility determinations (such as discussed herein) helps prevent game glitches or lags, both of which can make the game unplayable. Some of the lightweight visibility determination processes include hashing techniques or feature detection techniques (e.g., edge detection techniques), which are discussed in detail below. These two techniques enable visibility determination to be performed at a user device without interrupting or negatively affecting the rendering or presentation of a three-dimensional environment at the user device.

[0022] Verifying object visibility can provide content publishers with critical feedback on the validity of content embedded in a virtual environment (e.g., rendering registration). For example, the rendering of embedded content in a three-dimensional virtual game or experience environment can be more efficiently and accurately verified. The object visibility pipeline can be integrated into a dynamic gaming environment (e.g., a three-dimensional VR / AR experience) without substantially affecting the gaming environment. The user's field of view can constantly change in a dynamic gaming environment, so that content can move in and out of the user's field of view, and in this case, the pipeline can enable the user to more accurately track the rendering of publisher content. Additionally, the object visibility pipeline includes checks for determining that the characteristics of rendered objects are maintained in the environment, thereby not only enabling content publishers to verify accurate content rendering, but can also be used by developers to improve the environment during the development cycle.

[0023] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1A is a block diagram of an example operating environment in which content is distributed to client devices.

[0025] Figure 1B Depicted is a block diagram of an example client computing system configured to render a virtual environment showing third-party content specified by a content distribution system.

[0026] Figure 2 is a block diagram of an example computing environment in which content is presented to a user on a client device.

[0027] Figure 3 is a flowchart of an example object visibility pipeline.

[0028] Figure 4 is a flow chart of an example process of the object visibility pipeline.

[0029] Figure 5 is a block diagram of an example computing environment in which content is presented to a user on a client device.

[0030] Figure 6 is a block diagram of an example computing environment in which content is presented to a user on a client device.

[0031] Figure 7 is a block diagram of an example general purpose computing system.

[0032] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0033] Sometimes, content can be presented to a user of a virtual or augmented reality environment (e.g., a two-dimensional or three-dimensional environment) in a non-destructive format without disrupting the user's immersive experience, while providing a means of presenting content (e.g., third-party publisher content) to a user within a computing environment. The presented content can be a two-dimensional or three-dimensional object rendered within the computing environment. The field of view of a user within the computing environment can change over time (e.g., as the user moves relative to the computing environment). Additionally, the computing environment can include one or more other objects within the user's field of view rendered within the computing environment, which may obstruct (e.g., partially obstruct) the presented content. The visibility of the content to the user within the user's field of view can be determined using an object visibility pipeline, which can determine the visibility of the presented content to the user over a measured time period.

[0034] In some embodiments, tracking the visibility of the rendered content within a certain time period triggers classification of the rendered content within the computing environment as perceivable by the user, which can be referred to as an effective presentation, e.g., an effective impression. For the presentation of the rendered content, other metrics related to the rendered content can be additionally tracked, e.g., content exposure time, user interaction with the content, etc.

[0035] As discussed in more detail below, whether content presentation is classified as perceptible by a user may depend on "unscripted" actions within a virtual environment (e.g., a two-dimensional environment, a three-dimensional environment, or another type of immersive experience). For example, if a user crashes a car in a racing game, and smoke from the fire makes it difficult to perceive a set of content presented on a wall of a building (or a billboard in a three-dimensional environment), the set of content may be considered non-perceptible even if the smoke is not completely opaque and the smoke is not defined to enable the smoke to be directly identified as interfering with the visibility of the object (e.g., a physical collider or other property). Due to the dynamic nature of a three-dimensional environment (e.g., dynamically changing in response to the user's actions), and the possibility that scene objects without defined detectable properties may severely impair the perceptibility of a set of content, techniques may be used to determine the degree to which scene objects (e.g., smoke, clouds, ghosts, etc.) degrade the presentation of the set of content. As discussed in more detail below, these techniques may include one or more of hashing techniques or feature detection / matching techniques, such as edge detection, which may be used to quantify the degree to which a presentation of a set of content has degraded relative to a reference version of the set of content. It should be noted that the techniques discussed herein are described as being used in a three-dimensional environment, but these or similar techniques may also be implemented in a two-dimensional environment.

[0036] Figure 1A 1 is a block diagram of an example framework 100 in which third-party content is distributed for presentation as virtual two-dimensional or three-dimensional objects in a virtual environment (such as a virtual reality environment). The virtual environment can be a two-dimensional (2D) environment, a three-dimensional (3D) environment, or another type of immersive interactive environment, including an augmented reality environment and a virtual reality environment. The example environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 102 connects an application server 104, a user device 106, a third-party content server 108, and a third-party content distribution system 110 (also referred to as a content distribution system). The example framework 100 may include many different application servers 104, user devices 106, and third-party content servers 108.

[0037] User devices 106 are electronic devices capable of requesting and receiving resources (e.g., virtual environment applications) over network 102. Example user devices 106 include personal computers, mobile communication devices, and other devices that can send and receive data over network 102. User devices 106 typically include user applications (such as a web browser) to facilitate sending and receiving data over network 102, but native applications executed by user devices 106 may also facilitate sending and receiving data over network 102.

[0038] Resources (e.g., virtual environment applications or definition files for virtual environments) are resources related to rendering a virtual environment that may include text, images, videos, or other media types on a user device 106. Examples of resources include virtual reality applications, video games, mixed reality applications, augmented reality applications, and definitions of virtual environments that can be displayed in any of these types of applications. Resources may include data that defines one or more virtual environments and virtual objects within the virtual environment. Resources may include data that defines virtual objects (e.g., two-dimensional or three-dimensional objects) for presentation within the virtual environment. Resources may be provided to the user device 106 by an application server 104. For example, the application server 104 may include a server that hosts a publisher's website. In this example, the user device 106 may initiate a request for a given resource, and the application server 104 that hosts the given resource may respond to the request by sending the resource to the user device 106. In some implementations, the application server may provide one or more definition files to the user device 106. The definition file includes data representing the virtual environment, which may be processed by an application installed on the user device 106 to render the virtual environment.

[0039] In some cases, a given resource may include a third-party tag or a third-party script that references a third-party content distribution system 110. In these cases, when the given resource is processed by the user device 106, the third-party tag or the third-party script is executed by the user device 106. The execution of the third-party tag or the third-party script configures the user device 106 to generate a request 112 for third-party content (e.g., content that is not defined within the resource but is obtained from a third party and inserted into the resource), which is sent to the third-party content distribution system 110 via the network 102. For example, the third-party tag or the third-party script may enable the user device 106 to generate a packetized data request including header and payload data. The request 112 may include data such as the name (or network location) of the server from which the third-party content is being requested, the name (or network location) of the requesting device (e.g., the user device 106), and / or information that the third-party content distribution system 110 may use to select the third-party content to be provided in response to the request. The request 112 is sent by the user device 106 to a server of the third-party content distribution system 110 via the network 102 (e.g., a telecommunications network).

[0040] The request 112 may include data specifying a resource, data specifying characteristics of a virtual object (e.g., a two-dimensional or three-dimensional virtual object) on which the third-party content is to be presented, and data specifying characteristics of a virtual environment in which the virtual object appears. For example, data specifying the shape or geometry of a virtual object (e.g., a two-dimensional or three-dimensional virtual object) on which the third-party content is to be presented, the size (e.g., length, width, height, and / or volume) of the virtual object, the location of the virtual object in the virtual environment, the number of eligible surfaces on the virtual object that can receive the third-party content, descriptive keywords associated with the virtual environment, and / or media types eligible to be presented on the virtual object may be provided to the content distribution system 110.

[0041] The request 112 may also include data related to other information, such as information that the user has provided, geographic information indicating the state or region from which the request is submitted, or other information that provides context for the environment in which the third-party content will be displayed. Data specifying the characteristics of the user device 106 may also be provided in the request 112, such as information identifying the model of the user device 106, the selection capabilities of the device 106 (e.g., whether hand-based controls can be used to select virtual objects, whether controls are available on the headset itself that the user can tap to select items rendered in the virtual reality environment), the configuration of the user device 106, the type of electronic display (e.g., a touch screen of a smart phone, tablet computer, or gaming device, or a head-mounted display of a VR device 106). The request 112 may be sent, for example, over a packetized network, and the request 112 itself may be formatted as packetized data having a header and payload data. The header may specify the destination of the packet, and the payload data may include any of the information discussed above.

[0042] The third-party content distribution system 110 selects third-party content to be presented on or near a virtual object in the virtual environment in response to receiving the request 112 and / or using information included in the request 112 .

[0043] In some implementations, the distribution parameters (e.g., selection criteria) for a particular third-party content may include distribution keywords that must be matched (e.g., matching resources or terms specified in the request 112) in order for the third-party content to be eligible for presentation. The distribution parameters may also require that the request 112 include information specifying a particular geographic region (e.g., a country or state) and / or information specifying that the request 112 originated from a particular type of user device 106 in order for the third-party content to be eligible for presentation. The distribution parameters may also specify a bid and / or budget for distributing the particular third-party content.

[0044] The identification of eligible third-party content may be segmented into a plurality of tasks 117a-117c, which are then assigned among computing devices within a group of a plurality of computing devices 114. For example, different computing devices 114 in the group may each analyze different portions of the third-party corpus database 116 to identify various third-party content having distribution parameters that match the information included in the request 112. In some implementations, each given computing device 114 in the group may analyze a different data dimension (or set of dimensions) and communicate the results of the analysis (Res 1-Res 3) 118a-118c back to the third-party content distribution system 110. For example, the results 118a-118c provided by each of the computing devices in the group may identify a subgroup of third-party content that is eligible for distribution in response to the request and / or a subgroup of third-party content that has certain distribution parameters or attributes.

[0045] The third-party content distribution system 110 aggregates the results 118a to 118c received from a set of multiple computing devices 114 and uses information associated with the aggregated results to select one or more instances of third-party content to be provided in response to the request 112. For example, the third-party content distribution system 110 can select a set of winning third-party content based on the results of one or more content evaluation processes, as discussed in further detail below. Then, the third-party content distribution system 110 can generate reply data 120 (e.g., digital data representing the reply) and send the reply data over the network 102, which enables the user device 106 to integrate the set of winning third-party content into the virtual environment, for example, for presentation on a qualifying virtual object in the virtual environment.

[0046] In some implementations, the user device 106 executes instructions included in the reply data 120 that configure the user device 106 and enable the user device to obtain a set of winning third-party content from one or more third-party content servers. For example, the instructions in the reply data 120 may include a network location (e.g., a uniform resource locator (URL)) and a script that causes the user device 106 to send a third-party request (3PR) 121 to the third-party content server 108 to obtain a given winning third-party content from the third-party content server 108. In response to the request, the third-party content server 108 will send third-party data (TP data) 122 to the user device 106, which enables the given winning third-party content to be incorporated into the virtual environment and presented at the user device 106.

[0047] Figure 1B A block diagram of an example client computing system 150 configured to render a virtual environment (e.g., a two-dimensional or three-dimensional virtual environment) showing third-party content specified by a content distribution system 152 is depicted. In some implementations, the client computing system 150 is a user device, such as the user device 106 from FIG. 1 . The content distribution system 152 can be configured as the third-party content distribution system 110 from FIG. 1 , the third-party content server 108 from FIG. 1 , or can include aspects of both the server 108 and the system 110. The content distribution system 152 can generally be implemented as a system of one or more computers at one or more locations. The client computing system 150 communicates with the content distribution system 152 via a network (e.g., the Internet, a local area network, a wireless broadband network). Although Figure 1BAlthough not shown in the figure, client computing system 150 may communicate with other systems in addition to content distribution system 152 for various purposes. For example, client computing system 150 may communicate with a server for an online application store or a developer server to obtain virtual reality, augmented reality, and / or mixed reality applications that enable system 150 to render a virtual environment. Similarly, client computing system 152 may communicate with a server for an online application store or a developer server to obtain definition files for a virtual environment (e.g., an immersive virtual reality game).

[0048] The client computing system 150 can be any of a variety of computing systems and / or gaming devices configured and capable of rendering a virtual environment with combined third-party content. In some examples, the client computing system 150 is configured to present a virtual environment of a virtual reality type, and the user views the virtual environment via a head-mounted display. In other examples, the client computing system 150 is configured to present other types of virtual environments, such as an augmented reality environment, a mixed reality environment, or an environment on a conventional two-dimensional screen. The system 150 can be integrated into one device, or can include multiple individually connected components at one or more locations. In some implementations, the client computing system 150 includes a display 154, a memory subsystem 156, a virtual environment rendering engine 158, an input handler 160, a content manager 162, and a network interface 164.

[0049] Display 154 is an electronic display configured to visually display the virtual environment to the user. Display 154 can take a variety of forms for different types of systems. For example, display 154 may include a head mounted display, a display of a mobile device, a tablet computer, a television, or a game console, or another display through which a user can view the virtual environment. Figure 1BAs depicted, display 154 may be, for example, a head-mounted display or a display (e.g., a screen) of a mobile device. For example, in a virtual reality system, display 154 may be a head-mounted display in which a viewing screen of display 154 is fixed a few inches in front of the user's eyes. In a VR system, display 154 may provide a stereoscopic presentation of a virtual environment, for example, a three-dimensional virtual environment. When a user views the stereoscopic presentation of a virtual environment through a set of lenses, the virtual environment may appear to have depth, so the user feels as if he or she is immersed in the virtual environment. In some implementations, the screen is an integral component of the head-mounted display. In other implementations, a smartphone or other mobile unit is removably fixed to a head unit to form a head-mounted display that uses the screen of the mobile unit as the screen of the head-mounted display. Display 154 may be, for example, a liquid crystal display (LCD), an organic light emitting diode display (OLED), or an active matrix OLED (AMOLED) display.

[0050] Memory subsystem 156 includes one or more storage devices that store data representing a virtual environment. A virtual environment is a virtual environment that can be rendered in three dimensions. Examples of virtual environments include 3D games and video environments (e.g., live or recorded event streams, such as 3D concerts or sporting events). In some cases, a user of client computing system 150 can explore the virtual environment by moving his or her head to look around the environment (e.g., in a virtual reality system), by moving around in the environment, by manipulating objects in the environment, or a combination of these. Other components of client computing system 150 can access memory subsystem 156 to read, write, or delete data from the storage devices.

[0051] In some implementations, the data representing the virtual environment stored by the memory subsystem 156 includes declarations for third-party content. Third-party content, such as virtual objects, may be declared for the virtual environment using any of a variety of suitable programming techniques. In some implementations, a developer may insert tags, scripts, or executable code into a definition file for the virtual environment that, when executed, instantiates an object (e.g., a two-dimensional or three-dimensional object) in the virtual environment according to any parameters specified therein.

[0052] The virtual environment rendering engine 158 is a subsystem of the client computing system 150 that is configured to read the definition of the virtual environment from the memory subsystem 156 and render the virtual environment for presentation to the user via the display 154 and optionally using one or more additional peripheral output devices (e.g., speakers, hand controllers, tactile feedback devices). The rendering engine 158 may include one or more data processing devices (e.g., processors) configured and capable of performing the operations described herein. The data processing device may be dedicated to the rendering engine 158, or may be at least partially shared with other components of the client computing system 150. In some implementations, the rendering engine 158 includes one or more graphics processing units (GPUs) that process virtual environment definition files and render a presentation of the environment. For example, a rendering engine 158 for a virtual reality system may process one or more definition files of a virtual environment to generate a stereoscopic display of the virtual environment that provides an immersive 3D experience to the user when viewed by the user through specially configured lenses.

[0053] The input handler 160 is a subsystem of the client computing system 150 that is configured to monitor one or more input channels to monitor user input received when rendering a virtual environment for a user. The input handler 160 may include one or more data processing devices (e.g., processors) configured to and capable of performing the operations described herein. The input handler 160 may detect various types of user inputs depending on the specific configuration of the client computing system 150. For example, a basic virtual reality (VR) system may detect user input based on signals from one or more orientation and motion sensors in a head-mounted display unit. The orientation and motion sensors may include one or more accelerometers, compasses, gyroscopes, magnetometers, or a combination of such sensors. The orientation and motion sensors may generate signals indicating the direction of the user's gaze within the 3DVR environment in real time, and these signals may be interpreted by the input handler 160 to track the direction of the user's gaze in real time. For example, real-time tracking of the direction of the user's gaze may be used to determine the user's field of view within the virtual environment and define the user's viewport of the visible portion of the virtual environment. Additionally, the client computing system 150 may include one or more buttons or switches, such as on a hand-based controller or on a head-mounted display, that can be actuated by the user to provide input to the system 150. More advanced VR systems may provide additional user input channels, such as motion tracking sensors located outside the head-mounted display that track the movement of reference points on the head-mounted display. The input handler 160 may interpret signals from the external motion sensors to determine the user's motion in six degrees of freedom, including rotation and translation, for example.

[0054] In some implementations, the system 150 includes a content manager 162 to monitor the third-party content provided within the virtual environment. The content manager 162 can be a subsystem of the system 150 that manages the content (e.g., virtual objects) that appear in the virtual environment. The content manager 162 can be implemented as one or more data processing devices (e.g., processors) at one or more locations that are programmed to perform the operations described herein. The data processing device can be dedicated to the content manager 162, or can be shared with one or more other components of the system 150. For example, the data processing device can include a central processing unit (CPU) and / or a graphics processing unit (GPU) of a client device.

[0055] In some implementations, the content manager 162 is configured to receive information from the input handler 160 regarding the user's current field of view through the display 154 and determine that the presentation of the object within the virtual environment satisfies the visibility condition, e.g., as described in reference Figure 3 and Figure 4 Content manager 162 may also be configured to classify the presentation of the object, e.g., as a visual rendering of the object or a non-visual rendering of the object, and provide information related to the classified presentation to content distribution system 152. For example, content manager 162 may provide confirmation of a user interaction with the presentation of the object to content distribution system 152.

[0056] The client computing system 150 sends messages to and receives messages from the content distribution system 152. The content distribution system 152 may be implemented as one or more computers (e.g., data processing devices) at one or more locations. Generally speaking, the content distribution system 152 is configured to select third-party content for display within a virtual environment at the client computing system 150. The content distribution system 152 makes the selected third-party content available to the client computing system 150, for example, by sending the content to the client system 150 over a network (such as the Internet or a local area network). The content distribution system 152 may include one or more of a front-end server 166, a third-party content database 168, a content selector 170, a data repository storing selection criteria 172, a second data repository 174 storing end-user account and profile information, and a third data repository 176 storing third-party content provider account and profile information.

[0057] The front-end server 166 is configured to receive and send information from the content distribution system 150. The front-end server 166 provides an interface for the content distribution system 152 to interact with other computers through a communication network (e.g., the Internet). Figure 1BThe front-end server 166 is shown in communication with the client computing system 150. The front-end server 166 receives requests for third-party content, performs initial processing on the received requests, forwards information resulting from the requests to other appropriate components of the content distribution system 152, and sends responses generated by the system 150 in response to the requests. In some implementations, the front-end server 166 includes a network interface that provides interconnection between the content distribution system 152 and one or more networks, which may be public (e.g., the Internet) or private (e.g., a local area network). The network interface may include one or more network interface cards, which are configured, for example, to send and receive data over a packetized network.

[0058] Content database 168 is a database or other type of data repository that maintains an index of third-party content. The third-party content itself may also be stored by content database 168, may be stored by content distribution system 152 but external to content database 168, or may be stored at one or more other systems other than content distribution system 152. Generally speaking, content database 168 identifies a set of third-party content that is available for content distribution system 152 to return to a client system in response to a request for third-party content (e.g., for presentation within a virtual environment).

[0059] Content selector 170 is a component of content distribution system 152, which selects winning third-party content for a request, such as content to be displayed within a virtual environment. In order to determine winning third-party content, content selector 170 evaluates eligible third-party content items for various selection criteria 172 associated with the request. Selection criteria may include keywords or other contextual data specified in the request. In some implementations, selection criteria include profile data indicating the interests and preferences of the end user of client system 150, profile data of a third-party content provider, and information about the virtual environment in which a virtual object is presented. Selection criteria 172 may also include a bid put forward by a third-party content provider, the bid indicating the price that the third-party content provider is willing to pay for the third-party content of the provider selected and returned for display on or by a virtual object in response to the request. Content selector 170 applies selection criteria 172 to a given third-party content request and performs an evaluation process to select winning third-party content.

[0060] Figure 22 is an example operating environment 200 for object presentation within a virtual environment. A client computing system (e.g., client computing system 150) may render a virtual environment on a display (e.g., display 154). The virtual environment may be rendered by a rendering engine (e.g., virtual environment rendering engine 158 of client computing system 150). The virtual environment may include third-party content that may be rendered as two-dimensional or three-dimensional objects within the virtual environment, for example, third-party content provided by content selector 170. As used herein, a field of view 202 within a virtual environment 204 is defined based on a viewing position of a user 206 within the virtual environment 204. Sometimes, the field of view 202 of a user 206 of a virtual environment 204 may change as the user moves relative to the virtual environment. For example, when a user shifts their gaze within the virtual environment or moves their virtual representation within the virtual environment. In some implementations, as the user's field of view shifts within the virtual environment, an object 208 (e.g., publisher content) within the virtual environment 204 may move out of the user's field of view 202. In other words, the viewport 210 of the user 206 corresponding to the on-screen content may move relative to the object 208 such that at least a portion of the object 208 may be outside of the viewport 210 .

[0061] In some implementations, one or more other objects (e.g., objects 212, 214) are rendered in the virtual environment 204 within the field of view of the user 206. The one or more other objects 212, 214 can be positioned within the virtual environment such that at least one of the objects 212, 214 at least partially obscures the user 206's view of the object 208. For example, the object 212 obscures a portion of the object 208 from being viewed by the user 206 within the viewport 210.

[0062] In some implementations, a system (eg, content manager 162 of client computing system 150 ) may determine whether an object rendered within virtual environment 204 satisfies a visibility condition.

[0063] Figure 3 is a flow chart of an example process 300 for determining visibility of an object to a user in a virtual environment. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations and appropriately programmed according to the present specification. For example, referring to Figure 3 The described processes may be performed by client computing system 150 .

[0064] At 302, the system determines that the presentation of an object within the virtual environment satisfies a set of visibility conditions. The set of visibility conditions may include one or more checks by the system of visibility of the object within the user's field of view (e.g., in a viewport) within the virtual environment. In some implementations, the set of visibility conditions may include a set of sequentially ordered checks, wherein each visibility check is verified before the next visibility check may be performed by the system as required by a set of rules. In the event that a visibility condition is not satisfied, the system may not proceed to the next verification step of the set of visibility conditions, thereby saving processing resources that would otherwise be allocated to perform further visibility analysis. For example, the system may choose to terminate the process of determining that the presentation of an object satisfies the set of visibility conditions for each of the remaining visibility conditions in the set of visibility conditions.

[0065] In some implementations, a verification sequence for a set of visibility conditions can be ordered such that the computational requirements (e.g., computational complexity, resource usage, power requirements, duration to complete the computation, etc.) of each verification step are in increasing order. In other words, the least computationally intensive visibility conditions can be verified before more computationally intensive visibility conditions. Arranging the check sequence for a set of visibility conditions in increasing computational complexity can reduce the computational requirements for executing the set of visibility conditions, particularly in situations where visibility analysis is terminated based on a determination early in the sequence that an object is not visible (e.g., occluded or otherwise deemed not visible). In other words, if a low computational complexity visibility condition indicates that an object is not visible, then visibility conditions with higher levels of computational complexity need not be executed.

[0066] At 304, the system captures a two-dimensional projection of the object as presented in the virtual environment. Capturing the two-dimensional projection may include applying a projection transformation (e.g., using a homography) to the object in the virtual environment. The two-dimensional projection may be resized to match the dimensions of a reference version of the object. For example, assume that the reference version of the object is an 8×8 version of the object. In this example, the two-dimensional projection may be scaled down to the 8×8 version to perform analysis.

[0067] At 306, the system determines that the two-dimensional projection of the object matches the reference version of the object based on a comparison of the average color of the features of the reference version of the object and the average color of the features in the two-dimensional projection of the object. In some implementations, determining a match between the two-dimensional projection of the object and the reference version of the average includes determining that the two-dimensional projection of the object has less than a specified (e.g., threshold) difference between the two-dimensional projection of the object and the average color of the features of the reference version of the object.

[0068] In some implementations, the system determines a match between the two-dimensional projection of the object and a reference version of the object by calculating an average hash of the two-dimensional projection and an average hash of a reference object and comparing the average hash of the two-dimensional projection of the object to the average hash of the reference version of the object. The system can calculate the average hash of the two-dimensional projection of the object by calculating an average color value of pixels included in at least a portion of the two-dimensional projection of the object.

[0069] In some implementations, when an object (such as an image and / or an advertisement) is presented in a virtual environment, an average hash operation can be performed on the area of ​​the virtual environment occupied by the content and within the field of view presented to the user. The average hash operation can be performed on a 2D representation. The output of the average hash operation can be an actual hash value representing the presentation of the advertisement in the virtual environment to the user.

[0070] As part of the average hashing operation, each of the pixels included in at least a portion of the two-dimensional projection is encoded based on whether the color value of the pixel is at least the average color value. More specifically, if the color value of the pixel is equal to or greater than the average color of the evaluated portion, the pixel is encoded with a "1", otherwise, the pixel is encoded with a "0". In some implementations, the hashed image is converted to grayscale before encoding, where the mean grayscale pixel value is calculated using the grayscale values ​​of all pixels in the evaluated portion of the two-dimensional projection of the image. The system creates a bit string based on the encoded pixel and converts the bit string into a hexadecimal value.

[0071] Similarly, the system can calculate the average color value of the pixels included in the reference version of the object. Each of the pixels included in the reference version of the object can be encoded based on whether the color value of the pixel is at least the average color value, and the system can create a bit string based on the encoded pixel and convert the bit string into a hexadecimal value. The system can determine that the two-dimensional projection of the object matches the reference version of the object by determining the difference between the hexadecimal value corresponding to the two-dimensional projection of the object and the hexadecimal value corresponding to the reference version of the object. The match can be verified based on the difference between the hexadecimal values ​​being less than a threshold difference. For example, if the reference hash value is hexadecimal B98C0 and the actual hash value is hexadecimal B98B0, the average hash analysis outputs a hexadecimal difference of 10 (ie, B98C0-B98B0=10). This difference between the actual hash value and the reference hash value is compared with a predetermined threshold to reach a visibility determination. For example, if the difference is greater than the threshold, the object is classified as invisible / imperceptible, but if the difference is less than the threshold, the object is classified as visible / perceptible.

[0072] In some implementations, the system determines that the two-dimensional projection of the object matches the reference version of the object by identifying the locations of a set of features in the reference version of the object. Features can include, for example, edges and / or corners of the object. For example, the system determines that the two-dimensional projection of the object matches the reference version of the object by identifying the locations of a set of edges and / or corners in the reference version of the object. Features of the object can also include, for example, visual distinctions (such as text), transitions between light / dark colors, transitions between different colors, or other visually distinct features of the object.

[0073] In some implementations, determining the visibility of an object can depend on how many features in a reference object can also be found in a set of features detected in an actual object as presented in a virtual environment. For example, if there are 15 reference corners in the original version of the content, the visibility of the content as presented in the computing environment can be based on how many of those reference corners / edges are detected in the presentation of the content within the field of view presented to the user. If only 5 of the reference corners / edges are found in the actual corners detected, the corner detection analysis outputs a difference of 10 (i.e., 15-5). The difference between the actual corner and the reference corner is compared to a predetermined threshold to reach a visibility determination. For example, if the difference is greater than the threshold, the object is classified as invisible, but if the difference is less than the threshold, the object is classified as visible.

[0074] The system can search for the location of a set of features (e.g., edges / corners) in a two-dimensional projection of an object. For example, Harris Corner Detection can be used to search and identify features of an object. In some implementations, the system can convert the image to grayscale and apply filtering to smooth the noise of the image. The system can use the Sobel Operator to find the x and y gradient values ​​of each pixel in the image and consider an N×N (e.g., 3×3, 4×4, etc.) window surrounding each pixel to calculate the characteristics of the corners / edges. The calculated characteristics of the corners / edges can be a corner strength function, such as a Harris value. Pixels with Harris values ​​greater than a specified threshold can be confidently identified as corners / edges of an object.

[0075] In some implementations, the system may search for a threshold (e.g., sufficient) subset of the edges of a reference version of an object in a two-dimensional projection. In order to ensure that the edge detected in the actual presentation of the object is the same edge detected in the reference version of the object, the system may compare the average color of the pixels at the location of the edge in the two-dimensional projection with the average color of the pixels at the location of the edge in the reference version of the object. The match can be verified relative to a threshold based on a comparison between the average colors of corresponding pixels from the two-dimensional projection and the reference object. For example, a representation of the difference between the average colors of corresponding pixels from the two-dimensional projection and the reference object is calculated and an empirically determined threshold is applied to the calculated difference to determine a valid match. In some implementations, other visual features identified near the reference edge can be used to distinguish between the various reference corners and / or to ensure that the actual edge detected in the actual presentation of the object corresponds to the reference corner.

[0076] At 308, the system classifies the presentation of the object in the virtual environment based on whether the two-dimensional projection of the object matches the reference version of the object. In some implementations, the system classifies the presentation of the object as visible (310). The system increments the visibility count of the object (312). The system can repeat the process described about 302 to 308. For example, the system can continue to check the visibility of the object regularly for a certain duration according to 302 to 308. The duration of the regular inspection can be, for example, about 1 second. During the duration of the regular inspection of the visibility of the object, the system can determine that the object continues to meet the visibility condition and is classified as visible. In some implementations, the system determines that the sequential increment number of the visibility count meets the threshold visibility count (314). When the system determines that the sequential increment number meets the threshold visibility count, the system can align the presentation of the object (316). Aligning the presentation of the object can include providing confirmation to the publisher of the presented object.

[0077] In some implementations, the system classifies the presentation of the object within the virtual environment as a non-visible rendering of the object (318). In some implementations, the system determines that a sequentially increasing number of visibility counts satisfies a threshold non-visibility count (320). The system may iterate the process described with reference to 302 to 308 periodically (e.g., multiple times per second), wherein each non-visibility count is incremented when the system classifies the object as non-visible. In some implementations, the system determines that a counted number of visibility counts satisfies a threshold non-visibility count (322). For example, the number may be a sequential increment of the non-visibility count. In another example, the count of non-visibility may be a non-sequential increment, such as a cumulative non-visibility count that may be measured over a duration. In some implementations, when the non-visibility count satisfies the threshold non-visibility count, the system may provide an alert regarding the non-visibility of the object (324). In some implementations, an alert may be provided to a game engine to uninstall the object. For example, when the system determines that an object is classified as non-visible a threshold number of times, the system may provide an alert to swap the object for a different object.

[0078] In some implementations, a set of visibility conditions includes a series of checks on visibility of objects within the virtual environment to the user (eg, as described in reference 302 ( Figure 3 ) described in ). A set of visibility conditions may include a set of checks ordered in sequence, where each visibility check must be checked before the next visibility check can be performed by the system. If the visibility condition is not met, the system may not proceed to the next verification step of the set of visibility conditions. Figure 4 is a flow chart of an example process 400 for determining that the presentation of an object satisfies a set of visibility conditions.

[0079] At 402, the system verifies visibility of the object. In some implementations, checking visibility of the object includes receiving a rendering confirmation from a processor (e.g., a GPU or a CPU) that the object is rendered into a computing environment. For example, a game engine may provide instructions to the processor to render the object in a virtual environment. The rendered object may be outside a viewport visible to a user. For example, the rendered object may be within the computing environment but outside the user's field of view. In another example, the rendered object may be outside the user's viewport but may appear as a shadow within the user's field of view. The rendered object may be within a viewport visible to the user. In instances where the system determines that the object is not visible in the viewport, the system determines that the object does not meet a visibility condition.

[0080] At 404, the system determines that the viewing angle of the object satisfies a threshold angle criterion. The viewing angle can be determined as the angle from the surface normal of the object (e.g., Figure 2The angle of the object (normal 216 in the image) to the user's viewpoint. For example, the object can be a two-dimensional display in a virtual computing environment, where the normal is defined from the two-dimensional surface. In some implementations, the viewing angle is compared to a threshold angle criterion (e.g., a threshold viewing angle or a threshold viewing angle range), where viewing angles exceeding the threshold viewing angle or exceeding the viewing angle range are determined by the system to be non-visible. For example, the viewing angle can be set to be non-visible when greater than 55 degrees according to a standard (e.g., an Interactive Advertising Bureau (IAB) standard). In instances where the system determines that the viewing angle of the object does not meet the threshold angle criterion, the system determines that the object does not meet the visibility condition.

[0081] At 406, the system determines whether the object is completely or partially on the screen. In some implementations, determining whether the object is on the screen (e.g., within the user's field of view) includes determining that pixels included in the rendered object have corresponding coordinates that coincide with the user's field of view within the computing environment. For example, the system can map the object's world coordinates to the coordinates of the viewport to determine whether the object's coordinates are within the boundaries of the viewport. In instances where the system determines that the object is not on the screen, the system determines that the object does not meet the visibility condition. In instances where the system determines that the object is partially on the screen, the system can also determine that the portion of the object determined to be on the screen does not meet a minimum threshold for the object to be considered visible.

[0082] At 408, the system determines whether one or more features of an object within the user's field of view are occluded by one or more other objects located between the user's viewpoint and the object, such that the one or more other objects obstruct the user's view of the object. In some implementations, as part of determining whether the one or more other objects obstruct the user's view of the object, the system may determine whether the one or more other objects satisfy a transparency threshold. For example, if the one or more other objects are sufficiently transparent, as indicated by satisfying the transparency threshold, the one or more other objects may not be considered to obstruct features of the object. In some implementations, determining whether the one or more other objects obstruct the user's view of the object includes using a ray casting technique, for example, as described in reference Figure 5 described in further detail.

[0083] As in Figure 5 500, an object 502 may be viewed by a user 508 (e.g., Figure 5506). In some implementations, a ray cast (e.g., ray cast 510) can be initiated from the viewpoint of the user 508 toward a point on the object 502. For example, a ray cast can be initiated from the user's viewport toward a point on the object, where the point of the object can be a corner and / or edge of the object and a central feature of the object (e.g., a center point). In instances where the ray cast is intercepted by another object, the system can recursively initiate another ray cast from the other object toward the point on the object. In some implementations, a ray cast can alternatively (or additionally) be initiated from a point on the object 502 toward the viewpoint of the user 508. For example, a ray cast can be initiated from a point on the object to the user's viewport, where the point of the object can be a corner and / or edge of the object and a central feature of the object (e.g., a center point). In instances where the ray cast is intercepted by another object, the system can recursively initiate another ray cast from the other object toward the user's viewport.

[0084] In instances where the system determines that one or more other objects are located between the object and the user's viewport, the system may check the transparency values ​​of the one or more occluding objects. For example, the system may check the RGBA values ​​and / or physical properties of the one or more objects determined to be between the user's viewport and the object. When one or more objects are determined to be transparent based on the transparency values ​​of the one or more objects, the system may determine that the threshold transparency is met. In instances where one or more occluding objects located between the user's viewpoint and a feature of the object meet the transparency threshold, the system may determine that the feature is not occluded. In some implementations, the system may determine that the object is occluded if at least one feature of the object (e.g., two or more features of the object) is occluded. The features of the object may be, for example, points, regions, corners, edges, etc. of the object. For example, if at least one corner or edge of the object is occluded, the system may determine that the object is occluded. In some implementations, if a feature of the object located at the center (e.g., a center point) is occluded, the system may determine that the object is occluded. In instances where the system determines that one or more features of an object within the user's field of view are occluded, the system determines that the object does not meet the visibility condition.

[0085] refer to Figure 4At 410, the system verifies that the dimensionality of the object satisfies a threshold dimensionality. In some implementations, the system verifies the threshold dimensionality by determining that a pixel ratio (e.g., a percentage) of pixels included in the object compared to pixels on a screen (e.g., pixels of a viewport) satisfies a threshold. For example, the threshold pixel percentage of the object's pixels can be about 1.5% of the pixels on the screen, where an object having less than about 1.5% of the number of pixels on the screen does not satisfy the threshold dimensionality. For example, the system can verify the dimensionality of the object using polygon-based calculations, where the area of ​​a polygon is calculated using equation (1) to (i) calculate the area of ​​the object on the screen (e.g., within the viewport) and (ii) calculate the full area of ​​the object by using the corner vertices of the viewport. Figure 6 An example of an object 600 as seen through a viewport 602 of a client device 604 is depicted in FIG. The system may normalize the viewport space, for example, where the lower left corner is (0,0) and the upper right corner is (1,1). Equation (1) yields the percentage of the object area to the viewport area.

[0086] A(poly)=|(x0*y1-x1*y0)+(x1*y2-x2*y1)+…+(x n *y0-x0*y n )÷2 (1)

[0087] Where (x n ,y n ) are the corresponding coordinates of the n vertices of the polygon.

[0088] In some implementations, the system verifies the threshold dimensionality by determining the number of object pixels that are pixels on the screen. For example, the threshold object pixel number can be that at least about 50% of the object pixels are pixels on the screen. In other words, at least about half of the object pixels are rendered within the user's viewport. In instances where the system determines that the dimensionality of the object does not meet the threshold dimensionality, the system determines that the object does not meet the visibility condition. For example, the system can use polygon-based calculations to verify the dimensionality of the object, where the area of ​​the polygon is calculated using equation (1), where the object coordinates are constrained to the viewport boundaries, and the area of ​​the object visible on the screen is calculated. Dividing this value by the unconstrained area of ​​the object yields the percentage of the object that is visible in the viewport.

[0089] At 412, the system determines that the average brightness of the object satisfies the threshold brightness. In some implementations, the system determines the average brightness of the object by calculating the average brightness of the pixels included in the object and converting the average brightness value into a representative value. For example, the average brightness can be the mean, median, mode, or central tendency of the brightness of the pixels included in the object. Equation (2) is an example that can be used to calculate the brightness (e.g., lumens) per pixel:

[0090] Pixel Lumen = 0.299*R+0.587*G+0.114*B (2)

[0091] Where R, G, B are the red, green, and blue components of the pixel. For example, the representative value of the average brightness value can be a hexadecimal value. In some implementations, the system determines that the average brightness value of the object meets the threshold brightness by comparing the representative value with the threshold brightness value. In another example, the representative value of the average brightness of the object can be compared with a threshold brightness value and / or a threshold darkness value. In another example, the representative value can be compared with a certain brightness value range. In an instance where the representative value does not meet the threshold brightness value, the system can determine that the object does not meet the visibility condition.

[0092] At 414, the system determines that the two-dimensional projection of the object matches the reference version of the object based on a comparison of the average color of the features of the reference version of the object with the average color of the features in the two-dimensional projection of the object. For example, the reference version of the object can be obtained from the content distribution system 152 (e.g., in the content database 168).

[0093] In some implementations, the steps described by reference are performed by a processor. The processor can be, for example, a GPU, a CPU, or a TPU. Sometimes, one or more of the calculations are selectively performed by the processor based on, for example, calculation speed, available processing power, calculation type, etc. In some implementations, the process can be performed in parallel by two or more processors. In some implementations, determining pixel-based visibility conditions and / or classifying the presentation of objects is performed by a GPU. In some implementations, calculations that require less than a threshold calculation are performed on the CPU, and calculations that require more than a threshold calculation are performed on the GPU.

[0094] In some implementations, executing reference Figure 3 and Figure 4 The operations described above provide increasing computational complexity. For example, the object visibility pipeline can be Figure 3 Step 302 is initiated and executed in sequence Figure 4 402, 404, 406, 408, 410 and 412 in the process. Then it can be Figure 4 Step 414, wherein reference Figure 3 The operation of step 414 is described in more detail with reference to steps 304, 306, 308 and 310 / 318. After the classification in steps 310 / 318, the system may perform additional steps 312, 314 and 316 or steps 320, 322 and 324, respectively. Of course, the order of the operations performed may vary.

[0095] In some implementations, the reference may be iteratively performed Figure 3 and Figure 4 Some or all of the described operations may continue to check visibility of an object for the duration that the object is rendered in the virtual environment.

[0096] In some implementations, reference Figure 3 and Figure 4 The described operations may be used by a system to determine object visibility for multiple (eg, two or more) objects rendered within a virtual environment.

[0097] Figure 7 7 is a block diagram of an example computer system 700 that can be used to perform the operations described above. System 700 includes a processor 710, a memory 720, a storage device 730, and an input / output device 740. Each of components 710, 720, 730, and 740 can be interconnected, for example, using a system bus 750. Processor 710 is capable of processing instructions for execution within system 700. In one implementation, processor 710 is a single-threaded processor. In another implementation, processor 710 is a multi-threaded processor. Processor 710 is capable of processing instructions stored in memory 720 or on storage device 730.

[0098] The memory 720 stores information within the system 700. In one implementation, the memory 720 is a computer readable medium. In one implementation, the memory 720 is a volatile memory unit. In another implementation, the memory 720 is a non-volatile memory unit.

[0099] The storage device 730 can provide mass storage for the system 700. In one implementation, the storage device 730 is a computer-readable medium. In various implementations, the storage device 730 may include, for example, a hard disk device, an optical disk device, a storage device shared by multiple computing devices over a network (e.g., a cloud storage device), or some other mass storage device.

[0100] The input / output device 740 provides input / output operations for the system 700. In one implementation, the input / output device 740 may include one or more of a network interface device (e.g., an Ethernet card), a serial communication device (e.g., and an RS-232 port), and / or a wireless interface device (e.g., and an 802.11 card). In another implementation, the input / output device may include a driver device configured to receive input data and send output data to a peripheral device 760 (e.g., a keyboard, a printer, and a display device). However, other implementations may also be used, such as a mobile computing device, a mobile communication device, a set-top TV client device, etc.

[0101] Although already Figure 7An example processing system is described in the specification, but the subject matter and functional operations described in this specification may be implemented in other types of digital electronic circuit systems (circuitry) or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents) or in a combination of one or more of them.

[0102] An electronic document may, but need not, correspond to a file. A document may be stored in a portion of a file that holds other documents, in a single file dedicated to the document in question, or in multiple coordinated files.

[0103] The embodiments of the subject matter and operations described in this specification may be implemented in a digital electronic circuit system or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents) or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, that is, one or more modules of computer program instructions, which are encoded on a computer storage medium for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or in addition, program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical or electromagnetic signal), which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them, or may be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. In addition, although a computer storage medium is not a propagation signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagation signal. The computer storage media may also be, or be included in, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

[0104] The operations described in this specification may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0105] The term "data processing device" covers all kinds of devices, apparatuses and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or a plurality of the foregoing or a combination of the foregoing. The device may include a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The device and the execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0106] The data processing equipment can also take the form of game devices.Game devices are devices that enable users to participate in game applications, for example, in the device, users can control one or more roles, avatars or other rendered content presented in the game application.Game devices typically include computer processors, hardware memory devices and controller interfaces (physical or visually rendered in displays) that enable users to control the content rendered by the game application.Game devices can store and execute game applications locally, or execute game applications (for example, online game applications) that are at least partially stored and / or served by cloud servers. Similarly, game devices can be docked with game servers that execute game applications and "stream" game applications to game devices.Game devices can be tablet devices, mobile telecommunications devices, computers or another device that also performs other functions in addition to executing game applications.

[0107] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, be stored in a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed over multiple sites and interconnected by a communications network.

[0108] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and the apparatus can also be implemented as, a special purpose logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0109] For example, processors suitable for executing computer programs include general and special microprocessors, and any one or more processors of any kind of digital computer. Generally, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or operatively coupled to receive data from the one or more large-capacity storage devices or transfer data to the one or more large-capacity storage devices or both. However, a computer does not have to have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few. Devices suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0110] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0111] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include local area networks ("LANs") and wide area networks ("WANs"), interconnected networks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0112] A computing system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from the user). Data generated at the client device (e.g., the result of the user interaction) can be received from the client device at the server.

[0113] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but should be interpreted as descriptions of features that are peculiar to a particular embodiment of a particular invention. Certain features described in the context of a separate embodiment in this specification may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may involve a sub-combination or a change in the sub-combination.

[0114] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0115] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions set forth in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A method for determining visibility of an object to a user in a virtual environment, the method comprising: include: determining that presentation of the object within the virtual environment satisfies a set of visibility conditions; capturing a two-dimensional projection of the object as rendered in the virtual environment; determining that the two-dimensional projection of the object matches the reference version of the object based on a comparison of an average color of a feature of the reference version of the object and the average color of the feature in the two-dimensional projection of the object; as well as Classifying a presentation of the object in the virtual environment based on whether the two-dimensional projection of the object matches the reference version of the object, comprising: responsive to determining that the two-dimensional projection of the object matches the reference version of the object, classifying the presentation of the object within the virtual environment as a visual rendering of the object; as well as In response to determining that the two-dimensional projection of the object does not match the reference version of the object, the presentation of the object within the virtual environment is classified as a non-visible rendering of the object.

2. The method of claim 1, wherein the set of visibility conditions include: A rendering confirmation is verified from one or more processors for the object within the virtual environment.

3. The method of claim 1 or 2, wherein the set of visibility conditions include: A determination is made that a viewing angle of the object in the virtual environment within the field of view of the user satisfies a threshold angle criterion regarding a surface normal of the object in the virtual environment relative to the field of view of the user.

4. A method as claimed in any one of the preceding claims, wherein the set of visibility conditions include: Determining that object pixels constituting the object include coordinates that coincide with the field of view of the user within the virtual environment.

5. A method as claimed in any one of the preceding claims, wherein the set of visibility conditions include: Determining that one or more features of the object within the field of view of the user are not obscured by one or more other objects.

6. The method of claim 5, wherein determining that the one or more features of the object within the field of view of the user are not obscured by one or more other objects comprises determining that the one or more other objects determined to be located between the field of view of the user and the one or more features of the object satisfy a transparency threshold.

7. The method of claim 5 or 6, wherein the one or more features of the object include at least one corner feature of the object and a center feature of the object.

8. A method as claimed in any one of the preceding claims, wherein the set of visibility conditions include: Verifying that the dimension of the object satisfies a threshold dimension includes: Determining that a pixel ratio of the object pixel to the pixel on the screen satisfies a threshold; and Determine the threshold object pixel count to include pixels on the screen.

9. A method as claimed in any one of the preceding claims, wherein the set of visibility conditions include: It is determined that the average brightness of the object satisfies a threshold brightness.

10. The method of claim 9, wherein determining the average brightness of the object satisfies the threshold brightness include: Calculating the average brightness of pixels constituting the object; converting the average brightness into a representative value; as well as The representative value is compared to a threshold brightness value.

11. The method of any one of the preceding claims, wherein classifying the presentation of the object in the virtual environment further comprises: include: responsive to classifying the presentation of the object within the virtual environment as the visible rendering of the object, incrementing a visibility count of the object; Determining that a sequentially incremented number of times the visibility count of the object satisfies a threshold visibility count; as well as The representation of the object is registered.

12. The method of any one of the preceding claims, wherein classifying the presentation of the object in the virtual environment further comprises: include: responsive to classifying the presentation of the object within the virtual environment as a non-visible rendering of the object, incrementing a non-visibility count of the object; determining that the sequentially increasing number of times of the non-visibility count satisfies a threshold non-visibility count; and An alert is provided regarding the non-visibility of the object.

13. A method as claimed in any one of the preceding claims, wherein determining that the two-dimensional projection of the object matches the reference version of the object include: Calculating a hash of the two-dimensional projection; as well as The hash of the two-dimensional projection is compared to the hash of the reference version of the object.

14. The method of claim 13, wherein computing the hash of the two-dimensional projection of the object and the hash of the reference version comprises computing an average hash, include: calculating an average color value of at least a portion of the two-dimensional projection; encoding each pixel of the two-dimensional projection based on whether a color value of the pixel is at least the average color value; creating a bit string based on the encoded pixels; as well as Convert the bit string to a hexadecimal value.

15. The method of claim 14, wherein determining that the two-dimensional projection of the object matches a reference version of the object comprises determining a difference between the hexadecimal value and a reference hexadecimal value representing the reference version of the object.

16. The method of any one of claims 1 to 12, wherein determining that the two-dimensional projection of the object matches a reference version of the object include: identifying locations of a set of edges in the reference version of the object; searching for the positions of the set of edges in the two-dimensional projection; as well as An average color of pixels at the location in the two-dimensional projection is compared to the average color of pixels at the location of the edge in the reference version of the object.

17. One or more non-transitory computer storage media encoded with computer program instructions that, when executed by one or more computers, cause the one or more computers to perform operations, the operations include: Determining that presentation of an object within the virtual environment satisfies a set of visibility conditions; capturing a two-dimensional projection of the object as rendered in the virtual environment; determining that the two-dimensional projection of the object matches the reference version of the object based on a comparison of an average color of a feature of the reference version of the object and the average color of the feature in the two-dimensional projection of the object; as well as Classifying a presentation of the object in the virtual environment based on whether the two-dimensional projection of the object matches the reference version of the object, comprising: responsive to determining that the two-dimensional projection of the object matches the reference version of the object, classifying the presentation of the object within the virtual environment as a visual rendering of the object; as well as In response to determining that the two-dimensional projection of the object does not match the reference version of the object, the presentation of the object within the virtual environment is classified as a non-visible rendering of the object.

18. A system, include: One or more computers and one or more storage devices, wherein the one or more storage devices store operable instructions, and when the instructions are executed by the one or more computers, the one or more computers are caused to perform operations, wherein the operations include: Determining that presentation of an object within the virtual environment satisfies a set of visibility conditions; capturing a two-dimensional projection of the object as rendered in the virtual environment; determining that the two-dimensional projection of the object matches the reference version of the object based on a comparison of an average color of a feature of the reference version of the object with the average color of the feature in the two-dimensional projection of the object; and Classifying a presentation of the object in the virtual environment based on whether the two-dimensional projection of the object matches the reference version of the object, comprising: responsive to determining that the two-dimensional projection of the object matches the reference version of the object, classifying the presentation of the object within the virtual environment as a visual rendering of the object; and In response to determining that the two-dimensional projection of the object does not match the reference version of the object, the presentation of the object within the virtual environment is classified as a non-visible rendering of the object.

Images