Tamper-proof interactive data

Abstract

Methods, systems, and apparatus, including a computer program encoded on a computer storage medium, for verifying an interaction with a false presentation element. In one aspect, a method includes: receiving a presentation notification and a declaration of a presentation element defined in an active window on a client device, detecting an interaction with the presentation element at the client device, determining whether the interaction occurred at a declared location of the presentation element within the active window, and processing the interaction, including: in response to determining that the interaction occurred: capturing a screenshot of the active window on the client device; verifying the visual appearance of the presentation element in the screenshot using the declared appearance of the presentation element, and generating a proof of the interaction, thereby verifying the interaction. In response to determining that the interaction did not occur, refraining from generating the proof of the interaction.

Description

Technical Field

[0001] The present invention generally relates to methods, systems and apparatus for verifying interaction with false presentation elements, and more particularly to techniques related to data processing and generating unforgeable proof tokens to verify human interaction with content while protecting user privacy. Background Art

[0002] Programmatic clicks can lead to the widespread dissemination of false information, which can seriously deteriorate users' online experience and waste computing resources that could have been used to spread accurate information. Summary of the invention

[0003] This specification describes technologies related to data processing and generating unforgeable proof tokens to authenticate human interaction with content while protecting user privacy.

[0004] In general, one innovative aspect of the subject matter described in this specification can be embodied in a method comprising the following acts: receiving a presentation notification of a presentation element defined in an active window on a client device; receiving a declaration of a presentation element defined in an active window on the client device; detecting an interaction with the presentation element at the client device; determining, based on the received declaration, whether the interaction occurred at a declared location of the presentation element within the active window; and based on determining whether the interaction occurred at the declared location, processing the interaction. Processing the interaction comprises: in response to determining that the interaction occurred at the declared location of the presentation element: capturing a screenshot of the active window on the client device, verifying the visual appearance of the presentation element in the screenshot with the declared appearance of the presentation element, and generating an interaction proof for the interaction with the presentation element, thereby verifying the interaction. In response to determining that the interaction did not occur at the declared location of the presentation element, avoiding generating an interaction proof for the interaction with the presentation element.

[0005] Other embodiments of this aspect include corresponding systems, apparatus, and computer programs configured to perform the actions of the method encoded on a computer storage device.

[0006] These and other embodiments may each optionally include one or more of the following features. In some implementations, the presentation notification includes one or more of i) the location of the presentation element in the active window, ii) the stable ID and URI of the presentation element, and iii) the site or domain where the presentation element is presented.

[0007] In some implementations, receiving a declaration of a rendering element includes receiving a verified appearance of the rendering element. The verified appearance may include a unique stable ID and / or a stable URL and a verified shape.

[0008] In some implementations, verifying the visual appearance of the presentation element in the screenshot using the declared appearance of the presentation element includes using an image matching algorithm.

[0009] In some implementations, detecting an interaction includes receiving a selection signal specifying XY coordinates of a user click within the active window.

[0010] In some implementations, the method further includes aggregating interaction proofs for multiple interactions with multiple instances of the presentation element. The interaction proofs may include one or more of i) a stable ID and / or URI of the declared appearance, ii) a timestamp of the interaction, iii) a device integrity token, iv) a digital signature of the client device, v) XY coordinates of the interaction, and vi) a client device public key. The stable ID and / or URI of the declared appearance may include a user intent of the interaction, where the user intent may correspond to an action conveyed by the appearance of the presentation element.

[0011] In some implementations, the method further includes: in response to the interaction proof including the user intent, modifying the report of the interaction based on the user intent. The method may also include: in response to determining that the interaction is different from the user intent, determining that the presentation element is a false presentation element and triggering a false interaction response. The method may also include, in response to triggering the false interaction response, intercepting a path of the client device to a login page of the false presentation element.

[0012] In some implementations, in response to determining that the interaction did not occur at a declared location of the presentation element, the method includes triggering a false interaction response.

[0013] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. Verifying that a real user has interacted with a presentation element (e.g., a button, thumbs up / down, etc.) at a remote client device can safeguard the authenticity of a ranking mechanism for a digital component (e.g., a news article) and prevent bad actors from manipulating the ranked set of digital components. By using third-party verification to verify the appearance of a presentation component on a remote client device, the system can ensure that an actual user is interacting with the intended presentation element (e.g., a button) and not a hidden or manipulated element that may be attempting to initiate a secondary action or hijack the interaction entirely. The subject matter can be used to quickly identify pseudo-presentation elements distributed through an online system using a verified proof token that protects user privacy. Using a verified proof token, the system can ensure that the detected interaction was performed by an actual user on a remote client device and that the content was actually provided to the user at the moment of the detected interaction. This type of verification uses verifiable digital signatures created by one or more devices that cannot be forged by a human operator to verify interaction with content on a remote client device.

[0014] By ensuring that the presentation user interface (UI) element being processed is valid before processing subsequent actions (e.g., linking to a login page, adjusting the underlying ranking mechanism associated with the presentation element, etc.), processing, memory, and resources used to provide content to end users are reduced because resources are not used to store, distribute, or present false information associated with false presentation elements. In addition, the use of proof tokens can help classify presentation elements presented by digital components as false in a more reliable and efficient manner, reducing the resources required to store and distribute misleading information. For example, the proof token is configured to ensure that the interaction with the presentation element is performed by a user who is actually exposed to and / or clicks on the actual presentation element, while still maintaining the user's privacy. Therefore, it can be confirmed that the interaction is legitimate and not by a malicious actor (e.g., an entity attempting to harm a specific content distributor). Therefore, resources are not wasted on processing illegal false presentation element alerts, and, since each presentation element can be verified as legitimate or illegal using proof tokens, the result metrics and / or reports providing information about digital components containing false presentation elements become more accurate and are created more efficiently. Additionally, the techniques described in this document enable systems to warn users when they are presented with content that may be false or misleading, thereby providing users with the ability to avoid misleading or malicious content.

[0015] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a block diagram of an environment in which a digital component system distributes digital components.

[0017] Figure 2A and 2B A diagram of the environment in which the interaction of presentation elements is verified.

[0018] Figure 3 is a flow diagram illustrating an example process for validating interactions with a presentation element.

[0019] Figure 4 is a block diagram of an example computer system. DETAILED DESCRIPTION

[0020] Overview

[0021] In general, this document relates to computer-implemented methods and systems that use verified presentation element appearance and interaction attestations to verify interactions with false presentation elements (e.g., misleading and / or invisible buttons) that are presented with a digital component (e.g., a video clip, an audio clip, a multimedia clip, an image, text, or another content unit). An operating system or other trusted application or web browser may report a user's interaction with a presentation element in an active window on a client device (e.g., in an application environment or a website), along with information related to the appearance of the presentation element in the active window, such as a screenshot of the active window. The appearance of the presentation element may be verified by a third party and include a statement of the appearance accessible to the application and / or web browser that displayed the presentation element. The presentation element may be verified using a third-party statement of the presentation element's appearance and a screenshot of the active window of the client device to verify that the user's interaction was intentional. Verifying the interaction as a human interaction (rather than an automated or bot interaction) and verifying the presentation element as a non-false presentation element may limit, modify, or prevent the false presentation element from being used to distribute a digital component that displays false information, as described herein.

[0022] For example, an operating system (OS) and browser for an application can verify user interactions with presentation elements in a privacy-preserving manner so that false presentation elements can be identified using a central aggregation server that implements privacy-preserving aggregate measurements. An aggregated false presentation element report can be generated from the verified user interactions with presentation elements, which can be used to alert the user to the presence of potential false presentation elements in an active window on a client device. Systems and methods for verifying user interactions with presentation elements are described below with reference to Figure 2A , 2B and 3 are described in further detail.

[0023] Operating environment example

[0024] Figure 1 1 is a block diagram of an environment 100 in which a digital component distribution system 150 distributes digital components. Example environment 100 includes a data communication network 105, such as a local area network (LAN), a wide area network (WAN), the Internet, a mobile network, or a combination thereof. Network 105 connects client devices 110, publishers 130, websites 140, digital component distribution system 150, detection system 170, aggregation system 180, and reporting system 190. Example environment 100 may include different client devices 110, publishers 130, and websites 140. In some implementations, environment 100 may also include multiple digital component distribution systems 150.

[0025] A website 140 is one or more resources 145 associated with a domain name and hosted by one or more servers. An example website is a collection of web pages formatted in HTML that may contain text, images, multimedia content, and programming elements such as scripts. Each website 140 is maintained by a publisher 130, which is an entity that controls, manages, and / or owns the website 140.

[0026] A resource 145 is any data that can be provided over a network 105. A resource 145 is identified by a resource address (e.g., a universal resource locator (URL)) associated with the resource 145. Resources include HTML pages, word processing documents and portable document format (PDF) documents, images, videos, and feeds, to name a few examples. Resources can include content, such as words, phrases, images, and sounds, which can include embedded information (such as meta information in a hyperlink) and / or embedded instructions (such as scripts).

[0027] Client device 110 is an electronic device capable of communicating over network 105. Example client devices 110 include personal computers, mobile communication devices (eg, smart phones), and other devices capable of sending and receiving data over network 105.

[0028] The client device 110 typically includes an application 112, such as a web browser and / or a native application, to facilitate sending and receiving data over the network 105. A native application is an application developed for a specific platform or a specific device. A publisher 130 can develop and provide (e.g., for download) a native application to the client device 110. In some implementations, the client device 110 is a digital media device, for example, a streaming device that plugs into a television or other display to stream video to the television. The digital media device may also include a web browser and / or other application that streams video and / or presents resources.

[0029] For example, in response to a user of client device 110 entering a resource address of resource 145 in an address bar of a web browser or selecting a link referencing the resource address, the web browser may request resource 145 from a web server hosting website 140 of publisher 130. Similarly, a native application may request application content from a remote server of publisher 130.

[0030] Some resources 145, application pages, or other application content may include a digital component slot for presenting a digital component with a resource 145 or application page. In this document, the phrase "digital component" refers to a discrete unit of digital component or digital information (e.g., a video clip, an audio clip, a multimedia clip, an image, text, or another content unit). A digital component may be electronically stored in a physical storage device as a single file or a collection of files, and the digital component may take the form of a video file, an audio file, a multimedia file, an image file, or a text file, and include advertising information, such that an advertisement is a type of digital component. For example, a digital component may be content that is intended to supplement the content of a web page or other resource presented by an application 112. More specifically, a digital component may include digital content related to resource content (e.g., a digital component may be related to the same subject of web page content, or to a related subject). Thus, the digital components provided by the digital component distribution system 150 may supplement and generally enhance web page or application content.

[0031] When application 112 loads resource 145 (or application content) that includes one or more digital component slots, application 112 may request a digital component for each slot from digital component distribution system 150, which in turn requests the digital component from digital component provider 160. Digital component provider 160 is an entity that provides digital components for rendering with resource 145.

[0032] In some cases, digital component distribution system 150 may also request digital components from one or more digital component partners 157. Digital component partners 157 are entities that select digital components 129 (not shown) on behalf of digital component providers 160 in response to digital component requests.

[0033] Digital component distribution system 150 may select a digital component for each digital component slot based on various criteria. For example, digital component distribution system 150 may select a digital component from digital components received from digital component providers 160 and / or digital component partners 157 based on relevance to resource 145 (or application content), performance of the digital component (e.g., rate at which a user interacts with the digital component, etc.), etc. Digital component distribution system 150 may then provide the selected digital component to client device 110 for presentation with resource 145 or other application content.

[0034] When the application 112 presents the digital component, the application 112 (which may be a browser) may store impression data 114 for the presentation of the digital component. The impression data 114 for a particular presentation of the digital component may include a URL or domain for a landing page for the digital component (e.g., a page linked by the digital component and presented to the user by the application / browser when the user clicks on or interacts with the digital component). In some implementations, the impression data 114 for a particular presentation of the digital component may include, for example, a phone number for a click-to-call advertisement, or, for example, a store address for an in-store visit advertisement.

[0035] The impression data 114 for a particular presentation of a digital component may also include one or more identifiers for the digital component, event-level data associated with the impression, an expiration time that specifies when the impression data is deleted from the client device 110, and / or a reporting URL or domain to which conversion reports for the digital component are sent. This data may be provided by the digital component, for example, as metadata for the digital component or an anchor tag for the digital component. As described in more detail below, the application 112 may store impression data that is blindly signed by the detection system 170, rather than or in addition to storing the raw impression data for each impression of the digital component.

[0036] The application 112 may also store conversion data 116 in response to the detected conversion. The conversion of the digital component is to complete a specified user action after the digital component is presented to the user and / or is interacted with by the user (e.g., clicked or clicked). The conversion data 116 for the conversion may include data indicating the type of conversion, as some conversions may have multiple types. The conversion type may specify a subsequent action after the conversion is completed. For example, a conversion may be to add an item to an online shopping cart. In this example, a first type of conversion may be to add an item without checking out (e.g., not completing the purchase), and a second type of conversion may be to check out. Other examples of conversions include a phone call to an advertiser, installation of an application, a visit to a local store owned by an advertiser, and the like. As described in more detail below, the conversion data for the conversion may be a set of one or more bits indicating the type of conversion, and the application may store the conversion data blindly signed by the digital component distribution system 150 (or another appropriate system).

[0037] In some implementations, the transformation of the digital component includes interaction data for a user's interaction with a presentation element presented in the digital component. The interaction data 117 including the user's interaction with the presentation element may include data indicating the type of interaction (e.g., click, swipe, finger up / finger down, etc.) with the presentation element (e.g., button). Figure 2A-2B and Figure 3 Describes interaction with presentation elements in more detail.

[0038] The detection system 170 can evaluate the detection signals received from the client device 110 to determine whether the impression, interaction, and / or conversion is valid or invalid. The detection signals can be application dependent and can vary in different implementations. The application 112 (or the operating system on the client device 110) can include an API that enables the digital component to call the application 112 (or the operating system on the client device 110) to collect the detection signals and provide the detection signals to the detection system 170.

[0039] In addition to the detection signal, application 112 may also send blind impression data for the impression to detection system 170. If detection system 170 determines that the impression is valid (e.g., the identified content was actually rendered by the client device), detection system 170 may sign the blind impression data and provide the signed blind impression data to application 112. Example techniques for generating and signing blind impression data are described below.

[0040] Aggregation system 180 may generate aggregated network measurements based on data received from client devices 110. In the following description, techniques for generating and providing measurement data elements are described as being performed by application 112, which may be a web browser or a native application. However, in some implementations, an operating system of client device 110 may generate and send measurement data elements. In such implementations, web browsers and applications on client device 110 may be configured to report impressions and conversions to the operating system. The operating system may perform each of the operations for reporting impressions and conversions described below that are performed by application 112.

[0041] Application 112 on client device 110 may provide measurement data elements including encrypted data representing network data to aggregation system 180. The network data may include impression data and / or conversion data for each conversion. For example, application 112 may generate and send to aggregation system 180 measurement data elements for each impression, user interaction, and / or conversion that occurred at client device 110. Aggregated network measurements may include, for each of one or more digital components, a total number of impressions, user interactions, and / or conversions across multiple client devices 110 for the digital component and (optionally) presentation elements displayed using the digital component.

[0042] The application 112 may use a (t, n) threshold scheme to generate data in the measurement data element. In some implementations, when the application 112 detects a transformation or receives transformation data for a transformation, the application 112 generates a group key (e.g., a polynomial function) based on the impression data and the transformation data for the transformation. The application may then generate a group member key that represents a portion of the group key and may be used to regenerate the group key only when a sufficient number of group member keys for the same impression and transformation pair are received. In this example, the measurement data element for the transformation may include a group member key generated by the application and a label corresponding to the impression and transformation pair. Each unique impression and transformation pair may have a corresponding unique label so that the aggregation system 180 may aggregate the measurement data elements for each impression and transformation pair using its label.

[0043] In some implementations, when the application 112 detects an interaction and / or conversion or receives interaction and / or conversion data for an interaction and / or conversion, the application 112 generates a group key (e.g., a polynomial function) based on the impression data, the interaction data, and the interaction data for the interaction and / or conversion. The application can then generate a group member key that represents a portion of the group key, and the group member key can be used to regenerate the group key only when a sufficient number of group member keys for the same impression and interaction / conversion pair are received. In this example, the measurement data element for the interaction / conversion can include the group member key generated by the application and a label corresponding to the impression and interaction / conversion pair. Each unique impression and interaction / conversion pair can have a corresponding unique label so that the aggregation system 180 can aggregate the measurement data elements for each impression and interaction / conversion pair using its label.

[0044] In a (t, n) threshold encryption scheme, the aggregation system 180 will need to receive at least t group member keys for the same impression and interaction / conversion pair in order to be able to decrypt the impression and conversion and / or interaction data. If fewer than t group member keys are received, the aggregation system 180 cannot decrypt the impression and conversion and / or interaction data. Once at least t measurement data elements for the same impression and conversion and / or interaction pair are received from the client device 110, the aggregation system 180 can determine the group key from the at least t group member keys and obtain the impression and conversion and / or interaction data from the group key.

[0045] Aggregation system 180 may determine a conversion amount for an impression and conversion pair based on the number of measurement data elements received, the measurement data elements comprising impression data, interaction data, and / or conversion data for an impression and interaction / conversion pair. For example, after obtaining impression, interaction, and conversion data using at least t group member keys, aggregation system 180 may determine a count of the number of group member keys received for an impression and interaction / conversion pair as the conversion amount. Aggregation system 180 may report the impression data, interaction data, conversion data, and the number of interactions and / or conversions to reporting system 190, which may be implemented as a reporting domain corresponding to a reporting URL for a digital component, which in turn, corresponds to the impression and conversion data.

[0046] False rendering element reporting and verification

[0047] The digital component distribution system 150 can distribute digital components from various digital component providers 160 to the client device 110. The digital component can include a presentation element (e.g., a button) with which a user can interact. The presentation element can be verified using appearance declarations and screenshots of the appearance of the presentation element captured in an active window of the client device 110. In addition, in order to protect the privacy of the reporting end user and to protect the interests of the digital component provider 160, the interaction with the presentation element can be verified and aggregated.

[0048] Figure 2A and 2B It is a block diagram of an environment for verifying the interaction of presentation elements. A false presentation element refers to a presented element, such as a button or other optional feature, where the presented digital component is invisible to the user or misleading. In some implementations, a false presentation element may trigger an interaction that the user does not expect. In one example, a hidden button (e.g., a code-defined interaction area of ​​a button that does not include a graphical visualization of a button presented on the screen) on a "close window" button (the user achieves the expected effect by selecting the "close window" button) may initiate a request to another application or web page instead of closing the current window or application. In another example, a misleading presentation element may be displayed as a "thumbs-up" icon, but a user's selection of the misleading presentation element may trigger a "vote down" function instead of the expected "vote up" function.

[0049] Figure 2A2 is a block diagram of an environment 200 in which presentation element interactions are verified. An interaction verification system 202 is configured to receive presentation notifications 204 and interaction alerts 206 from an operating system 208 on a client device 210. The interaction verification system 202 is also configured to receive a declaration 212 of a verified appearance of a presentation element 214 in an active window 216 of an application 218. Additionally, the interaction verification system 202 is configured to receive a screenshot 220 of an active window 216 displayed on the client device 210 from the operating system 208 of the client device 210 or another trusted application 218.

[0050] In some implementations, one or more of the processes described herein and performed by the interactive authentication system 202 may be performed by an operating system 208 or another trusted application 218 on the client device 210 .

[0051] The interactive verification system 202 may include an interactive determination module 222, a visual analysis module 224, and a proof generator 226. Each of the interactive determination module 222, the visual analysis module 224, and the proof generator 226 may include a computer storage readable medium, the computer storage readable medium being encoded with the functionality of performing the tasks described herein with reference to the corresponding module and / or one or more processors configured to perform the tasks. The interactive determination module 222 is configured to receive the presentation notification 204 and the interactive alert 206 as input. The presentation notification 204 may include the location of the presentation element 214 in the active window 216, the stable identification (stable ID) and / or uniform resource identifier (URI) of the presentation element 214, and the site or domain where the presentation element 214 is presented.

[0052] Rendering element 214 is an element that is visible in active window 216 (e.g., the active window on client device 210). Rendering notification 204 may be generated by an application that notifies operating system 208 that rendering element 214 is visible in active window 216. In another embodiment, rendering notification 204 may be generated by a web page / JavaScript that notifies a web browser that rendering element 214 is visible in the active window of a web page, for example, as described below with reference to Figure 2B described.

[0053] The presentation element 214 may be requested by the operating system 208 as part of or in addition to a request for the digital component 205. The request for the digital component may include the presentation element 214, for example, a news post with share, upvote, and downvote buttons. The digital component provider 234 may provide a mode for generating the presentation element 214 and a presentation control including a digital signature for the presentation element 214.

[0054] Each time the appearance of the active window changes, such as by a user scrolling, resizing, or otherwise changing the current view of the active window, a presentation notification 204 may be generated. Tracking the position of the presentation element 214 within the active window 216 may include tracking user scrolling, zooming functions, etc., where user interaction with the scrolling function, zooming function, etc. will trigger a signal to update the presentation notification 204. In some implementations, the presentation notification may be generated by a script such as a mutation observer or another script that can detect changes to a user interface (e.g., the DOM of a web page or the structure of a native application user interface).

[0055] Return to reference Figure 2A , the interaction 207 with the presentation element 214 can include, but is not limited to, a user selection, such as a click, a swipe, a pinch, a mouse hover, or other motion indicating selection of a presentation element within the area declared as occupied by the presentation element 214. The interaction 207 can be a user scrolling at least a portion of the digital component 205, such as scrolling to the bottom of a news article. The interaction 207 can be a user passively allowing playback of the digital component 205, such as playback of an audio, video, or multimedia file. In one example, the interaction 207 is detected when a user's finger touches and / or lifts from the touch screen of the client device 210.

[0056] In addition, the interaction determination module 222 receives the declaration 212. Receiving the declaration 212 of the presentation element 214 may include receiving a verified appearance of the presentation element 214. The verified appearance of the presentation element 214 may include a unique stable ID, such as an encrypted hash of a resource file, or an encrypted hash of the presentation RGB pixel values, and / or a stable URI, and a verified shape of the presentation element 214. In one example, the verified shape includes a size and a shape type, such as a square, rectangle, circle, etc. In another example, the verified shape includes a size range and a general category and / or descriptive identifier of the shape, such as a thumbs up shape, a stop sign shape, a check box shape. In another example, the verified shape can be represented by a bitmap (2D array) where 1 represents "part of a shape" and 0 otherwise. Such a bitmap can represent arbitrarily complex shapes.

[0057] The announcement 212 is not user-specific, e.g., not specific to a particular user or a particular client device 210. The announcement appearance of the presentation element 214 is consistent for a sufficiently large group of users across a variety of client devices. For example, the announcement appearance includes a size range and a color palette to accommodate possible variations between different client devices, display characteristics, and user preferences. In another example, the announcement appearance includes possible multiple languages ​​and other regional variations of the presentation element 214.

[0058] The statement 212 including the declared appearance of the presentation element is independently verified by a third party as not misleading. In one example, the presentation element in the application can be verified by the application store or other provider that lists the application, for example, as part of the review and approval process for providing the application. In another example, the presentation element in the web resource can be verified by the privacy group of the web browser vendor and / or website publisher.

[0059] The third-party reviewer can determine whether the rendered button meets the review criteria, for example, that interaction with the rendered button will result in an action that matches the user's intent, and digitally sign a rendered element resource (e.g., a png or bitmap file) that can be used to verify the rendered element. Thus, the user's intent can correspond to the action conveyed by the appearance of the rendered element. In some implementations, the third-party reviewer can generate an enabled list of approved appearances that the web browser can access.

[0060] The interaction determination module 222 determines whether the interaction 207 of the interaction alert 206 occurred at a declared location of the presentation element 214 within the active window 216 of the client device 210 based on the received declaration 212. Detecting whether the interaction 207 occurred at the declared location of the presentation element 214 may include receiving an XY coordinate of a user click or other form of a selection signal from the operating system 208 or other trusted application 218 specifying a user interaction with the presentation element 214 within the active window 216, detecting the interaction 207 at the declared location of the presentation element includes receiving an X'-X" and Y'-Y" coordinate range of a declared area occupied by the presentation element in the active window 216, comparing the X coordinate of the interaction 207 and the Y coordinate of the interaction 207 to the X'-X" and Y'-Y" ranges, and determining whether the X coordinate of the interaction 207 and the Y coordinate of the interaction 207 are within the X'-X" and Y'-Y" ranges, respectively.

[0061] Based on determining that interaction 207 occurred at the declared location of presentation element 214 within active window 216, interaction determination module 222 provides confirmation of the interaction as an output to visual analysis module 224. In one example, operating system 208 will determine that interaction 207 occurred at the declared location of presentation element 214 by checking whether interaction 207 occurred within the area declared to be occupied by presentation element 214. In another example, if Figure 2B As depicted, for web resource 254 , browser 252 will check whether interaction 257 occurs within the area declared to be occupied by rendering element 264 .

[0062] The visual analysis module 224 is configured to receive as input a screenshot 220 of an active window of the client device 210 including a rendering element 214 from the operating system 208. In addition, the visual analysis module 224 is configured to receive as input a declaration of the rendering element 214 from a repository of declarations 212 for rendering elements. The visual analysis module 224 uses the declared appearance of the rendering element 214 from the declarations 212 of the rendering element 214 to verify the visual appearance of the rendering element 214 in the screenshot 220.

[0063] In some implementations, verifying the visual appearance of the presentation element includes comparing the declared appearance to the screenshot 220 using one or more image matching algorithms. For example, the image matching algorithm can extract image features (e.g., pixel data) of the presentation element, identify a stored set of image features corresponding to the presentation element (e.g., as previously declared by the publisher of the presentation element), and compare the extracted image features to the stored image features to determine whether there is a match between the extracted image features and the stored image features. When verifying the visual appearance of the presentation element, changes due to presentation differences, such as due to scaling, screen tint / brightness / contrast adjustments, fonts used for presentation element text, etc. can be taken into account. For example, when comparing the screenshot 220 to the declaration 212 for the presentation element 214, the image matching algorithm can allow for small changes in the presentation using an algorithm to normalize for size, contrast, brightness, etc., such as using a variation of a two-dimensional normalized cross-correlation algorithm or other suitable algorithm.

[0064] In some implementations, declarations 212 may include "true" rendering values ​​that are not affected by tint masks, aspect tilts, or other similar rendering variations that may occur when an element is rendered on a particular client device or web resource. In this embodiment, the operating system and / or browser may utilize declarations 212 without regard to any minor variations in rendering. In one example, an operating system may have a rendering architecture that includes a device-independent bitmap.

[0065] In some implementations, verification of the visual appearance of presentation element 214 and the declared appearance of the presentation element includes satisfying a similarity threshold, eg, the presentation appearance of the presentation element is within a range of variability with the declared appearance of the presentation element.

[0066] Based on verification that the visual appearance of the presentation element 214 is a match and / or within a similarity threshold, the visual analysis module 224 provides an interactive verification as an output to the proof generator 226 .

[0067] The proof generator 226 is configured to receive as input the verification of the visual appearance and information about the interaction with the presentation element 214, and to generate as output an interaction proof token 221. The interaction proof token 221 may include a stable ID and / or URI declaring the appearance, a timestamp of the interaction 207, a trust token (e.g., a device integrity token for an operating system or a signed redemption record (SRR) for a web browser), a digital signature generated by the client device 210 to verify the integrity of the interaction proof token (e.g., to verify the trusted device that generated the proof token), the XY coordinates of the location where the interaction occurred in the active window, the client device public key, and an unforgeable digital signature created by the operating system or web browser (e.g., to prove the authenticity of the interaction proof token).

[0068] In some implementations, the stable ID and / or URI of the appearance of the presentation element included in the proof token conveys the user's actual intent to interact with the presentation element. For example, if the URI points to a "thumbs-up" icon of a digital component (e.g., video content), the proof token proves that the user did click the "thumbs-up" button to vote up the video content. Based on the actual intent proved by the proof token, the digital content provider 234 can act accordingly, such as increasing the "thumbs-up" count.

[0069] In some implementations, the user intent corresponds to an action conveyed by the appearance of a rendered element. Reporting of user interactions can be modified based on the true intent of the user interaction included in the proof token. In other words, reporting can be modified based on the actual appearance of the rendered element (what the user thought they interacted with). For example, the server can prevent an "upvote" count from being incremented in response to an element rendered in a proof token that conveys a "downvote" user intent. This can prevent the spread of false or misleading information and can improve the user's experience in the future.

[0070] In scenarios where the interaction determination module 222 and / or the visual analysis module determines that the interaction did not occur at the declared location of the presentation element, the system 202 refrains from generating an interaction proof token 221 for the interaction 207 with the presentation element 214 .

[0071] The generated proof token 221 may be provided to the interaction verification server 203 and stored for further use in aggregation of interactions with the presentation element 214 of the digital component 205. The interaction verification server 203 may include an aggregation engine 228 and a response generator 230. The server 203 may receive the proof token 221 as input. The aggregation engine 228 may aggregate interaction proofs for multiple interactions with multiple instances of the presentation element 124, e.g., multiple verified presentations / interactions with the presentation element 214 on multiple client devices 210 by multiple users.

[0072] In some implementations, the system 202 can pair together an interaction proof token 221 for a "finger down" interaction of a user on the touch screen with an interaction proof token 221 for a "finger up" interaction of a user on the touch screen, e.g., sequential interactions, and provide the pair of interaction proof tokens 221 to the digital component provider 234 or another interested third party. The two interaction proof tokens can be paired based in part on a stable ID and / or URI having a declared appearance of the presentation element, a device public key, and a token creation timestamp included in the two proof tokens to prevent user fraud or misinformation.

[0073] Response generator 230 can receive interaction proof token 221 and fake interaction response and provide as output fake presentation element response 232. For example, fake presentation element response 232 can be provided to client device 210 and / or digital component provider 234 of digital component 205.

[0074] In some implementations, in response to determining that the interaction 207 did not occur at the declared location of the presentation element 214, the interaction verification system 202 can trigger a false interaction response. The false interaction response can be provided to the server 203. The aggregation engine 228 can use the proof token for the digital component 205 (e.g., the impression data 114 and the click conversion proof 116) and the interaction proof token 221 for the presentation element to verify the false interaction response. The aggregation engine 228 can aggregate the verified false interaction response to maintain the user privacy of the user who triggered the false interaction response. When a threshold portion of users interacting with the presentation element marks the presentation element as a false presentation element, the presentation element can be reported as a false presentation element. Reports of false presentation elements can be provided to content providers, publishers, and end users. Therefore, the distribution of digital components (including false presentation elements) can be restricted, modified, or stopped. The digital component provider 234 can regard the device 210 or application 218 as untrustworthy if they have an excessive amount of false presentation elements. The digital component provider can block or otherwise limit the provision of content to these devices or applications.

[0075] In some implementations, the processing performed by the interaction determination module 222, the visual analysis module 224, the proof generator 226, and optionally the aggregation engine 228 and the response generator 230 may be performed by more or fewer modules, and each module may include one or more processors.

[0076] In some implementations, the reference can be performed, for example, via a web browser. Figure 2A The methods and systems are described in terms of operating system 208 and application programs 218 . Figure 2B 2 is a block diagram of another environment 250 for verifying false presentation element interactions. Figure 2B As shown, the interactive verification system 202 is configured to receive a presentation notification 204 and an interactive alert 206 from a web browser 252. The interactive verification system 202 is also configured to receive a statement 212 of a verified appearance of a presentation element 214 in an active window 216 of a web resource 254. In addition, the interactive verification system 202 is configured to receive a screenshot 220 showing the active window 216 of the web resource 254 from the web browser 252.

[0077] like Figure 2B As shown, the interaction 257 with the presentation element 264 is the user interacting or selecting (eg, clicking with a mouse) a “close window” presentation element 264 for the digital component 255 (eg, a pop-up window). Figure 2B The presentation element 264 depicted in FIG. 2 may be a fake presentation element, where the action associated with the user interaction (e.g., click) is different from the user's intent. For example, the user's intent for interaction 257 with presentation element 264 is to close digital component 255, such as closing a pop-up window. However, instead of closing digital component 255, the interaction with presentation element 264 results in a redirect to a new login page or is counted as a click on an ad, which is a typical type of ad fraud.

[0078] In some implementations, in response to determining that interaction 257 has a false presentation element, a false interaction response can be triggered. In response to the false interaction response being triggered, the server can intercept the path of the client device to prevent the user from reaching a new login page due to the false presentation element. For example, the server can intercept the path by preventing redirection to the new login page. This reduces the use of resources and processing when presenting the login page to which the false presentation element redirects.

[0079] Figure 3 is a flow chart illustrating an example process 300 for verifying interaction with a presentation element. The process 300 for verifying interaction with a presentation element may be performed by an operating system 208 or another trusted application 218, such as Figure 2A As shown, and / or can be executed by the web browser 252, such as Figure 2B shown.

[0080] A presentation notification 204 is received for a presentation element defined in an active window on a client device (302). The presentation notification 204 is received by the system 202 from an application 218 or a web resource 254. In some implementations, the operating system 208 and / or another trusted application or web browser 252 may receive the presentation notification from the application 218 or the web resource 254, respectively. The presentation notification 204 includes information about a presentation element 214, 264 visible in an active window 216 of the client device or website, such as the location of the element presented in the active window, a stable ID and URI of the presentation element, and a site or domain that invokes the element. The presentation element 214, 264 may be an element that presents the digital component 205 on the client device 210.

[0081] A declaration of a presentation element defined in an active window on a client device is received (304). The declaration 212 may be received from a repository of declarations 212 verified by a third party. The operating system 208 or web browser 252 may request the declaration of the presentation element in response to receiving a presentation notification that the presentation element is visible in an active window on a client device or web page.

[0082] An interaction with a presentation element at the client device is detected (306). The interaction may be detected, for example, by an operating system 208 or a web browser 252 of the client device 210. An interaction 207 with a presentation element 214 may be, for example, a touch of a user's finger on a touch screen or a lift from a touch screen of the client device 210. In another example, an interaction 257 with a presentation element 264 may be a mouse click within an active window 216 of a web resource 254.

[0083] Based on the received declaration, it is determined whether the interaction occurs at the declared location of the presentation element within the active window (308). Figure 2A In the illustrated embodiment, the operating system 208 checks whether the interaction 207 (e.g., a click) occurs within the area declared to be occupied by the presentation element. Detecting an interaction 207 occurring at the declared location of the presentation element 214 may include receiving, from the operating system 208 or other trusted application 218, a selection signal specifying the XY coordinates of a user click or other form of user interaction with the presentation element 214 within the active window 216. In some implementations, detecting an interaction 207 at the declared location of the presentation element includes receiving an X'-X" and Y'-Y" coordinate range for the declared area occupied by the presentation element in the active window 216, comparing the X coordinate of the interaction 207 and the Y coordinate of the interaction 207 to the X'-X" and Y'-Y" ranges, and determining whether the X coordinate of the interaction 207 and the Y coordinate of the interaction 207 are within the X'-X" and Y'-Y" ranges, respectively. Figure 2BIn the illustrated embodiment, the web browser 252 will check whether the interaction 257 occurs within the area declared by the rendering element 264 as occupied.

[0084] In order to determine that the interaction actually occurred at the declared location of the rendered element within the active window, a screenshot of the active window on the client device is captured (310). Before notifying the application 218 or the web resource 254 that the rendered element 214, 264 may be malicious, the operating system 208 or the web browser 252 can capture a screenshot 220 of the active window 216. For example, a screenshot can be captured by generating a trigger to activate a screenshot function (e.g., of a camera application installed on the device). The generated signal can be passed to the camera application, causing the camera application to capture a screenshot of the active window on the client device. In some implementations, the device operating system can directly perform the screen capture operation by copying the display frame buffer. The captured screenshot can then be evaluated by one or more other applications installed on the client device and / or the server-implemented function.

[0085] The visual appearance of the rendered element in the screenshot is verified with the declared appearance of the rendered element (312). The operating system 208 or web browser 252 compares the screenshot 220 with the declared appearance of the rendered element 214, 264 from the declaration 212. For example, the comparison may be a pixel-by-pixel comparison or another comparison (e.g., a block or feature-based comparison). If the appearance of the rendered element 214, 264 in the screenshot 220 matches the declared appearance of the declaration 212 of the rendered element 214, 264, then the operating system 208 or web browser 252 generates an interaction proof 221 for the interaction with the rendered element (314). The interaction proof token 221 may include information about the interaction 207, 257, as described above with reference to FIG. Figure 2A Described in more detail.

[0086] In some implementations, an application 218 or web resource 254 that presents a presentation element 214, 264, respectively, may pair a "finger down" interaction proof token 221 with a "finger up" interaction for the same presentation element 214, 264 and provide the paired interaction proof tokens 221 to a digital component provider 234 or other interested party for measurement, auditing, or other reporting-based actions.

[0087] Upon determining that no interaction occurred at the declared location of the presentation element within the active window, the system refrains from generating an interaction proof for interaction with the presentation element (316). In some implementations, a false presentation element response is generated by the operating system 208 or the web browser 252 and may be provided to the server 203, notifying the server 203 that the presentation element 214, 264 or the application 218 or web resource 254 that claims that the user interacted with the presentation element may be malicious.

[0088] Figure 4 4 is a block diagram of an example computer system 400 that can be used to perform the above operations. System 400 includes a processor 410, a memory 420, a storage device 430, and an input / output device 440. Each of components 410, 420, 430, and 440 can be interconnected, for example, using a system bus 450. Processor 410 is capable of processing instructions for execution within system 400. In some implementations, processor 410 is a single-threaded processor. In another implementation, processor 410 is a multi-threaded processor. Processor 410 is capable of processing instructions stored on memory 420 or storage device 430.

[0089] Memory 420 stores information within system 400. In one implementation, memory 420 is a computer-readable medium. In some implementations, memory 420 is a volatile memory unit. In another implementation, memory 420 is a non-volatile memory unit.

[0090] The storage device 430 can provide mass storage for the system 400. In some implementations, the storage device 430 is a computer-readable medium. In various implementations, the storage device 430 may include, for example, a hard disk device, an optical disk device, a storage device shared by multiple computing devices (e.g., a cloud storage device) over a network, or some other mass storage device.

[0091] Input / output device 440 provides input / output operation for system 400.In some implementations, input / output device 440 can include one or more network interface devices, such as Ethernet card, serial communication device, such as RS-232 port and / or wireless interface device, such as 802.11 card.In another implementation, input / output device can include the driver device that is configured to receive input data and output data is sent to external device 460 (such as, keyboard, printer and display device).But, also can use other implementations, such as mobile computing device, mobile communication device, set-top box TV client device etc.

[0092] Despite Figure 4An example processing system is described in the specification, but the subject matter and implementation of the functional operations described in this specification may be implemented in other types of digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them.

[0093] Embodiments of the subject matter and operations described in this specification may be implemented in digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more computer program instruction modules, encoded on a computer storage medium (or medium) for execution or control of operations by a data processing device. Alternatively, or in addition, program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to an appropriate receiver device for execution by a data processing device.

[0094] A computer storage medium can be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more thereof. Furthermore, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium can also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0095] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0096] The term "data processing apparatus" includes various apparatuses, devices and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or a plurality or combination of the foregoing. The apparatus may include dedicated logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0097] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program may be deployed for execution on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected by a communications network.

[0098] The processes and logic flows described in this specification can be performed by one or more programmable processors, which execute one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by special-purpose logic circuits (such as FPGA (field programmable gate array) or ASIC (application-specific integrated circuit)), and the apparatus can also be implemented as special-purpose logic circuits.

[0099] For example, processors suitable for executing computer programs include both general-purpose and special-purpose microprocessors. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor that performs operations according to instructions and one or more memories that store instructions and data. Typically, a computer will also include or be operably coupled to receive data from or send data to one or more mass storage devices or both, and the mass storage device is used to store data, such as a magnetic disk, a magneto-optical disk, or an optical disk. However, it is not necessary for a computer to have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), etc. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM optical disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0100] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, as well as a keyboard and a pointing device (e.g., a mouse or trackball), through which the user can provide input to the computer. Other types of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on a user's client device in response to a request received from the web browser.

[0101] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), intranets (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0102] A computing system may include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is due to the computer programs running on the respective computers and the client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to the client device (e.g., in order to display data to a user interacting with the client device and receive user input from the user interacting with the client device). Data generated at the client device (e.g., the result of the user interaction) can be received from the client device at the server.

[0103] A user may be provided with controls that allow the user to make a choice to determine whether and when the systems, programs, or features described herein can collect user information (e.g., information about the user's social network, social behavior or activities, occupation, the user's preferences, or the user's current location), and whether the user receives personalized content or communications from the server. In addition, before certain data is stored or used, it may be processed in one or more ways to remove personally identifiable information. For example, the user's identity may be processed so that personally identifiable information cannot be determined for the user, or the user's geographic location may be summarized where location information is obtained (such as to a city, zip code, or state level) so that the user's specific location cannot be determined. Thus, the user can control what information is collected about the user, how the information is used, the information retention policy, and what information is provided to the user.

[0104] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or what may be claimed, but rather as descriptions of specific features of specific embodiments of specific inventions. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although the above-mentioned features may be described as working in certain combinations and even initially required to do so, in some cases one or more features from the required combination may be cut out from the required combination, and the required combination may be directed to a sub-combination or a variation of the sub-combination.

[0105] Similarly, although operations are described in a particular order in the accompanying drawings, this should not be construed as requiring that such operations be performed in the particular order or sequence shown, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0106] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes described in the accompanying drawings do not necessarily require the particular order or sequence shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by a data processing device, the method comprising: receiving a presentation notification for a presentation element defined in an active window on a client device, wherein the presentation notification includes a position of the presentation element in the active window; receiving a declaration of the presentation element defined in the active window on the client device, wherein the declaration includes a position of the presentation element in the active window; detecting, at the client device, an interaction with the presentation element; Based on the received declaration, determining whether the interaction occurs at a declared location of the presentation element within the active window; and Based on determining whether the interaction occurs at the declared location, processing the interaction includes: In response to determining that the interaction occurs at the declared location of the presentation element: capturing a screenshot of the active window on the client device; verifying the visual appearance of the presentation element in the screenshot with the declared appearance of the presentation element; and generating an interaction proof for the interaction with the presentation element, thereby verifying the interaction; and Responsive to determining that the interaction did not occur at the declared location of the presentation element, avoiding generating the interaction proof for the interaction with the presentation element.

2. The method according to claim 1, wherein: The presence notification also includes one or more of i) a stable ID and URI of the presence element, and ii) a site or domain where the presence element is presented.

3. The method according to claim 1 or 2, wherein: Receiving the declaration of the presentation element includes receiving a validated appearance of the presentation element.

4. The method according to claim 3, wherein: The verified appearance includes a unique stable ID and / or a stable URL and a verified shape.

5. The method according to claim 1 or 2, wherein: Verifying the visual appearance of the presentation element in the screenshot with the declared appearance of the presentation element includes utilizing an image matching algorithm.

6. The method according to claim 1 or 2, wherein: Detecting the interaction includes receiving a selection signal specifying XY coordinates of a user click within the active window.

7. The method of claim 1 or 2, further comprising aggregating the interaction proofs for a plurality of interactions with a plurality of instances of the presentation element.

8. The method according to claim 1, wherein: The interaction proof includes one or more of i) a stable ID and / or URI of the asserted appearance, ii) a timestamp of the interaction, iii) a device integrity token, iv) a digital signature of the client device, v) XY coordinates of the interaction, and vi) a client device public key.

9. The method according to claim 8, wherein: The stable ID and / or URI of the declarative facade includes the user's intent for the interaction.

10. The method according to claim 9, wherein: The user intent corresponds to an action conveyed by the appearance of the presentation element.

11. The method according to claim 9 or 10, further comprising: In response to the evidence of the interaction including user intent, modifying a report of the interaction based on the user intent.

12. The method according to claim 9, further comprising: In response to determining that the interaction is different from the user intent, determining that the presentation element is a false presentation element and triggering a false interaction response.

13. The method according to claim 12, further comprising: In response to triggering the false interaction response, intercepting a path of the client device to a login page of the false presentation element.

14. The method according to claim 1, further comprising: In response to determining that the interaction did not occur at the declared location of the presentation element, a false interaction response is triggered.

15. A computer storage medium encoded with a computer program, the program comprising instructions which, when executed by a data processing apparatus, cause the data processing apparatus to perform operations comprising: receiving a presentation notification for a presentation element defined in an active window on a client device, wherein the presentation notification includes a position of the presentation element in the active window; receiving a declaration of the presentation element defined in the active window on the client device, wherein the declaration includes a position of the presentation element in the active window; detecting, at the client device, an interaction with the presentation element; Based on the received declaration, determining whether the interaction occurs at a declared location of the presentation element within the active window; and Based on determining whether the interaction occurs at the declared location, processing the interaction includes: In response to determining that the interaction occurs at the declared location of the presentation element: capturing a screenshot of the active window on the client device; verifying the visual appearance of the presentation element in the screenshot with the declared appearance of the presentation element; and generating an interaction proof for the interaction with the presentation element, thereby verifying the interaction; and Responsive to determining that the interaction did not occur at the declared location of the presentation element, avoiding generating the interaction proof for the interaction with the presentation element.

16. A system for data processing, comprising: Client devices; as well as One or more computers operable to interact with the client device and perform operations including: receiving a presentation notification for a presentation element defined in an active window on a client device, wherein the presentation notification includes a position of the presentation element in the active window; receiving a declaration of the presentation element defined in the active window on the client device, wherein the declaration includes a position of the presentation element in the active window; detecting, at the client device, an interaction with the presentation element; Based on the received declaration, determining whether the interaction occurs at a declared location of the presentation element within the active window; and Based on determining whether the interaction occurs at the declared location, processing the interaction includes: In response to determining that the interaction occurs at the declared location of the presentation element: capturing a screenshot of the active window on the client device; verifying the visual appearance of the presentation element in the screenshot with the declared appearance of the presentation element; and generating an interaction proof for the interaction with the presentation element, thereby verifying the interaction; and Responsive to determining that the interaction did not occur at the declared location of the presentation element, avoiding generating the interaction proof for the interaction with the presentation element.

Images