SYSTEMS AND METHODS FOR DETECTING INAPPROPRIATE IMPLEMENTATION OF CONTENT ELEMENTS BY APPLICATIONS RUNNING ON CLIENT DEVICES

Abstract

Method (400) for detecting an inappropriate implementation of a presentation of content elements by applications running on client devices, comprising: from a first server, received (420), by an application (205) running on a client device (125) containing one or more data processors, a content package containing a first content element (245), a second content element (250) and an interaction acknowledgment script (215), wherein the interaction acknowledgment script (215) is configured to run within the application (205) running on the client device (125) and to cause the application (205) to: in response to the first content element (245) presented within the application (205) to the client device (125), to detect (430) a first interaction event with the first content element (245); upon detection of the first interaction event at the first content element (245), to present the second content element (250) within the application (205) (435); to detect a second interaction event with the second content element (250) (440); to transmit an acknowledgment of the first interaction event to the first server upon detection of the second interaction event (445); and to transmit a query (450) to a second server to access an information resource (210) linked to the second content element (250); and through the client device (125) to present the information resource (210) in response to the second interaction event at the second content element (250) (460), wherein the first server is configured to maintain a content restriction list (150) in a database (145), wherein the content restriction list (150) contains a plurality of application identifiers corresponding to a plurality of applications, wherein the content restriction list (150) is used to restrict a selection of the first content element (245) for transmission in response to content queries from the plurality of applications to which the application identifiers on the list correspond, and the first server is further set up for: in response to a failure to receive confirmation from the application (205), to add an application identifier of the application (205) to the content restriction list (150); and as a response to receiving a subsequent content query, to determine the selection of the first content element (245) for transmission to the application which corresponds to the application identifier on the content restriction list (150).

Description

BACKGROUND

[0001] In a computer-networked environment such as the internet, data can be delivered from a server to a client device in response to a content request. The client device receives the data from the server. This data can contain instructions for rendering content according to the data for display on the client device. Content publishers can prescribe these instructions according to the presentation parameters of a content element (e.g., a native content element).

[0002] DE 20 2014 010 890 U1 describes a system comprising one or more processors and a memory comprising instructions which, when executed by the one or more processors, cause the one or more processors to: receive an advertisement and a content element, wherein the advertisement is linked to the content element; display the advertisement on a web page; identify a selection of the advertisement; display, in response to the identification, the content element on the web page, wherein the content element is displayed neither next to nor overlapping with the advertisement; and send, in response to the selection, an indication of a click-through of the advertisement. SUMMARY

[0003] At least one aspect is directed toward a method for detecting an inappropriate implementation of content element presentation by applications running on client devices. The method can detect unintended interactions with native content elements in client-running applications that occur due to the inappropriate presentation of the content elements by the applications. The method includes receiving, by an application running on a client device containing one or more data processors, from a first server, a content package comprising a first content element, a second content element, and an interaction acknowledgment script, wherein the interaction acknowledgment script is configured to run within the application running on the client device and cause the application to: in response to the first content element,The procedure involves the process of detecting a first interaction event with the first content element within the application on the client device; upon detection of the first interaction event on the first content element, presenting the second content element within the application; detecting a second interaction event with the second content element; upon detection of the second interaction event, transmitting an acknowledgment of the first interaction event to the first server; and transmitting a query to a second server to access an information resource associated with the second content element. The procedure further includes the client device presenting the information resource in response to the second interaction event on the second content element, the first server being configured to maintain a content constraint list in a database.wherein the content restriction list contains a plurality of application identifiers corresponding to a plurality of applications, wherein the content restriction list is used to restrict the selection of the first content element for transmission in response to content queries from the plurality of applications to which the application identifiers on the list correspond, and wherein the first server is further configured: in response to a failure to receive an acknowledgment from the application, to add an application identifier of the application to the content restriction list; and in response to receiving a subsequent content query, to determine the selection of the first content element for transmission to the application to which the application identifier on the content restriction list corresponds.

[0004] In some implementations, the first content element contains a native content element configured to fine-tune display parameters of other application content elements. In some implementations, the second content element contains an interstitial content element configured to occupy a main portion of the application's display area when it is shown. In some implementations, the second content element contains a script with instructions for the application to detect the second interaction event and transmit the acknowledgment of the first interaction event. In some implementations, the first content element contains the same text as text contained in the second content element.

[0005] In some implementations, the interaction acknowledgment script also contains computer-executable instructions which, when executed by the application, cause the application to transmit a statement of the first interaction event to the first server upon detection of the first interaction event. In some implementations, the acknowledgment of the first interaction event includes an application identifier to identify the application.

[0006] At least one aspect is directed toward a system for detecting inappropriate implementation of content element presentation by applications running on client devices. The system can detect unintended interactions with native content elements in client-side applications that occur due to the applications' inappropriate presentation of the content elements. The system includes a processor and memory coupled to the processor. The memory stores computer-executable instructions which, when executed by the processor, cause the processor, through an application running on a client device, to receive a content packet from a first server. This packet contains a first content element, a second content element, and an interaction acknowledgment script. The interaction acknowledgment script is configured to...to run within the application running on the client device and to cause the application, in response to the first content element presented within the application to the client device, to detect a first interaction event with the first content element; upon detection of the first interaction event with the first content element, to present the second content element within the application; to detect a second interaction event with the second content element; upon detection of the second interaction event, to transmit an acknowledgment of the first interaction event to the first server; and to transmit a query to a second server to access an information resource associated with the second content element. The computer-executable instructions further cause the processor tothrough the client device to present the information resource in response to the second interaction event at the second content element, wherein the first server is configured to maintain a content restriction list in a database, wherein the content restriction list contains a plurality of application identifiers corresponding to a plurality of applications, wherein the content restriction list is used to restrict a selection of the first content element for transmission in response to content queries from the plurality of applications to which the application identifiers on the list correspond, and wherein the first server is further configured to: add an application identifier of the application to the content restriction list in response to a failure to receive an acknowledgment from the application; and determine, in response to receiving a subsequent content query,to restrict the selection of the first content element for transfer to the application that corresponds to the application identifier on the content restriction list.

[0007] In some implementations, the first content element contains a native content element configured to tune display parameters of other application content elements. In some implementations, the second content element contains an interstitial content element configured to occupy a main portion of the application's display area when it is shown. In some implementations, the second content element contains a script containing instructions for the application to detect the second interaction event and transmit the acknowledgment of the first interaction event. In some implementations, the first content element contains the same text as text contained in the second content element.

[0008] In some implementations, the interaction acknowledgment script also contains computer-executable instructions which, when executed by the application, cause the application to transmit a statement of the first interaction event to the first server upon detection of the first interaction event. In some implementations, the acknowledgment of the first interaction event includes an application identifier to identify the application.

[0009] At least one aspect is directed toward a method for detecting an inappropriate implementation of the presentation of content elements in applications running on client devices. The method can detect unintended interactions with native content elements in the applications running on the client devices, which occur due to the applications' inappropriate presentation of the content elements. The method involves receiving, by a data processing system containing one or more processors, content queries from a multitude of client devices, each content query being generated by an application running on a specific client device among the multitude of client devices.The method further comprises transmitting, by the data processing system to each client device of a first subset of the plurality of client devices in response to a content query from the client device, a content package containing a first content element, a second content element, and an interaction acknowledgment script, wherein the interaction acknowledgment script is configured to cause the application to: configure the first content element to be presented in a first presentation format, present the first content element in the first presentation format, display the second content element in response to a detection of a first interaction event at the first content element, and transmit an acknowledgment to the data processing system in response to a detection of a second interaction event at the second content element.Furthermore, in response to a content request from the client device, the procedure includes the data processing system transmitting the first content element to each client device of a second subset of the multitude of client devices. The procedure further includes the data processing system determining a first performance metric of the first content package based on acknowledgments received from the first subset of the multitude of client devices. The procedure further includes the data processing system determining a second performance metric of the first content element based on actions received from the second subset of client devices. The procedure further includes the data processing system comparing the first performance metric with the second performance metric.Furthermore, the procedure based on comparison includes an addition by the data processing system of an application identifier of the application to a content restriction list in order to limit the selection of the first content element for a transfer in response to a subsequent content query that identifies the application.

[0010] In some implementations, the first content element contains a native content element that is set up to tune display parameters of other content elements in the application.

[0011] These and other aspects and implementations are discussed in more detail below. The preceding information and the following detailed description include illustrative examples of various aspects and implementations and provide an overview or framework for understanding the nature and character of the claimed aspects and implementations. The drawings provide an illustration and further understanding of the various aspects and implementations and are incorporated into and form part of this description. It is understood that aspects and implementations can be combined and that the features described in connection with one aspect or implementation can be implemented in connection with other aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are not to scale. The same reference symbols and labels are used in the different drawings to denote the same elements. For clarity, not every component may be labeled in every drawing. The drawings include: Fig. Figure 1 is a block diagram that represents an environment for detecting inappropriate presentation of content elements by applications running on client devices, according to some implementations. Fig. Figure 2 is a block diagram that represents a system for detecting inappropriate presentation of content elements by applications running on client devices, according to some implementations. Fig. Figure 3 is a signaling diagram that illustrates communication within an environment to detect inappropriate presentation of content elements by applications running on client devices, according to some implementations. Fig. Figure 4 is a flowchart that represents a procedure for detecting unintended interactions with native content elements according to some implementations. Fig. Figure 5 is a flowchart describing a procedure for detecting an inappropriate presentation of content elements by a data processing system in communication with the applications running on the client devices, with respect to Fig. 4, according to some implementations. Fig. 6 is a block diagram that represents a general architecture for a computer system that can be used to implement various elements of the systems and procedures described and illustrated herein. DETAILED DESCRIPTION

[0013] The following are detailed descriptions of various concepts relating to methods, devices, and systems for detecting inappropriate presentation of content elements by applications running on client devices, and their implementations. The various concepts introduced above and discussed in more detail below can be implemented in numerous ways, as the described concepts are not limited to any specific implementation method.

[0014] This disclosure relates to systems and methods for detecting inappropriate presentation of content elements by applications running on client devices. In particular, this disclosure relates to systems and methods for detecting incorrect interactions with content elements in an application running on a mobile computing device.

[0015] Within a mobile application, dynamically loaded content elements such as images, videos, or other displayable content can be embedded within the application's appearance in either a native or an interstitial format. In the native format, content element publishers are free to individually configure characteristics of the content elements, such as fonts, colors, the location of content element components, or other details, independent of the mobile application in which the content elements are ultimately displayed. In the interstitial format, content elements are typically displayed at natural transition points or logical breaks in the application's flow and can overlay the regular user interface (e.g., an interstitial content element can be displayed in full screen).

[0016] For content elements in the native format, output typically involves providing individual components of that content element (e.g., title, image, description, etc.) that a content publisher can implement or display as they see fit. However, the publisher will often display the native content element inappropriately, resulting in a poor user experience due to unintended user interactions. For example, consider a "swipe to unlock" functionality that allows a mobile device to be unlocked by swiping across a locked screen. The locked screen may also display a content element that is likewise activated by swiping. This content element may be incorrectly displayed over or very close to the "swipe to unlock" area.In fact, if a user swipes to unlock the mobile device, the swipe may not only be recognized as a swipe to unlock, but may also (and incorrectly) be recognized as an interaction with the content element, resulting in the mobile device performing unwanted operations, such as opening a link in a browser that is associated with the unintentionally swiped content element.

[0017] There are technical problems related to incorrectly presented content elements within a client device application (e.g., content elements that are rendered inappropriately within an application). For example, causing a client device to execute the computer-executable instructions to handle an unintentional click on an incorrectly displayed or presented content element can result in increased battery consumption, wasted network resources delivering data associated with the unintentionally clicked content element, and additional computer resources that can adversely affect the client device's performance. Furthermore, unintended interactions with content elements can lead to a poor user experience.

[0018] Fig. Figure 1 is a block diagram representing an environment for detecting inappropriate presentation of content elements by applications running on a client device, according to some implementations. The environment 100 includes at least one data processing system 110. The data processing system 110 may include at least one processor (or processing circuit) and memory. The memory stores processor-executable instructions which, when executed on a processor, cause the processor to perform one or more operations described herein. The processor may include a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc., or combinations thereof.The memory may, but is not limited to, include an electronic, optical, magnetic, or other storage or transmission device capable of supplying program instructions to the processor. The memory may further include a floppy disk, CD-ROM, DVD, magnetic disk, memory chip, ASIC, FPGA, read-only memory (ROM), random-access memory (RAM), electrically erasable ROM (EEPROM), erasable-programmable ROM (EPROM), flash memory, optical media, or any other suitable memory from which the processor can read instructions. The instructions may contain code in any suitable computer programming language. The 110 data processing system may comprise one or more computer units or one or more servers capable of performing various functions.

[0019] Network 105 can include computer networks such as the Internet, local area networks, wide area networks, metro or other area networks, intranets, satellite networks, other computer networks such as voice or data-related mobile communication networks, and combinations thereof. The data processing system 110 of the environment 100 can communicate via Network 105 with, for example, at least one content provider computer device 115, at least one content publisher computer device 120, or at least one client device 125. Network 105 can be any form of computer network that transmits information between the client device 125, the data processing system 110, and one or more content sources, such as web servers or content servers, to name just a few. Network 105 can, for example, include the Internet and / or other types of data networks, such as...Network 105 can include a local area network (LAN), a wide area network (WAN), a cellular network, a satellite network, or other types of data networks. Network 105 can also include any number of computing devices (e.g., computers, servers, routers, network switches, etc.) configured to receive and / or transmit data within Network 105. Furthermore, Network 105 can include any number of wired and / or wireless connections. For example, Client Device 125 can communicate wirelessly (e.g., via Wi-Fi, cellular network, radio, etc.) with a transceiver that is hardwired (e.g., via a fiber optic cable, a CAT5 cable, etc.) to other computing devices within Network 105.

[0020] The content provider computer device 115 may contain servers or other computer devices operated by a content provider entity to provide one or more content elements for display on information resources at the client device 125. The content provided by the content provider computer device 115 may include third-party content elements for display on information resources, such as a website or web page that includes primary content, for example, content provided by the content publisher computer device 120. The content elements may also be displayed on a search results web page. The content provider computer device 115 may, for example, provide or be the source of one or more content elements for display in content slots of content web pages.A company website where the primary website content is provided by the company, or for an advertisement on a search engine landing page. The content elements associated with the content provider computer device 115 can be displayed on information resources other than websites, such as content displayed as part of the execution of an application (such as a global positioning system (GPS), a map application, or other application types) on a smartphone or other client device 125.

[0021] The content publisher computer device 120 can include servers or other computer devices operated by a content-publishing entity to provide primary content for display over the network 105. For example, the content publisher computer device 120 can include a website operator that provides primary content for display on the website. The primary content can be different from that provided by the content publisher computer device 120, and the website can contain content slots configured by the content provider computer device 115 to display third-party content elements. For example, the content publisher computer device 120 can operate a company's website and provide content about that company for display on web pages of the website.The web pages may include content slots configured to display third-party content elements from the Content Provider computer device 115. In some implementations, the Content Publisher computer device 120 includes a search engine computer device (e.g., a server) belonging to a search engine operator that runs a search engine website. The primary content of search engine web pages (e.g., a results or landing page) may include both search results and third-party content elements displayed in content slots, such as content elements from the Content Provider computer device 115. In some implementations, the Content Publisher computer device 120 may include a server for outputting video-related content.

[0022] The client device 125 can include computer devices configured to communicate over the network 105 to display data, such as content provided by the content publisher computer device 120 (e.g., the primary website content or other information resources) and content provided by the content provider computer device 115 (e.g., third-party content elements configured for display in a content slot of a website or application). For example, the client device 125 can be configured to run an application created or published by the content publisher computer device 120, and the application can be configured to display content elements (e.g., native content elements) within it.The client device 125, the content provider computer device 115, and the content publisher computer device 120 can include desktop computers, laptop computers, tablet computers, smartphones, personal digital assistants, mobile devices, consumer computing devices, servers, clients, digital video recording devices, a set-top box for a television, a video game console, or other computing devices configured to communicate over the network 105. The client device 125 can be a communication device through which an end user can send queries to receive content. These queries can be queries to a search engine and can include search queries. In some implementations, the queries can include a query to access a web page.

[0023] The content provider computer device 115, the content publisher computer device 120, and the client device 125 can include a processor and memory, i.e., processing circuitry. The memory stores machine instructions which, when executed on the processor, cause the processor to perform one or more operations described herein. The processor can be a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc., or combinations thereof. The memory can be, but is not limited to, an electronic, optical, magnetic, or any other storage or transmission device capable of supplying program instructions to the processor.The memory may further comprise a floppy disk, CD-ROM, DVD, magnetic disk, memory chip, ASIC, FPGA, read-only memory (ROM), random access memory (RAM), electrically erasable ROM (EEPROM), erasable-programmable ROM (EPROM), flash memory, optical media, or any other suitable memory from which the processor can read instructions. The instructions may contain code in any suitable computer programming language.

[0024] The content provider computer device 115, the content publisher computer device 120, and the client device 125 may also include one or more user interface devices. A user interface device generally refers to any electronic device that transmits data to a user by generating sensory information (such as visualization on a display, one or more sounds, etc.) and / or converts sensory information received from a user (e.g., through a keyboard, mouse, pointing device, touchscreen display, microphone, etc.) into electronic signals. The one or more user interface devices may be located inside an enclosure of the content provider computer device 115, the content publisher computer device 120, and the client device 125 (e.g., a built-in display, microphone, etc.).) or outside the enclosure of the content provider computer device 115, the content publisher computer device 120, and the client device 125 (e.g., a monitor connected to the user computer device 115, a speaker connected to the user computer device 115, etc.). The content provider computer device 115, the content publisher computer device 120, and the client device 125 may, for example, include an electronic display that visually presents web pages using web page data received over the network 105 from one or more content sources and / or the data processing system 110. In some implementations, a content placement campaign manager or an external content provider may communicate with the data processing system 110 via the content provider computer device 115.In some implementations, the content placement manager or external content provider can communicate with the data processing system 110 via a user interface displayed on the user interface devices of the content provider computer device 115.

[0025] The data processing system 110 can contain at least one server. The data processing system 110 can, for example, comprise a large number of servers located in at least one data center or server farm. In some implementations, the data processing system 110 can include an external content placement system, such as a content server. The data processing system 110 can include at least one content query module 130, at least one content selection module 135, at least one script provider module 140, and at least one database 145.The content query module 130, the content selection module 135, and the script provider module 140 can each contain at least one processing unit, server, virtual server, circuit, machine, agent, device, or other logic device, such as the programmable logic arrays, configured to communicate with the database 145 and with other computer devices (e.g., the content provider computer device 115, the content publisher computer device 120, or the client device 125) over the network 105.

[0026] The content query module 130, the content selection module 135, and the script provider module 140 can contain or execute at least one computer program or at least one script. The content query module 130, the content selection module 135, and the script provider module 140 can be separate components, a single component, or part of the data processing system 110. The content query module 130, the content selection module 135, and the script provider module 140 can include combinations of software and hardware, such as one or more processors, configured to execute one or more scripts.

[0027] The data processing system 110 can also include one or more content repositories or databases 145. The databases 145 can be located decentrally to the data processing system 110. In some implementations, the databases 145 can be located remotely from the data processing system 110 but communicate with it via the network 105. The databases 145 can contain, among other things, web pages, parts of web pages, third-party content elements, and a script for asynchronous click transmission to serve a client device 125. In some implementations, the database 145 is configured to store a content restriction list 150. The content restriction list 150 can identify applications running on the client device 125 that inappropriately present native content and are therefore prevented from receiving native content (such as...).(identified by Content Restriction List 150) should be excluded. Furthermore, a disclosure regarding Content Restriction List 150 is presented below.

[0028] The content query module 130 can receive a query about content from the client device 125. The content query can include a query about an information resource, a query about one or more third-party content elements, a query about a script, or a combination thereof. In some implementations, the content query can include a query about third-party content. In some implementations, the query about third-party content can include an address or identifier of an information resource where the third-party content is to be displayed. The query about third-party content can also include or identify one or more parameters that can be used by the data processing system 110 to determine the content to be provided in response to the content query.The parameters can, for example, identify the size of a content slot into which the requested content should be inserted. The parameters can identify a content type associated with the information resource, a type of requested third-party content (e.g., text, image, video, etc.), client device information, size information for the requested third-party content element, or a combination thereof.

[0029] The content selection module 135 can be configured to determine the content to be transmitted to the client device 125 in response to a received content query. The content selection module 135 can determine the content to be sent to the client device 125 based on the information contained in the content query. For example, upon receiving a query for an information resource, the content selection module 135 can use the address or identifier of the information resource in the content query to determine the content to be sent to the client device. In the case of receiving a query for one or more third-party content elements, the content selection module 135 can select the third-party content element(s) based on an address or identifier for the information resource on which the third-party content element is to be presented, content type information (e.g.,The content selection module (e.g., sports, news, music, movies, travel, etc.) is selected for the information resource, along with size information for the slot(s) in which the third-party content element(s) should be displayed, and client device information (e.g., device type, device identifier, device location, etc.). In some implementations, the content selection module 135 can access the database 145 and retrieve the content for sending to the client device 125.

[0030]

[0029] In some implementations, the content selection module 135 can be configured to determine whether or not to send a script to the client device 125 in response to the content query. The content selection module 135 can determine whether to send a script to the client device 125 based on the type of content query (e.g., a content query for native content), an identifier of an information resource in the content query, or a combination thereof. In some implementations, the content selection module 135 can be configured to determine whether or not to send a script to the client device 125 by classifying the received content query for use with an interaction acknowledgment protocol.Upon a request to send a script to client device 125, the content selector module 135 can forward the content query, or a specification thereof, to the script provider module 140. In some implementations, the content selector module 135 can select a third-party content element to send to client device 125 along with a script, such as the interaction confirmation script.

[0031] The script provider module 140 can be configured to retrieve an interaction confirmation script from the database 145 and transmit the retrieved script to the client device 125 (or to another entity for sending to the client device 125). In some implementations, the interaction confirmation script can be transmitted to the client device 125 separately or together with a third-party content element. For example, the script provider module 140, the content picker module 135, or another entity of the data processing system 110 can insert the interaction confirmation script into the third-party content element before sending the third-party content element to the client device 125. In some implementations, the script provider module 140, the content picker module 135, or another entity of the data processing system 110 can send the information confirmation script together with an information resource to the client device 125.

[0032] The interaction confirmation script can contain computer-executable applications. These computer-executable instructions can include a script such as, but are not limited to, Hypertext Markup Language (HTML), Extensible Hypertext Markup Language (XHTML), Extensible Markup Language (XML), Cascading Style Sheets (CSS), and JavaScript. The computer-executable instructions can be executed within an application on the client device 125, such as the application that causes the client device 125 to transmit the content query received from the content query module 130. The application can include, for example, a web browser, a mobile application, a game application, a GPS application, or any other computer program capable of reading and executing the computer-executable instructions.

[0033] In brief, the computer-executable instructions, when executed by a processor of the client device 125, can cause an application of the client device to: (a) detect a first interaction event with the first content element in response to a first content element presented within the application on the client device, (b) present the second content element within the application upon detection of the first interaction event with the first content element, (c) detect a second interaction event with the second content element, (d) transmit an acknowledgment of the first interaction event to the first server upon detection of the second interaction event, and (e) transmit a query to a second server to access an information resource associated with the second content element.Additional details relating to the interaction confirmation script are further detailed herein in relation to . Fig. 2, Fig. 3, Fig. 4 and Fig. 5 delivered.

[0034] Fig. Figure 2 is a block diagram representing a system for detecting inappropriate presentation of content elements by applications running on client devices, according to some implementations. The interaction acknowledgment system 200 may include an application 205, an information resource 210, an interaction event detection module 220, a content element presentation module 225, an acknowledgment generation module 230, and a query generation module 235. In some implementations, the interaction event detection module 220, the content element presentation module 225, the acknowledgment generation module 230, and the query generation module 235 may be modules within an interaction acknowledgment script 215. The interaction acknowledgment script system 200 may run on one or more processors of a computer device or be executed by such processors in other ways, such as those found in Fig. 6 below. In some implementations, any one of the interaction event detection module 220, content element presentation module 225, acknowledgment generation module 230, and query generation module 235 can be part of the application 205, the information resource 210, or the interaction acknowledgment script 215.

[0035] In some implementations, the interaction acknowledgment script 215 is received by client device 125 via application 205, which runs on client device 125. The interaction acknowledgment script 215 can be sent by data processing system 110 to client device 125 in response to a content query sent by client device 125 to data processing system 110. In some implementations, the content query sent by client device 125 includes a query for native content. In some implementations, data processing system 110 transmits a content package 240 containing the interaction acknowledgment script 215. The content package 240 may further contain a first content element 245 and a second content element 250 for rendering and presentation by application 205.In some implementations, the first content element 245 contains a native content element, and the second content element 250 contains an interstitial content element.

[0036] In some implementations, in response to the content query received from client device 125, the data processing system 110 can classify the received query as one that will trigger packaging and transmission of the interaction acknowledgment script 215 to client device 125, along with a selected native content element (e.g., the first content element 245) and a similar interstitial content element (e.g., the second content element 250). Further disclosure regarding content package 240, first content element 245, and second content element 250 is presented below.

[0037] The application 205 can include, for example, an internet browser, a mobile application, a game application, a GPS application, or any other computer program that can execute or otherwise invoke computer-executable instructions, such as the computer-executable instructions contained in the information resource 210, the interaction acknowledgment script 215, the interaction detection module 220, the content element presentation module 225, the acknowledgment generation module 230, and the query generation module 235. The application 205 can obtain or receive the information resource 210 from a server hosting the information resource, such as a server of the content publisher 120, a server of the content provider 115, or a server of the data processing system 110.

[0038] Fig. Figure 3 is a signaling diagram that illustrates communication within an environment for detecting inappropriate presentation of content elements by applications running on client devices, according to some implementations. Taking together... Fig. 1 - Fig. 3. With reference, the client device 125 (e.g., via application 205) can now transmit a content query to signal 302 and then to the data processing system 110. In some implementations, the content query is a query for a native content element. In some implementations, a native content element may appear to have the same appearance as content generated by the content publisher 120 within the environment of application 205. For example, a native content element may have visual characteristics that match, or are substantially similar to, the surrounding environment of application 205 in which the native content element is to be displayed.In some implementations, the data processing system 110 sends a native content element to the client device 125 in the form of a variety of content components that constitute the native content element, and the content components are rendered on the client device 125 by the application 205. For example, the content components may include a heading, an image, a body, a call to action, a logo, an attribution, a clickable URL, or the like.

[0039] In some implementations, the content selection module 135, in response to the content query module 130, which receives a content query from the client device 125, can classify the received query at block 303 as one that will trigger packaging and transmission of the interaction acknowledgment script 215 to the client device 125 along with a selected native content element and a similar interstitial content element. In some implementations, the content query includes an application identifier to identify the application 205. Accordingly, the data processing system 110 (e.g., in the database 145) can maintain a list of applications that are candidate applications, in which determinations are to be made as to whether native content is appropriately displayed. In some implementations, the application identifiers identify the application 205 and a specific version of the identified application 205.In some implementations, the application identifiers identify a type of application 205 (e.g., based on a type of client device 125 on which the application 205 is installed). For example, the application identifier can identify a specific operating system on which the client device 125 runs and with which the application 205 is compatible.

[0040] In some implementations, the content query module 130 searches the database 125 to determine if the application identifier received in the content query (e.g., the application identifier that identifies that the query is sent from application 205) matches any of the stored application identifiers used to identify candidate applications in order to trigger a determination of whether to appropriately display native content.

[0041] In some implementations, once the content query module 130 identifies the application 205 as one to be evaluated for a presentation execution of native content, the content query module 130 determines whether the client device 125 that sent the content query receives the interaction acknowledgment script 215 containing the selected native content element. For example, the database 145 may also store a predetermined percentage that is associated with, or otherwise linked to, each of the candidate application identifiers stored to trigger an evaluation of a presentation execution of native content with respect to the applications identified by those candidate application identifiers.The predetermined percentage can identify a percentage of content queries to be classified as script queries in order to send the interaction acknowledgment script 215 in response to the content query classified as a script query.

[0042] As an example, if an application identifier is assigned to application 205 a predetermined percentage of 35%, the content query module 130 can then classify 35% of the content queries received by application 205 (e.g., across a multitude of client devices 125) as script queries, so that the interaction acknowledgment script 215 is sent to the client devices 125 that sent those content queries classified as script queries. In some implementations, application 205 corresponds to a sample application installed on a single client device. In some implementations, application 205 corresponds to a specific version of a specific application installed across a multitude of client devices. In some implementations, application 205 corresponds to any version of application 205 across a multitude of client devices 125.Accordingly, the remaining 65% of content queries can be classified as non-script queries, so that the client devices 125 that sent content queries which the data processing system 110 classified as non-script content queries do not receive the interaction acknowledgment script 215, but do receive a queried native content item. In some implementations, the content query module 130 classifies content queries as either scripted or non-scripted content queries arbitrarily (e.g., at a rate prescribed by the associated predetermined percentage). In other implementations, the content query module 130 classifies content queries as scripted content queries based on a predetermined pattern (e.g., every fourth content query is classified as a scripted content query if the predetermined percentage is 25%).

[0043] In addition to sending the interaction acknowledgment scripts 215 in response to content queries classified as script content queries, the content picker module 135 can select and provide a native content element (e.g., the one originally queried) and an interstitial content element, so that the data processing system 110 provides a content package to signal 304 containing the interaction acknowledgment script 215, the queried native content element, and an interstitial content element. The interstitial content element can contain a content element that is presented and displayed, thus occupying a substantial portion of the application 205's display area. For example, the interstitial content element can overlay the application 205's display. In some implementations, the core of the interstitial content element is essentially identical to that of the native content element contained in the content package.In some implementations, the interstitial content element contains inline HTML code that is sent along with the native content element.

[0044] In some implementations, the native content can be rendered by application 205 and presented for display within application 205's environment, at block 305 or signal 306. In such implementations, a native content element containing one or more content components is transferred from the data processing system 110 to the client device 125, where application 205 on the client device 125 assembles the content components of the content element according to predefined parameters of application 205. Once application 205 has rendered the native content using the content components, it presents the rendered native content for display within application 205's environment, so that the rendered content appears to have the same appearance as content generated by the content publisher 120 within application 205.In some implementations, the content publisher 120 can set the predefined parameters that the application 205 uses to assemble and present native content elements. However, because these parameters vary from application to application and rely on individual content publishers 120 for setup, some content publishers 120 may implement parameters incorrectly, resulting in native content elements being presented incorrectly on the client device 125, which can lead to unintended interactions with the incorrectly presented native content elements.

[0045] For example, the content publisher 120 can configure parameters for a display regarding its application. These parameters can prescribe the appearance of native content components received from the data processing system 110. For instance, the received native content components might include elements of a native content element, such as, but not limited to, a heading, image, body, call to action, logo, attribution, clickable URL, or the like. The configured parameters can specify the appearance of the native content components, such as their location within a native content element, color, font, font size, interaction type (e.g., tap or swipe for interaction), and so on. As such, the parameters provide instructions to the application for rendering and presenting received native content components within an application.In some implementations, Content Publisher 120 provides the native content parameters within the Content Publisher 120 application.

[0046] However, since the content parameters are set and implemented by the respective content publishers, there can be significant variation in how native content components are displayed within applications. Furthermore, content publishers may set their parameters inappropriately, resulting in native content components being rendered and displayed incorrectly within an application. This can lead to unintended interactions with an inappropriately displayed native content element by a user (e.g., if a native content element inappropriately obscures a separate link that a user intends to click). Because these display parameters of native content can vary from application to application, from version to version of the same application (e.g., version 1.0 compared to version 2.0), or between different types of the same application (e.g.,(of an application configured for one operating system, compared to the same application configured for another operating system), the parameters set by Content Publisher 120 may vary similarly with respect to different applications, different versions of applications, and different types of applications, and native content elements may therefore be displayed inappropriately with respect to different versions or types of the same application.

[0047] In some implementations, Signal 308's interaction event detection module 220, running on client device 125, can cause client device 125 to detect an interaction event with respect to the presented native content element within application 205. For example, the interaction event could include a click, swipe, tap, cursor hover, or any other interaction with the presented native content element.

[0048] In some implementations, at block 311, in response to the detection of the interaction event at signal 308, the content element presentation module 225, running on client device 125, can cause client device 125 to render the interstitial content element received in the content packet at signal 304. In some implementations, the application 205 running on client device 125 can cause client device 125 to detect whether the interstitial content element was received in accordance with the presented native content element (e.g., containing the same or similar information presented in the native content element). For example, application 205 can cause client device 125 to detect that the content packet at block 304 contains more than the queried native content element.In some implementations, application 205 causes client device 125 to detect that the content package contains the interstitial content element by determining whether an HTML inline code representing the interstitial content element exists within the content package. In some implementations, the content package contains a flag or some other indicator to specify that the content package contains the interstitial content element in addition to the native content element, and application 205 can cause client device 125 to detect the presence of the flag in the content package.

[0049] In some implementations, in response to the detection of the presence of the interstitial content element in the received content packet and the detection of the interaction event at signal 308, the content element presentation module 225, running on client device 125, can cause client device 125 to render the interstitial content element received in the content packet at signal 304. In some implementations, the content element presentation module 225 causes client device 125 to parse the inline HTML code received in the content packet for instructions on rendering the interstitial content element and then renders the interstitial content element based on those instructions. Once the interstitial content element has been rendered, the content element presentation module 225 can cause the client device 125 to present the interstitial content element within the application 205.For example, the interstitial content element can be presented in information resource 210 (e.g., by occupying a substantial part of the space in information resource 210).

[0050] In some implementations, Signal 312's acknowledgment generation module 230, running on client device 125, can cause client device 125 to detect an interaction event with respect to the presented interstitial content element within application 205. For example, the interaction event could include clicking, swiping, tapping, hovering with a cursor, or any other interaction with the presented interstitial content element. In response to the detection of the interaction event with respect to the interstitial content element, acknowledgment generation module 230 can cause client device 125 to send an acknowledgment of the interaction event with respect to the native content element to the data processing system 110.

[0051] In some implementations, the data processing system 110 receives the acknowledgment of the interaction event regarding the native content element and can implement execution or performance evaluations based on received acknowledgments of interaction events of native content across a variety of client devices 125. For example, as described above, the data processing system 110 can select a predetermined percentage of client devices 125 that query native content elements via the application 205 such that they are client devices 125 that receive the content package containing the native content element, the interstitial content element, and the interaction acknowledgment script (e.g.,A content package subset of all client devices 125 that query native content items from the application 205, while the remaining percentage of client devices 125 are intended to receive the native content item without the interstitial content item and without the interaction acknowledgment script (e.g., a non-content package subset of all client devices 125 that query native content items from the application 205). Accordingly, in some implementations, the data processing system 110 compares and analyzes performance metrics of the content package subset and the non-content package subset when determining whether the content publisher 120 is correctly rendering and displaying native content items (e.g., whether the first interaction event regarding the native content item at signal 308 was purposeful or accidental due to an inappropriate presentation of the native content item).

[0052] In some implementations, the data processing system 110 determines a performance metric regarding the content packet subset of all client devices 125 that made a query for native content items from the application 205. For example, the performance metric might include a percentage or ratio representing the number of acknowledgments received by the data processing system 110 (e.g., in Signal 314) compared to the number of client devices 125 to which the content packet was sent (e.g., in Signal 304). Similarly, the content packet subset performance metric might represent the percentage of users who purposefully interacted with the native content item, relative to the number of individual content packets sent to the content packet subset of all client devices 125 that made a query for native content items from the application 205.Similarly, in some implementations, the data processing system 110 determines a performance metric regarding the non-content-package subset of all client devices 125 that made a query for native content items from the application 205. For example, the performance metric might include a percentage or ratio representing the number of interaction events related to the native content item received by the data processing system 110, compared to the number of client devices 125 to which the requested native content item was sent. Accordingly, the non-content-package subset performance metric might represent a percentage of users who interacted with the native content item (e.g.,(both intentionally and accidentally) interacted, with respect to the number of individual native content items sent to the non-content package subset of all client devices 125 that made a native content item query from the application 205.

[0053] In some implementations, the data processing system compares the performance metric of the content package subset with the performance metric of the non-content package subset to determine whether the native content element is displayed inappropriately in application 205. For example, data processing system 110 might compare a difference between the performance metric of the content package subset and the performance metric of the non-content package subset with a predetermined threshold. In some implementations, if the difference between the two performance metrics exceeds the predetermined threshold, then data processing system 110 identifies application 205 as one that displays native content elements inappropriately. As an example, data processing system 110 might identify application 205 (e.g., a specific version of application 205, application 205 across multiple versions, this particular example of application 205, etc.).) as one that displays native content inappropriately if the performance metric of the non-content-package subset is greater than the performance metric of the content-package subset by at least one predetermined threshold. In some implementations, the predetermined threshold is input to the data processing system 110 and stored in the database 145 and can be any suitable value to determine what difference between the performance metrics (e.g., represented as percentages) determines that the application 205 displays native content items inappropriately (e.g., a predetermined threshold value can be 5%, 10%, 15%, and so on).

[0054] In some implementations, the data processing system 110 determines that the application is inappropriately displaying 205 native content elements based on the performance metric of the content package subset, but not on the performance metric of the non-content package subset. The data processing system 110 can compare the content package subset performance metric itself against a predetermined threshold and make a determination based on the comparison.For example, if the performance metric contains a percentage or ratio representing the number of acknowledgments received by the data processing system 110 compared to the number of client devices 125 to which the content packet was sent, the data processing system 110 can compare the percentage performance metric to a threshold percentage, and, if the performance metric is below the threshold percentage, the data processing system 110 can identify the application 205 as one that is inappropriately displaying native content.

[0055] In some implementations, in response to identifying application 205 as inappropriately displaying native content, data processing system 110 may perform one or more restrictive procedures on application 205. For example, data processing system 110 may add application 205 to content restriction list 150. Content restriction list 150 may be stored in database 145 and may contain the application identifier that identifies application 205, so that data processing system 110 can cross-reference the list when receiving incoming subsequent content queries that include application 205's application identifier.If the data processing system 110 detects a match between an application identifier of a querying application and an application identifier on the content restriction list 150, the data processing system 110 can restrict itself from transmitting a native content element in response to the query (e.g., until the execution of a display of a native content element by the application 205 is fixed by the content publisher 120). In some implementations, the data processing system 110 receives an alert from the content publisher 120 of the application 205 indicating that the inappropriate presentation of native content has been resolved (e.g., in response to a corresponding alert sent to the content publisher 120 indicating that its application 205 is inappropriately displaying native content).In addition to application 205, which sends the acknowledgment of the interaction event of the native content element, in some implementations, and in response to the detection of the interaction event at signal 312, the query generation module 235, running on client device 125, can cause client device 125 to send a query to access an information resource linked by the interstitial content element. The query can be sent to a server of content provider 115, which provided the native content element and the interstitial content element displayed in application 205.

[0056] Fig. Figure 4 is a Flowchart 400 that depicts a method for detecting inappropriate presentation of content elements by applications running on client devices, according to some implementations.

[0057] In some implementations, procedure 400 involves client device 125, which transmits a content query for a content element (block 405), and data processing system 110, which receives the content query for a content element (block 410). Procedure 400 further involves data processing system 110, which transmits a content packet containing an interaction acknowledgment script (block 415), and client device 125, which receives the content packet containing the interaction acknowledgment script (block 420). Procedure 400 also involves client device 125, which presents the first content element (block 425). Finally, procedure 400 involves client device 125, which detects the first interaction event with the presented content element (block 430). Furthermore, procedure 400 includes client device 125, which presents a second content element (block 435).Procedure 400 further involves client device 125, which detects a second interaction event with the presented content element (block 440). Procedure 400 further involves client device 125, which transmits an acknowledgment of the first interaction event (block 445). Procedure 400 further involves data processing system 110, which receives the acknowledgment of the first interaction event (block 450). Procedure 400 further involves client device 125, which transmits a query for an information resource of the second content element (block 455). Procedure 400 further involves client device 125, which presents the information resource of the second content element (block 460).

[0058] In some implementations, procedure 400 specifically involves the client device 125, which transmits a content query about a content element (block 405), and the data processing system 110, which receives the content query about a content element (block 410). The client device 125 can transmit a content query to the data processing system 110. In some implementations, the content query is a query about a native content element. In some implementations, a native content element can appear to have the same appearance as content generated by the content publisher 120 within the application's environment. For example, a native content element can have visual characteristics that match or substantially match the surrounding environment of the application in which the native content element is intended to be displayed.In some implementations, the data processing system 110 sends a native content element to the client device 125 in the form of a variety of content components that constitute the content element, and the content components are rendered by the application on the client device 125. For example, the content components may include, but are not limited to, a heading, an image, a body, a call to action, a logo, an attribution, a clickable URL, or the like.

[0059] In addition to sending the interaction acknowledgment scripts 215 in response to content queries classified as script content queries, the content picker module 135 can select and provide a native content element (e.g., the originally queried one) and an interstitial content element, so that the data processing system 110 provides a content package to signal 304 containing the interaction acknowledgment script 215, the queried native content element, and an interstitial content element. The interstitial content element can contain a content element that is presented and displayed, thus occupying a substantial portion of the application 205's display area. For example, the interstitial content element can overlay the application 205's display.In some implementations, the core of the interstitial content element is essentially identical to that of the native content element included in the content package. In some implementations, the interstitial content element contains inline HTML code that is sent along with the native content element.

[0060] Procedure 400 further involves the data processing system 110, which transmits a content package containing an interaction confirmation script (block 415), and the client device 125, which receives the content package containing the interaction confirmation script (block 420). In addition to sending the interaction confirmation scripts in response to the content request, the data processing system 110 can select and provide a native content element (e.g., as the originally requested element) and an interstitial content element, so that the data processing system 110 provides a content package containing the interaction confirmation script, the requested native content element, and an interstitial content element. The interstitial content element can include a content element that is presented and displayed, thus occupying a substantial portion of the application's display area.For example, the interstitial content element can overlay the application's display. In some implementations, the core of the interstitial content element is essentially identical to that of the native content element included in the content package. In some implementations, the interstitial content element includes inline HTML code that is sent along with the native content element.

[0061] Procedure 400 further involves the client device 125, which presents the first content element (block 425). In some implementations, native content can be rendered by the application and presented for display within the application's environment. In such implementations, the native content element, containing one or more content components, is transferred from the data processing system 110 to the client device 125, where the application assembles the content components of the content element according to predefined application parameters. Once the application has rendered the native content using the content components, it presents the rendered native content for display within the application's environment, so that the rendered content appears to have the same appearance as content generated by the content publisher 120 within the application.In some implementations, the content publisher 120 can set predefined parameters that the application uses as instructions for how to assemble and present native content elements. Because these parameters vary from application to application and rely on individual content publishers 120 to set them, content publishers 120 may implement parameters incorrectly, resulting in native content elements being presented incorrectly on the client device 125 and potentially leading to unintended interactions with the incorrectly presented native content elements.

[0062] For example, the content publisher 120 can configure parameters for a display regarding its application. These parameters can prescribe the appearance of native content components received from the data processing system 110. For instance, the received native content components might contain elements of a native content element, such as, but not limited to, a heading, image, body, call to action, logo, attribution, clickable URL, or the like. The configured parameters can specify the appearance of the native content components, such as their location within a native content element, color scheme, font, font size, interaction type (e.g., tap or swipe for interaction), and so on. Ultimately, the parameters provide instructions to the application for rendering and presenting received content components within an application.In some implementations, Content Publisher 120 provides the native content parameters within the Content Publisher 120 application.

[0063] However, since the content parameters are set and implemented by the respective content publishers, there can be significant variation in how native content components are displayed within applications. Furthermore, content publishers may set their parameters inappropriately, resulting in native content components being rendered and displayed incorrectly within an application. This can lead to unintended interactions with an inappropriately displayed native content element by a user (e.g., if a native content element inappropriately obscures a separate link that a user intends to click). Because these display parameters for native content can vary from application to application, from version to version of the same application (e.g., version 1.0 compared to version 2.0), or between different types of the same application (e.g.,(An application configured for one operating system, compared to the same application configured for another operating system), the parameters set by Content Publisher 120 may vary similarly with respect to different applications, different versions of applications, and different types of applications, and native content elements may therefore be displayed inappropriately with respect to different versions or types of the same application.

[0064] Procedure 400 further involves the client device 125, which detects an initial interaction event with the presented first content element (block 430). In some implementations, the client device 125 detects an interaction event with respect to the presented native content element within the application. For example, the interaction event could include a click, a swipe, a tap, a cursor hover, or any other interaction with the presented native content element.

[0065] Procedure 400 further involves client device 125, which presents a second content element (block 435). In some implementations, client device 125 renders the interstitial content element received in the content packet in response to the detection of the interaction event. In some implementations, the application running on client device 125 can cause client device 125 to detect whether the interstitial content element was received in accordance with the presented native content element (e.g., containing the same or similar information presented in the native content element). For example, the application can cause client device 125 to detect that the content packet contains more than the queried native content element.In some implementations, the application instructs the client device 125 to detect that the content package contains the interstitial content element by determining whether an HTML inline code representing the interstitial content element exists within the content package. In some implementations, the content package includes a flag or some other indicator to specify that the content package contains the interstitial content element in addition to the native content element, and the application can instruct the client device 125 to detect the presence of the flag within the content package.

[0066] In some implementations, in response to detecting the presence of the interstitial content element in the received content packet and in response to detecting the interaction event, the application running on client device 125 can cause client device 125 to render the interstitial content element received in the content packet. In some implementations, the application causes client device 125 to parse the inline HTML code received in the content packet for instructions on how to render the interstitial content element and then renders the interstitial content element based on these instructions. Once the interstitial content element is rendered, the application can cause client device 125 to present the interstitial content element within the application.

[0067] Procedure 400 further involves the client device 125, which detects a second interaction event with the presented second content element (Block 440). In some implementations, the application running on the client device 125 can cause the client device 125 to detect an interaction event with respect to the presented interstitial content element within the application 205. For example, the interaction event can include clicking, swiping, tapping, hovering with a cursor, or any other interaction with the presented interstitial content element.

[0068] Procedure 400 further involves client device 125, which transmits an acknowledgment of the first interaction event (block 445). Procedure 400 also involves data processing system 110, which receives the acknowledgment of the first interaction event (block 450). In response to the detection of the interaction event with respect to the interstitial content element, the application can cause client device 125 to send an acknowledgment of the interaction event with respect to the native content element to data processing system 110.

[0069] Procedure 400 further involves client device 125, which transmits a query to an information resource of the second content element (block 455). Procedure 400 further involves client device 125, which presents the information resource of the second content element (block 460). The query can be sent to a server of content provider 115, which provided the native content element and the interstitial content element displayed in application 205.

[0070] Fig. 5 is a flowchart that describes a procedure 500 for detecting an inappropriate presentation of content elements by a data processing system in a communication with the applications running on the client devices, which in relation to Fig. As described in section 4, according to some implementations, procedure 500 involves the data processing system 110, which receives a multitude of content queries from a multitude of client devices (block 505). Procedure 500 further involves the data processing system 110, which transmits a content packet to a first subset of the multitude of client devices (block 510). Procedure 500 further involves the data processing system 110, which transmits a first content element to a second subset of the multitude of client devices (block 515). Procedure 500 further involves the data processing system 110, which determines a first performance metric with respect to the transmitted content packet (block 520). Procedure 500 further involves the data processing system 110, which determines a second performance metric with respect to the first content element (block 525).Furthermore, procedure 500 incorporates data processing system 110, which compares the first performance metric with the second performance metric (block 530). Additionally, based on this comparison, procedure 500 includes adding an application identifier to a content restriction list (block 535).

[0071] In particular, in some implementations, Procedure 500 involves Data Processing System 110, which receives a multitude of content queries from a multitude of client devices (Block 505). In some implementations, in response to receiving content queries from client devices, Data Processing System 110 can classify the received queries as those that will trigger packaging and transmission of the interaction acknowledgment script along with a selected native content element and a similar interstitial content element. In some implementations, a content query includes an application identifier to identify an application. Accordingly, Data Processing System 110 can maintain a list of applications that are candidate applications for which decisions should be made regarding whether native content is appropriately displayed.In some implementations, application identifiers identify the application and a specific version of that application. In other implementations, application identifiers identify the application type (e.g., based on the type of client device on which the application is installed). For example, the application identifier might identify the specific operating system on which the client device is running and with which the application is compatible.

[0072] In some implementations, the data processing system 110 searches the database to determine if the application identifier received in the content query (e.g., the application identifier that identifies that the query is sent by the application) matches any of the stored application identifiers used to identify candidate applications, in order to trigger a determination of whether native content should be appropriately displayed.

[0073] Procedure 500 further involves data processing system 110, which transmits a content package to a first subset of the multitude of client devices (Block 510). Procedure 500 further involves data processing system 110, which transmits a first content item to a second subset of the multitude of client devices (Block 515). In some implementations, when data processing system 110 identifies the application as one to be evaluated for presentation execution of native content, data processing system 110 determines whether the client device that sent the content request receives the interaction confirmation script containing the selected native content item.For example, the database can also store a predetermined percentage that is assigned to or otherwise associated with each of the candidate application identifiers stored to trigger an evaluation of a presentation execution of native content with respect to the applications identified by those candidate application identifiers. The predetermined percentage can identify a percentage of content queries to be classified as script queries in order to send the interaction confirmation script in response to the content query classified as a script query.

[0074] As an example, if an application identifier is assigned to the application at a predetermined percentage of 35%, the data processing system 110 can then classify 35% of the content queries received by the application (e.g., across a multitude of client devices) as script queries, so that the interaction acknowledgment script is sent to the client devices that sent those content queries classified as script queries. In some implementations, the application corresponds to an application sample installed on a single client device. In some implementations, the application corresponds to a specific version of a specific application installed across a multitude of client devices. In some implementations, the application corresponds to any version of the application across a multitude of client devices.Accordingly, the remaining 65% of content queries can be classified as non-script queries, so that client devices sending content queries classified as non-script content queries by the data processing system 110 will not receive the interaction acknowledgment script 215, but will receive a queried native content item. In some implementations, the data processing system 110 classifies content queries as scripted content queries or non-scripted content queries arbitrarily (e.g., at a rate determined by the assigned predetermined percentage). In other implementations, the data processing system 110 classifies content queries as scripted content queries based on a predetermined pattern (e.g., every fourth content query is classified as a scripted content query if the predetermined percentage is 25%).

[0075] In addition to sending the interaction acknowledgment scripts in response to content queries classified as script content queries, the data processing system 110 can select and provide a native content element (e.g., the one originally queried) and an interstitial content element. Thus, the data processing system 110 provides the content package containing the interaction acknowledgment script 215, the queried native content element, and an interstitial content element. The interstitial content element can contain a content element that is presented and displayed, occupying a substantial portion of the application's display area. For example, the interstitial content element can overlay the application's display. In some implementations, the core of the interstitial content element is essentially identical to that of the native content element contained in the content package.In some implementations, the interstitial content element contains inline HTML code that is sent along with the native content element.

[0076] Procedure 500 further involves the data processing system 110, which determines an initial performance metric regarding the transmitted content package (Block 520). In some implementations, the data processing system 110 receives an acknowledgment of the interaction event regarding the native content element and can implement performance evaluations based on received acknowledgments of native content interaction events across a variety of client devices. For example, as described above, the data processing system 110 can select a predetermined percentage of client devices querying native content elements via the application as client devices that receive the content package containing the native content element, the interstitial content element, and the interaction acknowledgment script (e.g.,A content package subset of all client devices that query the application for native content items, while the remaining percentage of client devices should receive the native content item without the interstitial content item and without the interaction acknowledgment script (e.g., a non-content package subset of all client devices that query the application for native content items). Accordingly, in some implementations, the data processing system compares and analyzes performance metrics of the content package subset and the non-content package subset when determining whether the content publisher is correctly rendering and displaying native content items (e.g., whether the first interaction event with respect to the native content item at signal 308 was purposeful or accidental due to an inappropriate presentation of the native content item).

[0077] In some implementations, Data Processing System 110 determines a performance metric regarding the content package subset of all client devices that made a query for native content items from the application. For example, the performance metric might include a percentage or ratio representing the number of acknowledgments received by Data Processing System 110 compared to the number of client devices to which the content package was sent. Similarly, the content package subset performance metric might represent the percentage of users who purposefully interacted with the native content item, relative to the number of individual content packages sent to the content package subset of all client devices that made a query for native content items from the application.

[0078] Procedure 500 further involves the data processing system 110, which determines a second performance metric with respect to the first content item (Block 525). Similarly, in some implementations, the data processing system 110 determines a performance metric with respect to the non-content-package subset of all client devices that made a query for native content items from the application. For example, the performance metric might include a percentage or ratio representing the number of interaction event reports with respect to the native content item received by the data processing system 110, compared to the number of client devices to which the queried native content item was sent. Accordingly, the non-content-package subset performance metric might represent a percentage of users who interacted with the native content item (e.g.,(both intentionally and accidentally) interacted, with respect to the number of individual native content items sent to the non-content package subset of all client devices that made a native content item query from the application.

[0079] Procedure 500 further incorporates the data processing system 110, which compares the first performance metric with the second performance metric (Block 530). In some implementations, the data processing system 110 compares the content package subset performance metric with the non-content package subset performance metric to determine whether the native content element is displayed inappropriately in the application. For example, the data processing system 110 may compare a difference between the content package subset performance metric and the non-content package subset performance metric with a predetermined threshold. In some implementations, if the difference between the two performance metrics exceeds the predetermined threshold, then the data processing system 110 identifies the application as one that displays native content elements inappropriately. As an example, the data processing system 110 may identify the application (e.g.,(a specific version of the application, the application across multiple versions, this particular example of application 205, etc.) as one that displays native content inappropriately if the performance metric of the non-content-package subset is greater than the performance metric of the content-package subset by at least a predetermined threshold. In some implementations, the predetermined threshold is entered into the data processing system 110 and stored in the database, and can be any suitable value to determine what difference between the performance metrics (e.g., represented as percentages) determines that the application displays native content elements inappropriately (e.g., a predetermined threshold value can be 5%, 10%, 15%, and so on).

[0080] In some implementations, the data processing system 110 determines that the application is inappropriately displaying 205 native content elements, based on the performance metric of the content package subset, but not on the performance metric of the non-content package subset. The data processing system 110 can compare the content package subset's performance metric itself against a predetermined threshold and make a determination based on the comparison.For example, if the performance metric contains a percentage or ratio representing the number of acknowledgments received by the data processing system 110 compared to the number of client devices 125 to which the content packet was sent, the data processing system 110 can compare the percentage performance metric to a threshold percentage, and if the performance metric is below the threshold percentage, the data processing system 110 can identify the application as one that is inappropriately displaying native content.

[0081] Procedure 500 further involves the data processing system 110, which, based on the comparison, adds an application identifier to a content constraint list (Block 535). In some implementations, in response to identifying the application as inappropriately displaying native content, the data processing system 110 may perform one or more restrictive procedures on the application. For example, the data processing system 110 may add the application to the content constraint list. The content constraint list may be stored in the database and contain the application identifier that identifies the application, so that the data processing system 110 can cross-reference it when it receives incoming subsequent content queries that include the application's application identifier.If the data processing system 110 detects a match between an application identifier of a requesting application and an application identifier on the content restriction list, the data processing system 110 can restrict itself from sending a native content element in response to the query (for example, until the execution of a display of a native content element by the application is determined by the content publisher 120). In some implementations, the data processing system 110 receives an alert from the application's content publisher indicating that the inappropriate presentation of native content has been resolved (for example, in response to a corresponding alert sent to the content publisher indicating that their application is inappropriately displaying native content).

[0082] Fig. Figure 6 is a block diagram that represents a general architecture for a Computer System 600, which can be used to implement elements of the systems and procedures described and illustrated herein (including System 110 and its components, such as the Content Query Module 130, the Content Selection Module 135, and the Script Provider Module 140), according to some implementations. Computer System 600 can be used to provide information over Network 105 for display. Computer System 600 of Fig. 6 comprises one or more processors 620 communicatively coupled to a memory 625, one or more communication interfaces 605, one or more output devices 610 (e.g., one or more display units), and one or more input devices 615. The processors 620 may be contained in the data processing system 110 or in other components of the system 110, such as the content query module 130 and the content selection module 135.

[0083] In the computer system 600 of Fig. 6. Memory 625 can comprise any computer-readable storage medium and can store computer instructions, such as processor-executable instructions for implementing the various functionalities described herein for respective systems, as well as any related data generated or received through one or more communication interfaces or one or more input devices (if any). Returning to System 110 of Fig. 1. Referring to this, the data processing system 110 can include the memory 625 to store information relating, among other things, to the availability of the stock of one or more units, reservations of one or more content units. The memory 625 can contain the database 145. The processor(s) 620 in Fig. 6 can be used to execute the instructions stored in memory 625 and can also read from or write to memory various pieces of information that are processed and / or generated according to the execution of the instructions.

[0084] The 620 processor of the Computer System 600, which was in Fig. The communication interface(s) shown in Figure 6 can also be communicatively coupled to or controlled by the communication interface(s) 605 to transmit or receive various information according to the execution of instructions. The communication interface(s) 605 can, for example, be coupled to a wired or wireless network, bus, or other means of communication and enable the computer system 600 to transmit information to or receive information from other devices (e.g., other computer systems). Although this is shown in the system in Fig. Unless explicitly shown in Figure 1, one or more communication interfaces can facilitate the flow of information between the components of System 600. In some implementations, the communication interface(s) (e.g., via various hardware or software components) may be configured to provide a website as an access portal to at least some aspects of Computer System 600. Examples of Communication Interfaces 605 include user interfaces (e.g., web pages) through which the user can communicate with the Data Processing System 110.

[0085] The output devices 610 of the computer system 600, which is in Fig.The input device(s) shown in Figure 6 may, for example, be provided to allow various pieces of information to be viewed or otherwise perceived in connection with the execution of instructions. The input device(s) 615 may, for example, be provided to allow a user to make manual adjustments, selections, input data, or interact with the processor in various ways during the execution of instructions. Additional information regarding a general computer system architecture that may be used for various systems discussed herein is further provided herein.

[0086] Implementations of the subject matter and the operations described herein may be implemented in a digital electronic circuit or in computer software embodied on a physical medium, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or combinations thereof. Implementations of the subject matter described herein may be implemented as one or more computer programs, i.e., as one or more modules containing computer program instructions encoded on a computer storage medium for execution by a data processing device or for controlling its operation. The program instructions may be written on an artificially generated, propagated signal, e.g.,A computer storage medium is a machine-generated electrical, optical, or electromagnetic signal that is produced to encode information for transmission to a suitable receiver for execution by a data processing device. A computer storage medium may be or include a computer-readable storage device, a computer-readable storage substrate, a freely addressable or serial access memory array or device, or a combination thereof. Furthermore, although a computer storage medium is not a propagated signal, it may be a source or destination of computer program instructions encoded in an artificially generated, propagated signal. The computer storage medium may also be or include one or more distinct physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0087] The features disclosed herein may be implemented in a smart TV module (or connected TV module, hybrid TV module, etc.) which may include a processing module configured to integrate internet connectivity with more conventional television programming sources (e.g., those received via cable, satellite, radio, or other signals). The smart TV module may be physically integrated into a television set or may include a separate device, such as a set-top box, Blu-ray or other digital media player, game console, hotel television system, and other companion devices. A smart TV module may be designed to allow viewers to search for and locate videos, movies, photos, and other content on the web, on a local cable television channel, on a satellite television channel, or stored on a local hard drive.A set-top box (STB) or set-top unit (STU) can be an information application device that may contain a tuner and connect to a television set and an external signal source, converting the signal into content that is then displayed on the television screen or other display device. A smart TV module can be configured to provide a home or top-level screen with icons for a variety of different applications, such as a web browser and various streaming media services, a connected cable or satellite media source, other web "channels," and so on. The smart TV module can also be configured to provide an electronic program guide (EPG) for the user.A companion application for the Smart TV module can run on a mobile computing device to provide the user with additional information about available programs, to allow the user to control the Smart TV module, etc. In alternative implementations, the features can be implemented on a laptop computer or other PC, a smartphone, another mobile phone, a handheld computer, a tablet PC, or another computing device.

[0088] The operations described in this description can be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.

[0089] The terms "data processing device," "data processing system," "user device," or "computer device" encompass all types of devices, equipment, and machines for processing data, including, for example, a programmable processor, a computer, a system-on-a-chip, or several or combinations thereof. The device may include a logic circuit for a specific purpose, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, such as code representing processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination thereof.The device and execution environment can implement various computer model infrastructures, such as web services, as well as distributed computing and grid computing infrastructures. The content query module 130 and the content selection module 135 can comprise or share one or more data processing devices, computer equipment, or processors.

[0090] A computer program (also known as a program, software, software application, module, software module, script, or code) can be written in any form of a programming language, including compiled or interpreted languages, declarative or procedural languages, and can be provided in any form, including a standalone executable program or module, component, subroutine, object, or other unit suitable for use in a computer environment. A computer program may, but need not, correspond to a file in a file system. A program may be contained in a portion of a file that includes other programs or data (such as one or more scripts stored in a markup language document), in a single file permanently associated with that program, or in multiple coordinated files (e.g.,A computer program can be stored in files that contain one or more modules, subprograms, or code snippets. It can be deployed to run on one computer or on multiple computers located at one site or distributed across multiple sites and connected via a communication network.

[0091] The processes and logical sequences described in this description can be performed by one or more programmable processors, which execute one or more computer programs to perform actions by processing input data and generating output. The processes and logical sequences can also be performed by a purpose-built logic circuit, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC), and devices can be implemented in the form of such circuits.

[0092] Processors suitable for executing a computer program include, for example, general-purpose and specialized microprocessors, as well as any one or more processors of any type of digital computer. Typically, a processor receives instructions and data from read-only memory or random-access memory, or both. The essential components of a computer are a processor for performing actions in accordance with instructions, and one or more storage devices for storing instructions and data. A computer also typically includes one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical hard disks, or is operationally coupled to them to receive data from or transfer data to them. However, a computer does not necessarily have to have these devices. Furthermore, a computer may, for example, be contained within another device, such as a...A computer may be integrated into a mobile phone, a personal digital assistant (PDA), a portable audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory elements such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM drives. The processor and memory may be supplemented by, or integrated into, a logic circuit for special purposes.

[0093] To enable user interaction, implementations of the item described in this description can be implemented on a computer with a display device, such as a cathode ray tube (CRT), plasma, or liquid crystal display (LCD) monitor, to show information to the user, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to enable user interaction; for example, feedback provided to the user can include any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including auditory, speech, or tactile input.Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on a user's client device in response to queries received by the web browser.

[0094] The implementations of the subject matter described in this specification can be implemented in a computer system that includes a backend component, such as a data server, or a middleware component, such as an application server, or a frontend component, such as a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. The system components can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), an inter-network (e.g., a 24 / 7 network), and a 24 / 7 network.the Internet) and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0095] The computer system, such as System 600 or System 110, can include clients and servers. For example, the System 110 might include one or more servers located in one or more data centers or on one or more server farms. A client and a server are generally located remotely and typically interact via a communication network. The client-server relationship arises from computer programs running on the respective computers, establishing a client-server relationship. In some implementations, a server sends data (e.g., an HTML page) to a client device (e.g., for displaying data and receiving user input from a user interacting with the client device). Data generated on the client device (e.g., a result of user interaction) can be received by the user device on the server.

[0096] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of the present disclosure or claims, but rather as descriptions of features specific to certain implementations of the systems and methods described herein. Certain features described in this description in connection with separate implementations may also be implemented in combination within a single implementation. Conversely, various features described in connection with a single implementation may also be implemented separately in multiple implementations or in any suitable combination thereof.Furthermore, although the features described above are initially claimed as acting in certain combinations, in some cases one or more features of a claimed combination can be removed from the combination and the claimed combination can be directed to a sub-combination or a variant of a sub-combination.

[0097] Similarly, although the operations in the drawings are shown in a specific sequence, this should not be interpreted as meaning that such operations must be performed in that specific or sequential order, or that all illustrated operations must be performed to achieve the desired results. In some cases, the actions specified in the claims can be performed in a different sequence and still achieve the desired results. Furthermore, the processes depicted in the accompanying figures do not necessarily require the specific or sequential order shown to achieve the desired results.

[0098] Under certain circumstances, multitasking and parallel processing can be advantageous. Furthermore, the separation of different system components in the implementations described above should not be interpreted as requiring such separation in all implementations. It should be understood that the described program components and systems can generally be integrated together in a single software product or bundled together in several software products. For example, the content query module 130 and the content selection module 135 can be part of the data processing system 110, a single module, a logic device comprising one or more processing modules, one or more servers, or part of a search engine.

[0099] Having described several illustrative implementations, it is evident that the foregoing is illustrative and not limiting in nature, presented in an exemplary manner. In particular, although many of the examples presented herein involve specific combinations of procedures or system elements, these procedures and elements can be combined in other ways to achieve the same goals. Actions, elements, and features discussed only in connection with one implementation are not precluded from playing a similar role in other implementations.

[0100] In implementations of this disclosure, the content or content elements can be any suitable content. In some applications, the content or content elements can be advertisements (“ads”); however, this disclosure is not limited to this. If the content or content elements are advertisements, the content server 205 can be an ad server and / or the provider of third-party content can be an advertiser.

[0101] In some implementations, the query for third-party content may also include a minimum sales amount (or minimum bid value) for displaying a third-party content item in the information resource. In some implementations, the Content Selector Module 135 may communicate with an auction system, such as an advertising auction system, and provide the auction system information contained in the third-party content query received from the client device. The Content Selector Module 135 may also receive one or more third-party content items from the auction system in response to the information contained in the third-party content query.

[0102] In some implementations, the data processing system 110 may include an advertising auction system configured to host auctions. In other implementations, the data processing system 110 does not include the auction system but is configured to communicate with the auction system over the network 105.

[0103] In some applications, a landing page may contain additional product information for a product advertised in the content element. A conversion related to the content element can occur on the landing page; for example, a user might purchase the product or service advertised by a content element on the landing page. A conversion can also occur when a user clicks on a content element and then performs an action such as making a purchase, signing up, registering, etc., on the landing page.

[0104] The language and terminology used herein serve a descriptive purpose and should not be considered restrictive. The use of the words "including," "comprehensive," "exhibiting," "containing," "incorporating," "characterized by," "characterized by," and variations thereof is intended to mean that the items listed thereafter, their equivalents, and additional items, as well as alternative implementations consisting solely of the items listed thereafter, are included. In an implementation, the systems and procedures described herein consist of one, any combination of more than one, or all of the elements, modes of operation, or components described herein.

[0105] Any references to implementations, elements, or modes of operation of the systems and procedures mentioned herein in the singular may also include implementations that comprise a multitude of such elements, and any plural references herein to an implementation, element, or mode of operation of any kind may also include implementations that comprise only a single element. References to the singular or plural form are not intended to limit the systems and procedures disclosed herein, their components, modes of operation, or elements to single or multiple configurations. References to a mode of operation or element of any kind, based on any information, mode of operation, or element, may include implementations whose mode of operation or element is based, at least in part, on any information, mode of operation, or element of any kind.

[0106] Any implementations disclosed herein may be combined with any other implementation, and references to “an implementation,” “some implementations,” “an alternative implementation,” “various implementations,” “an implementation,” or the like are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with the implementation may be incorporated into at least one implementation. Such terms as used herein do not necessarily refer to the same implementation. Any implementation may be combined with any other implementation, including or exclusively, in any manner consistent with the aspects and implementations disclosed herein.

[0107] References to "or" can be interpreted as including, so that any terms described with "or" can refer to one, more than one, or each of the described terms.

[0108] If technical features in the drawings, the detailed description, or any claim are followed by reference numerals, these reference numerals have been introduced solely to improve the readability of the drawings, the detailed description, and the claims. Accordingly, neither the use nor the absence of reference numerals has a limiting effect on the scope of the claim elements.

[0109] The systems and methods described herein can be embodied in other specific forms without deviating from their essential characteristics. Although the examples provided here relate to controlling the display of content from information resources, the systems and methods described herein can also be applied to other environments. The preceding implementations are considered illustrative rather than limiting to the systems and methods described herein. The scope of the systems and methods described herein is thus defined more by the appended claims than by the preceding description, and modifications that fall within the scope and equivalence of the claims are included therein.

Claims

[1] Method (400) for detecting an inappropriate implementation of a presentation of content elements by applications running on client devices, comprising: from a first server, received (420), by an application (205) running on a client device (125) containing one or more data processors, a content package containing a first content element (245), a second content element (250) and an interaction acknowledgment script (215), wherein the interaction acknowledgment script (215) is configured to run within the application (205) running on the client device (125) and to cause the application (205) to: in response to the first content element (245) presented within the application (205) to the client device (125), to detect (430) a first interaction event with the first content element (245); upon detection of the first interaction event at the first content element (245), to present the second content element (250) within the application (205) (435); to detect a second interaction event with the second content element (250) (440); to transmit an acknowledgment of the first interaction event to the first server upon detection of the second interaction event (445); and to transmit a query (450) to a second server to access an information resource (210) linked to the second content element (250); and through the client device (125) to present the information resource (210) in response to the second interaction event at the second content element (250) (460), wherein the first server is configured to maintain a content restriction list (150) in a database (145), wherein the content restriction list (150) contains a plurality of application identifiers corresponding to a plurality of applications, wherein the content restriction list (150) is used to restrict a selection of the first content element (245) for transmission in response to content queries from the plurality of applications to which the application identifiers on the list correspond, and the first server is further set up for: in response to a failure to receive confirmation from the application (205), to add an application identifier of the application (205) to the content restriction list (150); and as a response to receiving a subsequent content query, to determine the selection of the first content element (245) for transmission to the application which corresponds to the application identifier on the content restriction list (150). [2] Method (400) according to claim 1, wherein the first content element (245) contains a native content element configured to adjust display parameters of other content elements of the application (205). [3] Method (400) according to claim 2, wherein the second content element (250) contains an interstitial content element configured to occupy a major part of a display area of ​​the application (205) when it is displayed. [4] Method (400) according to claim 1, wherein the second content element (250) contains a script containing instructions for the application (205) to detect the second interaction event and transmit the acknowledgment of the first interaction event. [5] Method (400) according to claim 1, wherein the first content element (245) contains the same text as a text contained in the second content element (250). [6] Method (400) according to claim 1, wherein the interaction confirmation script (215) contains further computer-executable instructions which, when executed by the application (205), cause the application (205) to transmit a statement of the first interaction event to the first server upon detection of the first interaction event. [7] Method according to claim 1, wherein the confirmation of the first interaction event includes an application identifier for identifying the application (205). [8] System for detecting an inappropriate implementation of a presentation of content elements by applications running on client devices, comprising: a processor (620); and a memory coupled to the processor (620), wherein the memory stores computer-executable instructions which, when executed by the processor (620), cause the processor (620) to: by an application (205) running on a client device (125) to receive from a first server a content packet containing a first content element (245), a second content element (250) and an interaction confirmation script (215), wherein the interaction confirmation script (215) is set up to run within the application (205) running on the client device (125) and to cause the application (205) to: to detect a first interaction event with the first content element (245) in response to the first content element (245) presented within the application (205) to the client device (125); upon detection of the first interaction event at the first content element (245), the second content element (250) is presented within the application (205); to detect a second interaction event with the second content element (250); Upon detection of the second interaction event, an acknowledgment of the first interaction event is transmitted to the first server; and to transmit a query to a second server to access an information resource (210) linked to the second content element (250); and through the client device (125) to present the information resource (210) in response to the second interaction event at the second content element (250), wherein the first server is configured to maintain a content restriction list (150) in a database (145), wherein the content restriction list (150) contains a plurality of application identifiers corresponding to a plurality of applications, wherein the content restriction list (150) is used to restrict a selection of the first content element (245) for transmission in response to content queries from the plurality of applications to which the application identifiers on the list correspond, and the first server is further set up for: in response to a failure to receive confirmation from the application (205), to add an application identifier of the application (205) to the content restriction list (150); and as a response to receiving a subsequent content query, to determine the selection of the first content element (245) for transmission to the application which corresponds to the application identifier on the content restriction list (150). [9] System according to claim 8, wherein the first content element contains a native content element configured to adjust display parameters of other content elements of the application (205). [10] System according to claim 9, wherein the second content element (250) contains an interstitial content element configured to occupy a major part of a display area of ​​the application (205) when it is displayed. [11] System according to claim 8, wherein the second content element (250) contains a script containing instructions for the application (205) to detect the second interaction event and transmit the acknowledgment of the first interaction event. [12] System according to claim 8, wherein the first content element (245) contains the same text as a text contained in the second content element (250). [13] System according to claim 8, wherein the interaction confirmation script (215) contains further computer-executable instructions which, when executed by the application (205), cause the application (205) to transmit a statement of the first interaction event to the first server upon detection of the first interaction event. [14] System according to claim 8, wherein the confirmation of the first interaction event includes an application identifier for identifying the application (205). [15] Method (500) for detecting an inappropriate implementation of a presentation of content elements in applications running on client devices, comprising: Receiving (505) by a data processing system containing one or more processors, content queries from a plurality of client devices, each of the content queries being generated by an application (205) running on a respective client device of the plurality of client devices; Transmitted (510) by the data processing system to each client device of a first subset of the plurality of client devices in response to a content query from the client device, a content package containing a first content element (245), a second content element (250) and an interaction acknowledgment script (215), wherein the interaction acknowledgment script (215) is configured to cause the application (205): to configure the first content element (245) to be presented in an initial presentation format, to present the first content element (245) in the first presentation format, to display the second content element (250) in response to a detection of a first interaction event at the first content element (245), and in response to the detection of a second interaction event at the second content element (250) to transmit a confirmation to the data processing system; Transfer (515) of the first content element (245) by the data processing system to each client device of a second subset of the plurality of client devices in response to a content query from the client device; Determine (520), by the data processing system, an initial performance metric of the content package, based on acknowledgments received from the first subset of the multitude of client devices; Determine (525), by the data processing system, a second performance metric of the first content element (245), based on acknowledgments received from the second subset of client devices; Compare (530), through the data processing system, the first performance metric with the second performance metric; and Adding (535), by the data processing system based on the comparison, an application identifier of the application (205) to a content restriction list (150) to restrict a selection of the first content element (245) for a transmission in response to a subsequent content query that identifies the application (205). [16] Method (500) according to claim 15, wherein the first content element (245) contains a native content element configured to adjust display parameters of other content elements of the application (205).

Images