A selective advertisement removal method based on object detection and motion recognition using smartphone sensors

Abstract

The present invention relates to a method for selectively removing advertisements based on motion recognition and object detection utilizing smartphone sensors. The present invention is a method performed on a computing device comprising one or more processors and a memory storing one or more programs executed by said one or more processors, characterized by comprising: a step of checking whether an advertisement is displayed on a smartphone screen; a step of performing a sensing operation of said smartphone and transmitting it when said advertisement is displayed on the smartphone screen; a step of detecting said sensing operation and transmitting a captured image of said advertisement to an AI model; a step of analyzing data of said captured image of said advertisement in said AI model to determine whether there is an ad removal button; and a step of, when an ad removal button is determined in said step, capturing said advertisement and touching it to delete it.

Description

Technology Field

[0001] The present invention relates to a method for selective ad removal based on motion recognition and object detection using smartphone sensors. Background Technology

[0003] As the digital advertising market grows, countless advertisements are flooding various web pages and apps. In particular, since the smartphone mobile environment has a relatively smaller screen compared to a PC, just a few advertisements often cover most of the screen. Furthermore, because removal buttons such as 'X' and 'Close' within the advertisements are very small, users often end up navigating to the ad page while trying to delete the advertisement.

[0004] Furthermore, existing ad-blocking applications have a disadvantage and limitation in that they are sometimes unable to completely block ads, and even potentially interesting advertisements are blocked. In such cases, users may feel severely annoyed, and this often hinders rapid searching or information gathering.

[0005] Therefore, there has been a demand for the development of an ad removal service that allows users to easily delete ads without touching them.

[0006] In addition, there has been a demand for the development of a feature that saves advertisements appearing on smartphones, allowing users to review them later when needed. Prior art literature

[0008] Republic of Korea Patent Registration No. 10-1364490 Republic of Korea Patent Registration No. 10-2539684 The problem to be solved

[0009] Accordingly, the present invention aims to provide a selective ad removal method based on motion recognition and object detection using a smartphone sensor, which allows a user to easily remove an ad without touching it when an ad appears on the screen while using a smartphone, in order to solve this problem.

[0010] In addition, there are cases where users accidentally touch the screen while removing ads and are redirected to the advertiser's page; the purpose is to enable easy and selective removal of ads without having to remove all of them by automatically touching the center of the ad removal button.

[0011] In addition, the aim is to enable the upcycling of advertisements by allowing users to retrieve deleted ad images from their archive as needed. means of solving the problem

[0013] To achieve the above objective, the present invention is a method performed in a computing device having one or more processors and a memory for storing one or more programs executed by said one or more processors, the method comprising: a step of checking whether an advertisement is displayed on a smartphone screen; a step of performing a sensing operation of said smartphone and transmitting it when said advertisement is displayed on the smartphone screen; a step of detecting said sensing operation and transmitting a captured image of said advertisement to an AI model; a step of analyzing data of the captured image of said advertisement in said AI model to determine whether there is an ad removal button; and a step of, when an ad removal button is determined in said step, capturing said advertisement and touching it to delete it.

[0014] The above sensing method is characterized by sensing the shaking motion (operation) of the smartphone through an accelerometer inside the smartphone, or sensing whether the smartphone is approaching an object through an internal proximity sensor.

[0015] The detection of the area corresponding to the ad removal button using the above AI model is characterized by using a YOLO or Faster R-CNN model.

[0016] The implementation of the above touch is characterized by simulating touch events using an Accessibility Service.

[0017] It is characterized by the ability to selectively remove an advertisement appearing on the screen of the smartphone through the sensing action of the user.

[0018] It is characterized by further including the step of recognizing an advertisement deleted from the screen of the smartphone and separately saving the deleted advertisement.

[0019] The present invention is characterized by pre-training the AI ​​model with the image of the advertisement and the shape of the ad removal button, thereby enabling selective removal of the advertisement through the detection of the ad removal button.

[0020] When the smartphone identifies and displays an advertisement, it is characterized by receiving the coordinates of an advertisement delete button displayed on the smartphone screen and capturing the location of the advertisement delete button.

[0021] The data analysis of the captured image of the above advertisement is characterized by including: a step of marking the location of an ad removal button based on a pre-learned input ad image; a step of sub_imging the ad image size to a certain size for learning; a core processing process of configuring a sub_label containing button location marking information in the above step; and a post-processing process of generating a mask frame to be used in the second step by up-sampling the down-size mask frame.

[0022] A computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises one or more instructions, and when the instructions are executed by a computing device having one or more processors, the computing device,

[0023] The computer program includes a step of checking whether an advertisement is displayed on a smartphone screen; a step of performing a sensing operation of the smartphone and transmitting it when the advertisement is displayed on the smartphone screen; a step of detecting the sensing operation and transmitting a captured image of the advertisement to an AI model; a step of analyzing the data of the captured image of the advertisement in the AI ​​model to determine whether there is an ad removal button; and a step of capturing the advertisement and touching it to delete it when an ad removal button is determined in the above step. Effects of the invention

[0025] Accordingly, the present invention has the effect of allowing advertisements to be deleted quickly and easily without touch action when using a smartphone, and has the effect of eliminating the risk of moving to an unnecessary advertiser page due to a user's accidental touch.

[0026] In addition, it can reduce unpleasant experiences caused by indiscriminate advertisements during smartphone use, thereby creating a pleasant smartphone environment to reduce user stress and allowing for the expectation of an upcycling effect for advertisements. Brief explanation of the drawing

[0028] FIG. 1 is a configuration diagram of a selective ad removal device based on motion recognition and object detection using a smartphone sensor according to an embodiment of the present invention. FIG. 2 is a flowchart of a method for selective ad removal based on motion recognition and object detection using a smartphone sensor according to an embodiment of the present invention. Figure 3 is a picture of a smartphone screen where the user can choose whether to run an optional ad removal service on the smartphone. Figure 4 is a screenshot showing the Faster R-CNN model analyzing an advertisement image through 'x' shape learning. Figure 5 is a captured image showing the coordinates of the identified ad delete button being transmitted and displayed. Figure 6 is a photograph showing a touch simulation using an accessibility service. Figure 7 is a screenshot showing the location of the ad removal button through data analysis. Fig. 8 is a captured image distinguishing between an area with an ad delete button and an area without one. FIG. 9 is a block diagram illustrating a computing environment including a computing device suitable for use in an exemplary embodiment of the present invention. Specific details for implementing the invention

[0029] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, this is merely illustrative and the present invention is not limited thereto.

[0030] In describing the embodiments of the present invention, detailed descriptions of known technologies related to the present invention are omitted if it is determined that such detailed descriptions may unnecessarily obscure the essence of the present invention. Furthermore, the terms described below are defined in consideration of their functions within the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe the embodiments of the present invention and should not be limiting in any way. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as "include" or "comprise" are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described.

[0032] FIG. 1 is a configuration diagram of a selective ad removal device based on motion recognition and object detection using a smartphone sensor according to an embodiment of the present invention, FIG. 2 is a flowchart of a selective ad removal method based on motion recognition and object detection using a smartphone sensor according to an embodiment of the present invention, FIG. 3 is a screen image of a smartphone where a user can select whether to execute a selective ad removal service on the smartphone, FIG. 4 is a captured image showing a Faster R-CNN model analyzing an ad image through 'x' shape learning, FIG. 5 is a captured image showing the coordinates of a determined ad removal button being transmitted and displayed, FIG. 6 is a picture showing a touch simulation using an accessibility service, FIG. 7 is a captured image showing the location of an ad removal button through data analysis, FIG. 8 is a captured image distinguishing between an area with an ad removal button and an area without one, and FIG. 9 is a block diagram for illustrating and explaining a computing environment including a computing device suitable for use in an exemplary embodiment of the present invention.

[0034] Hereinafter, a selective ad removal device based on motion recognition and object detection using a smartphone sensor according to an embodiment of the present invention will be described with reference to the attached drawings.

[0035] Referring to FIG. 1, the present invention is largely composed of a smartphone (100) that detects a user's movement through a sensor when an advertisement is displayed on the screen of the smartphone (100), and an AI (Artificial Intelligence) model (200) that receives a captured image of the advertisement through the smartphone (100) and finds and detects an area corresponding to an advertisement removal button.

[0036] As described above, the user's motion is detected by a sensor (not shown), and a captured image of the smartphone (100) is transmitted to an AI model (200). The coordinates on the image are returned as the output of the AI ​​model (200), and the advertisement is removed through a series of processes (motion - sensing - determination by AI model - capture - touch) in which the corresponding point is touched on the smartphone (100). In other words, the advertisement is removed by touching a specific desired point on the advertisement screen using only the user's actions or behaviors without physical contact, by utilizing a sensor. Therefore, the AI ​​model (200) is trained in advance to display a predetermined removal indicator, such as an 'x' or 'close' mark, through the original image and the label image.

[0037] In one embodiment, the AI ​​model (200) may be installed on a smartphone (100) or on a server computing device that is remotely connected to the smartphone (100).

[0038] Here, the sensor may include one or more of a proximity sensor that recognizes when a finger, etc. is placed near the upper sensor of the smartphone, and an accelerometer that recognizes when the smartphone (100) is shaken and detects the shaking speed.

[0040] Hereinafter, with reference to the drawings, a method for selective ad removal based on motion recognition and object detection using a smartphone sensor according to an embodiment of the present invention will be described.

[0041] Referring to FIG. 2, it is checked whether an advertisement is displayed on the smartphone screen (S100).

[0042] The above step is to check whether an advertisement appears in the middle of a user's search for a website while using an application (not shown) of a smartphone (100), and in this case, the user checks whether an advertisement is displayed and has appeared on the screen of the smartphone (100).

[0043] When an advertisement is displayed on the screen of the smartphone (100), if the user wants to remove the advertisement, the smartphone (100) performs a sensing operation to transmit it (S200).

[0044] In S200, when an advertisement is displayed on the screen in the above S100 and the user confirms it, the user decides whether to remove the advertisement or to keep it as is so that they can view it. Therefore, the user can selectively remove the advertisement appearing on the screen of the smartphone (100) by making a decision on an arbitrary sensing operation themselves.

[0045] If the user wants to remove the advertisement, a sensing operation is performed as a signal for this, and the sensing method is selected by the user from among an internal proximity sensor (not shown) and an accelerometer (not shown) according to the user's preference.

[0046] The above proximity sensor detects whether an object is approaching the smartphone (100) by touching (contacting) the sensor located on the top of the smartphone using a user's body part, such as a finger. This is similar to how the screen turns off when the proximity sensor recognizes it simply by bringing the smartphone to one's ear, and the proximity sensor can recognize the approach itself through the distance from the object.

[0047] The above acceleration sensor automatically recognizes the shaking motion (movement) of the smartphone (100) when the user shakes the smartphone (100) with their hand.

[0048] Accordingly, the above acceleration sensor and proximity sensor are embedded in the smartphone (100), and the smartphone (100) recognizes whichever sensor the user selects from the two sensors. A detailed description of the operation and configuration of the above acceleration sensor and proximity sensor is omitted as it is a widely known technology.

[0049] Also, referring to FIG. 3, the user can voluntarily execute the relevant ad removal service of the smartphone (100), allowing for free choice. Therefore, the user can execute the ad removal on the smartphone (100) screen only when they want to.

[0050] Therefore, it is preferable that an application (not shown) for executing an ad removal method according to an embodiment of the present invention be installed on the smartphone (100).

[0051] Next, the smartphone (100) detects the user's sensing action and transmits the captured image of the advertisement to the AI ​​model (S300).

[0052] When a user intends to remove an advertisement on a smartphone (100) and performs a sensing action, an ad capture screen, i.e., an ad capture image, of the smartphone (100) is transmitted to an AI model (200) that has been trained to have ad removal buttons such as 'X' and 'Close' on the screen of the smartphone (100).

[0053] The AI ​​model (200) analyzes the data of the captured image of the advertisement to determine whether there is an ad removal button (S400).

[0054] In the above S400, the AI ​​model (200) detects an object through deep learning, captures the presence or absence of an ad delete button determined from the transmitted ad capture screen, and checks its coordinates.

[0055] Therefore, the AI ​​model (200) is trained in advance with the image of the advertisement and the shape of the ad removal button, so that the advertisement can be removed through the detection of the ad removal button.

[0056] Detection of the area corresponding to the ad removal button using an AI model (200) is done by using artificial intelligence and a YOLO or Faster R-CNN model.

[0057] Below, we will explain the process of the above Faster R-CNN model detecting an object (the area of ​​the ad remove button).

[0058] (1) CNN pass

[0059] The input image is first passed through a ConvNet (CNN) to generate a feature map. The feature map is the result of compressing the important spatial features of the image.

[0060] (2) Region Proposal Network (RPN)

[0061] Using a Region Proposal Network, object candidate regions are extracted in a learnable manner via the RPN, instead of Selective Search.

[0062] RPN proposes candidate regions where objects are likely to exist directly from the feature maps passed through the aforementioned ConvNet. This process is performed in a structure integrated with CNN, and since it is trained on a GPU, computational speed and efficiency are significantly improved.

[0063] RPN uses small networks at various locations in the image to propose regions (proposals) that are likely to contain objects, and the proposed candidate regions are used in subsequent steps.

[0064] (3) RoI Pooling

[0065] The candidate regions proposed in the RPN are converted into fixed-size vectors through the RoI Pooling Layer.

[0066] (4) Classifier

[0067] Fixed-size vectors generated through RoI Pooling perform object classification and bounding box regression using a Fully Connected Layer and a Softmax Classifier. In other words, they predict the object class and fine-tune the position of the bounding box.

[0068] Referring to Fig. 4, the 'x' that is the ad removal button is found by using the 'x' shape learned by the AI ​​model (300) in advance through data analysis using this Faster R-CNN model.

[0069] Next, if an ad removal button is identified through the AI ​​model (200) in the above S400, the ad is captured and deleted by touching it (S500).

[0070] In the above step, the AI ​​model (200) performs a touch to delete an advertisement, but the touch is implemented using code rather than a physical touch. That is, it means that the touch is automatically implemented by the smartphone (100) rather than the user performing a physical touch.

[0071] Referring to FIG. 5, the screen of the user's smartphone (100) is transmitted to the AI ​​model (200) of the server (not shown) to check the 'x' button or 'close' button of the advertisement and transmit the coordinates.

[0072] In other words, when an advertisement is identified and displayed through the smartphone (100), the AI ​​model (200) receives the coordinates of the ad removal button displayed on the screen of the smartphone (100) and captures the location of the ad removal button.

[0073] The ad removal method involves an AI model (200) implementing a touch to remove the ad, wherein the implementation of the ad touch utilizes an accessibility service to implement a touch event, thereby programmatically simulating the touch code.

[0074] In this regard, Fig. 6 is a screen showing a touch simulation using an accessibility service.

[0075] The above diagram simulates a touch action using code that artificially triggers the touch action in an Android application by utilizing accessibility services. By leveraging the accessibility services provided by Android, it is possible to simulate touch events without physical contact, regardless of the execution environment.

[0077] Hereinafter, a procedure for identifying the location of an ad removal button through data analysis based on prior learning of a captured image of an ad input on the screen of a smartphone (100) will be explained by attaching a drawing.

[0078] Referring to FIG. 7, the location of the ad removal button is identified and displayed based on the content learned in advance for the ad image input into the smartphone (100) (Step 1). The content of the advance learning is to learn the 'x' mark or 'close' mark through the original image and the label image.

[0079] The second step is to crop the image to a specific size for learning the ad image size.

[0080] In the above second step, a core processing process is performed to configure a sub_label containing button position display information (third step).

[0081] It includes a post-processing process (step 4) for generating a new mask frame to be used in step 2 by up-sampling a downsized mask frame.

[0082] Therefore, through the process described above, the data is compressed and saved as an npy file (a binary file that stores multidimensional array data generated by the numpy library), and when a signal is generated through a user's sensing action, the ad is removed by touching an ad delete button ('x' or 'close', etc.). Fig. 8 is a photograph showing the distinction between an area with the ad delete button and an area without it. As shown in the figure, the area with the button is recognized, and a touch action is executed to delete the ad.

[0084] Furthermore, the selective ad removal method based on motion recognition and object detection using a smartphone sensor according to an embodiment of the present invention may further include the step of recognizing deleted ads through an AI model (200) on the screen of a smartphone (100) and separately storing the deleted ads.

[0085] When a captured image is transmitted to the server, it remains in the database; therefore, if a user wishes to view an advertisement again or check the history of previously displayed advertisements, they can subsequently access it from the stored location. Thus, it would be desirable for the aforementioned server to store a relevant database.

[0086] Therefore, unlike other applications where ads are completely blocked, captured images are stored in a designated repository as needed, allowing ad history to be checked at any time. A detailed explanation of the technology for storing ads in the aforementioned database and verifying the stored ads will be omitted, as it is a widely known technology.

[0088] Accordingly, the selective ad removal method based on motion recognition and object detection using a smartphone sensor according to one embodiment of the present invention can be utilized in all situations where the smartphone screen (100) cannot be directly touched, as the AI ​​model (200) is trained in advance to have an image to be touched, and the desired point can be touched by action alone without separate physical contact with the smartphone screen (100).

[0089] In addition, similar to the previously mentioned method for removing smartphone ads, it can be utilized when convenient operation is desired in situations requiring touches at precise locations or repetitive touches at different locations.

[0090] In addition, it can be applied to various environments, such as when turning pages in recipes, webtoons, web novels, or e-books, operating music or video players, or when operating a smartphone in specific situations like answering a call while driving or cooking.

[0091] It is included in the accessibility features within the smartphone's default settings, and its functionality can be expanded, such as by utilizing more types of sensors or changing the touch image for each app.

[0093] FIG. 9 is a block diagram illustrating a computing environment (10) including a computing device suitable for use in exemplary embodiments. In the illustrated embodiments, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those described below.

[0094] The illustrated computing environment (10) includes a computing device (12). In one embodiment, the computing device (12) may be a server (not shown). Additionally, the computing device (12) may be a terminal (not shown).

[0095] The computing device (12) includes at least one processor (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) can enable the computing device (12) to operate according to the exemplary embodiment mentioned above.

[0096] For example, the processor (14) may execute one or more programs stored in a computer-readable storage medium (16). The one or more programs may include one or more computer-executable instructions, and the computer-executable instructions may be configured to cause the computing device (12) to perform operations according to an exemplary embodiment when executed by the processor (14).

[0097] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by a processor (14). In one embodiment, the computer-readable storage medium (16) may be memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other forms of storage media that are accessed by a computing device (12) and capable of storing desired information, or a suitable combination thereof.

[0098] The communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and the computer-readable storage medium (16).

[0099] The computing device (12) may also include one or more input / output interfaces (22) and one or more network communication interfaces (26) that provide interfaces for one or more input / output devices (24). The input / output interfaces (22) and the network communication interfaces (26) are connected to a communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) through the input / output interfaces (22). An exemplary input / output device (24) may include input devices such as a pointing device (mouse or trackpad, etc.), a keyboard, a touch input device (touchpad or touchscreen, etc.), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or output devices such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (24) may be included inside the computing device (12) as a component constituting the computing device (12), or it may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0101] Although representative embodiments of the present invention have been described in detail above, those skilled in the art will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof. Explanation of the symbols

[0105] 10 : Computing Environment 12 : Computing Device 14: Processor 16: Computer-readable storage medium 18 : Communication bus 22 : Input / Output interface 24: Input / Output Device 26: Network Communication Interface 100 : Smartphone 200 : AI Model

Claims

Claim 1 A method performed on a computing device having one or more processors and a memory storing one or more programs executed by said one or more processors, the method comprising: a step of checking whether an advertisement is displayed on a smartphone screen; a step of receiving input of a sensing operation of said smartphone from a user when said advertisement is displayed on the smartphone screen; a step of, when said sensing operation is detected, capturing the advertisement displayed on said screen and transmitting the captured image of said advertisement to an AI model; and a step of analyzing data of the captured image of said advertisement in said AI model to determine whether there is an ad removal button. A selective ad removal method comprising the step of detecting the ad removal button in the above step, capturing the ad and deleting it by touching it, wherein the sensing operation is generated when the user wishes to remove the ad displayed on the screen, and is based on a proximity sensor that recognizes the proximity of the user's finger to the smartphone or an accelerometer that detects shaking of the smartphone, and the user can selectively remove the ad displayed on the screen through the sensing operation, and the step of deleting by touching it is achieved by the AI ​​model utilizing the accessibility service of the smartphone to implement a touch event regardless of the execution environment, thereby programmatically generating a touch code, and further comprising the step of storing a capture image corresponding to the deleted ad; and the step of displaying the previously deleted ad by outputting the previously stored capture image according to the user's request. Claim 2 A selective ad removal method according to claim 1, wherein the sensing method senses shaking motion (operation) of the smartphone through an accelerometer sensor embedded in the smartphone, or senses whether an object approaches the smartphone through an embedded proximity sensor. Claim 3 A selective ad removal method according to claim 1, characterized in that the detection of the area corresponding to the ad removal button using the AI ​​model utilizes a YOLO or Faster R-CNN model. Claim 4 delete Claim 5 delete Claim 6 A selective ad removal method according to claim 1, further comprising the step of recognizing an ad deleted from the screen of the smartphone, separately saving the deleted ad for later verification. Claim 7 A selective ad removal method according to claim 1, characterized in that the AI ​​model is trained in advance with the image of the ad and the shape of the ad removal button, thereby enabling selective removal of the ad through the detection of the ad removal button. Claim 8 delete Claim 9 A selective ad removal method based on motion recognition and object detection using a smartphone sensor, characterized in that, in claim 1, the step of determining the presence or absence of an ad removal button by analyzing data based on a pre-learned ad image input to the smartphone comprises: a sub_img step of cutting the ad image size into a certain size for learning; and a core processing step of configuring a sub_label including button position display information in the above step. Claim 10 A computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises one or more instructions, and when the instructions are executed by a computing device having one or more processors, the computing device performs the steps of: checking whether an advertisement is displayed on a smartphone screen; receiving input of a sensing operation of the smartphone from a user when the advertisement is displayed on the smartphone screen; when the sensing operation is detected, capturing the advertisement displayed on the screen and transmitting the captured image of the advertisement to an AI model; and analyzing the data of the captured image of the advertisement in the AI ​​model to determine whether there is an ad removal button. A computer program stored on a non-transient computer-readable storage medium, wherein, when the ad removal button is determined in the above step, the step of capturing the ad and deleting it by touching it is performed, the sensing operation is generated when the user wishes to remove the ad displayed on the screen, and is based on a proximity sensor that recognizes the proximity of the user's finger to the smartphone or an accelerometer that detects shaking of the smartphone, and the user can selectively remove the ad displayed on the screen through the sensing operation, the step of deleting by touching is performed by the AI ​​model utilizing the accessibility service of the smartphone to implement a touch event regardless of the execution environment, thereby programmatically generating a touch code, and the commands further perform the step of the computing device storing a capture image corresponding to the deleted ad; and the step of outputting the previously stored capture image to display the previously deleted ad upon the user's request.

Images