Processing system and processing method for user interface

Abstract

The present invention discloses a processing system and a processing method for a user interface. The processing method includes a learning phase and an application phase. After a specific model is established in the learning phase, a user interface representing a specific meaning such as closing or rejecting can be automatically found in the application phase for emphasis processing.

Description

Technical Field

[0001] The present invention relates to a processing system and a processing method, and in particular to a processing system and a processing method for a user interface.

[0002] Prior Art

[0003] With the popularization of personal computers and the booming development of the Internet, modern people have become accustomed to using personal computers to handle various affairs and browse various information on the Internet through browsers in personal computers. Based on commercial considerations, most commercial websites currently provide web pages with many advertisements for various products or services related to the content of the web pages or other businesses. Whenever users link to these web pages or at specific times, advertisements may pop up and appear in front of users, thereby achieving advertising and marketing effects.

[0004] However, whether it is a mobile phone screen or a browser screen, it is now easy to be covered by ads. In some cases, the cover ads even make the entire screen almost empty of the content that the user wants to watch, leaving only ads and pop-up windows.

[0005] However, in the above situation, whether using a mouse, touch screen, or even a remote control, there is a high probability of pressing the wrong button, and then being directed to an unwanted screen, wasting the user's time and energy. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a processing system and a processing method for a user interface in view of the deficiencies in the prior art, which can automatically find a close option and emphasize it.

[0007] In order to solve the above technical problems, one of the technical solutions adopted by the present invention is to provide a processing method for a user interface, comprising: configuring a processor to enter a learning phase, including: configuring the processor to capture a first screen of a user interface; configuring the processor to detect whether there is a user input from an input module, wherein the user input corresponds to an input position on the user interface; in response to detecting the user input, configuring the processor to capture a second screen of the user interface; configuring the processor to compare the difference between the first screen and the second screen and store them in a memory; configuring the processor to execute a first recognition program to detect a closed outer frame from the difference according to the input position as a button key object; configuring the processor to execute a second recognition program to identify a feature object from the key object; and configuring the processor to associate the key object with the feature object and store it in the memory; and configuring the processor to enter an application stage, including: configuring the processor to capture a current screen of the user interface, and using the first recognition program to detect whether the key object exists in the current screen; in response to detecting that the key object exists in the current screen, configuring the processor to execute the second recognition program to determine whether the key object exists; and in response to the key object existing in the feature object, configuring the processor to perform an emphasis processing on the key object in the current screen of the user interface.

[0008] In order to solve the above technical problems, another technical solution adopted by the present invention is to provide a processing system for a user interface, including a user interface, an input module, a memory and a processor. The processor is configured to enter a learning phase and an application phase. In the learning phase, the processor is configured to: capture a first screen of the user interface; detect whether there is a user input from the input module, wherein the user input corresponds to an input position on the user interface; in response to detecting the user input, capture a second screen of the user interface; compare the difference between the first screen and the second screen, and store them in a memory; execute a first recognition program to detect a closed outer frame from the difference according to the input position as a key object; execute a second recognition program to identify a feature object from the key object; and associate the key object with the feature object and store it in the memory. Wherein, in the application stage, the processor is configured to: capture a current screen of the user interface, and use the first recognition program to detect whether the button object exists in the current screen; in response to detecting that the button object exists in the current screen, execute the second recognition program to determine whether the button object has the feature object; and in response to the button object having the feature object, perform an emphasis processing on the button object in the current screen of the user interface.

[0009] One of the beneficial effects of the present invention is that the processing system and method for user interface provided by the present invention, after establishing a specific model in the learning stage, can automatically find the user interface representing specific meanings such as closing and rejecting for emphasis processing, so as to increase the sensing range or dynamically amplify, colorize, flash, etc., and then prompt the user to close unnecessary advertisements or windows here, so as to reduce the chance of accidental touch by the user and reduce the waste of the user's time and energy.

[0010] In addition, for different types of buttons, such as button objects with text objects or graphic objects, the processing system and processing method for user interface provided by the present invention can perform targeted learning on the characteristics of the above objects, and can even perform learning on non-button type objects, thereby enhancing the user's freedom in system learning.

[0011] To further understand the features and technical contents of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are only for reference and description and are not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 FIG. 5 is a functional block diagram of a processing system for a user interface according to an embodiment of the present invention.

[0013] Figure 2 The first flow chart of the processing method for the user interface according to the embodiment of the present invention is shown.

[0014] Figure 3 FIG. 2 is a second flow chart of a processing method for a user interface according to an embodiment of the present invention.

[0015] Figure 4A is a schematic diagram of a first screen of a user interface according to an embodiment of the present invention.

[0016] Figure 4B is a schematic diagram of a second screen of a user interface according to an embodiment of the present invention.

[0017] Figure 5 FIG. 4 is a flowchart of a first identification procedure according to an embodiment of the present invention.

[0018] Figure 6 FIG. 4 is a first flow chart of a second identification procedure according to an embodiment of the present invention.

[0019] Figure 7 FIG. 4 is a second flow chart of a second identification procedure according to an embodiment of the present invention.

[0020] Figure 8 FIG. 4 is another flow chart of the application phase according to an embodiment of the present invention.

[0021] Fig. 9 FIG. 4 is another flow chart of the application phase according to an embodiment of the present invention.

[0022] Fig.10 Detailed description of the drawings are given below, showing multiple examples of emphasis processing according to an embodiment of the present invention.

[0023]

Explanation of symbols

[0024] 1: Processing system

[0025] 10: User Interface

[0026] 12: Input module

[0027] 14: Memory

[0028] 16: Processor DETAILED DESCRIPTION

[0029] The following is an explanation of the implementation methods of the "processing system and processing method for user interface" disclosed in the present invention through specific embodiments. Those skilled in the art can understand the advantages and effects of the present invention from the contents disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the concept of the present invention. In addition, the drawings of the present invention are only simple schematic illustrations and are not depicted according to actual dimensions. It is stated in advance. The following embodiments will further explain the relevant technical contents of the present invention in detail, but the disclosed contents are not intended to limit the scope of protection of the present invention. In addition, the term "or" used in this article may include any one or more combinations of the associated listed items depending on the actual situation.

[0030] Figure 1 FIG. 1 is a functional block diagram of a processing system for a user interface according to an embodiment of the present invention. Figure 1 As shown, an embodiment of the present invention provides a processing system 1 for a user interface, including a user interface 10 , an input module 12 , a memory 14 and a processor 16 .

[0031] The processing system 1 is, for example, a desktop computer, a notebook computer, a smart phone, a tablet computer, a game console, an e-book, or a smart TV, etc., but the present invention is not limited thereto.

[0032] The user interface 10 may be, for example, a liquid crystal display (LCD), a light-emitting diode (LED), a field emission display (FED), an organic light-emitting diode (OLED), or other types of displays, but the present invention is not limited thereto. In other embodiments, the user interface 10 may be, for example, a browser executed by the processor 16 in an operating system.

[0033] The input module 12 is used for receiving user operations issued by the user, such as a mouse, keyboard, touch device or remote controller.

[0034] The memory 14 is used to store data such as images, program codes, software modules, etc. It can be, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk or other similar devices, integrated circuits and combinations thereof.

[0035] The processor 16 is, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose microprocessor, digital signal processor (DSP), programmable controller, application specific integrated circuits (ASIC), programmable logic device (PLD), graphics processing unit (GPU) or other similar devices or a combination of these devices. The processor 16 can execute program codes, software modules, instructions, etc. recorded in the memory 14 to implement the automatic advertisement closing method of the embodiment of the present invention.

[0036] Figure 2 and Figure 3 The first and second flowcharts of the processing method for the user interface according to the embodiment of the present invention are shown in FIG. Figure 2 and Figure 3 The processing method of this embodiment is applicable to the processing system 1 in the above-mentioned embodiment. The following is a description of the detailed steps of the processing method for the user interface of this embodiment in conjunction with the various components in the processing system 1.

[0037] The treatment method includes a learning phase and an application phase, such as Figure 2 As shown, the learning phase may include configuring the processor 16 to perform the following steps:

[0038] Step S20: Capture the first screen of the user interface 10. For example, please refer to Figure 4A , which is a schematic diagram of the first screen of the user interface according to an embodiment of the present invention. Figure 4A A mobile device browser screen is shown with a banner ad area instructing to join the membership with a close option.

[0039] Step S21: Detect whether there is user input inp from the input module 12. Figure 4A As shown, the user input inp corresponds to the input position on the user interface 10. For example, the user input inp can be obtained by scanning keystrokes, including touch input or remote control input, and recording the key code value (Key code) corresponding to the user input and the corresponding input position, for example, the coordinates on the user interface 10.

[0040] In response to detecting the user input inp in step S21, the process proceeds to step S22: capturing the second screen of the user interface 10. Figure 4B , which is a schematic diagram of a second screen of a user interface according to an embodiment of the present invention. Figure 4B The browser screen of a mobile device is also displayed. After the user clicks Figure 4A After clicking the Close option, the banner area disappears.

[0041] In response to the user input inp not being detected in step S21 , step S21 is repeatedly executed until the user input inp is detected, and then the process proceeds to step S22 .

[0042] In detail, steps S20 to S22 are mainly to record the changes in the screen after the user input. For example, when an advertisement block and an accompanying close option appear on a web page, when the user operates the close option through the input module 12, the user input inp and the changes in the screen are recorded. Optionally, the user interface 10 can be used to ask the user whether to automatically record this association, or the user can automatically agree to record this association.

[0043] Step S23: Compare the differences between the first frame and the second frame and store them in the memory 14. For example, the disappeared advertisement banner area (including the closing option portion) can be regarded as a difference and stored.

[0044] Step S24: Execute a first recognition procedure to detect a closed frame from the difference according to the input position as a key object. The first recognition procedure may be, for example, an image processing method, which will be described below by way of example.

[0045] See also Figure 5 , which is a flow chart of a first identification procedure according to an embodiment of the present invention. Figure 5 As shown, in some embodiments, the first identification procedure may include:

[0046] Step S50: According to the input position, a blob detection procedure is executed to search for a closed frame outward from the input position. In the field of vision, the main concept of blob detection is to detect an area with a gray value greater than or less than the gray value of surrounding pixels from an image, but the present invention is not limited to this image processing method.

[0047] Step S51: Taking the closed frame as a reference, add a set margin to obtain the key frame. Figure 4A For the close option, this step regards the circle around the option as a closed frame, and extends outward by a user-set or preset distance as a set margin to generate a button frame.

[0048] Step S52: Take the captured image corresponding to the key frame as the key object. Figure 4A The first picture is captured, and the captured part of the image is used as the button object.

[0049] Please refer to Figure 2 The processing method enters step S25: executing a second recognition procedure to recognize a feature object from the key object. Specifically, this step may use different recognition methods according to the content of the key object. For example, if the key object contains text, text recognition may be used, and if the key object contains an image, image recognition may be used.

[0050] For more details, see Figure 6 , which is a first flow chart of a second identification procedure according to an embodiment of the present invention. Figure 6 As shown, the second identification procedure includes:

[0051] Step S60: Perform binary pre-processing on the key object obtained by the first recognition procedure. Specifically, considering that the text in the key object may be highlighted, framed, or represented by other colors, it is necessary to perform binary pre-processing on the key object before recognition. However, in general, the text in the key object is usually intended to be easy for users to read, and no robot blocking mechanism is deliberately added. Therefore, this step does not require a more difficult image pre-processing method, but the present invention is not limited to this.

[0052] Step S61: Execute a text recognition program to recognize text objects from the pre-processed key objects as feature objects. In this step, the text recognition program may be, for example, an optical character recognition (OCR) method. In addition to recognizing individual characters, the text recognition program may further include a single word correction mechanism or a short text correction mechanism.

[0053] In addition, see Figure 7 , which is a second flow chart of the second identification procedure according to an embodiment of the present invention. Figure 7 As shown, the second identification procedure includes:

[0054] Step S70: Execute a graphic recognition program to recognize at least one graphic object from the button object as a feature object.

[0055] In some embodiments, the graphic recognition process may involve identifying image features through a machine learning model. For example, the graphic recognition process may include step S71: inputting the key object obtained by the first recognition process into the machine learning model to train the machine learning model to classify the key object including the graphic object into a key graphic category.

[0056] For example, a machine learning model (e.g., a YOLO V3 model) may be used to identify a graphic object in a button object. In other embodiments, the machine learning model may be a CNN model in deep learning, a model using an NMS algorithm, or other machine learning models that may be used for object detection, but the present invention is not limited thereto.

[0057] In more detail, the machine learning model for identifying graphic objects can be trained by many button objects including graphic objects. During the training process of the machine learning model, a large number of button object images can be collected and input into the machine learning model to gradually train a set of rules (i.e., parameters of the machine learning model) that can be used to predict graphic objects, and finally a machine learning model that can be used to detect graphic objects is established.

[0058] Please refer to Figure 2 , the learning phase enters step S26: associating the button object with the feature object and storing it in the memory 14 for use in the subsequent application phase.

[0059] In addition, in response to not detecting a closed frame from the difference in step S23, the learning phase enters step S27: configuring the processor to execute a third recognition procedure to recognize a feature object from the difference.

[0060] In an embodiment of the present invention, the third recognition procedure includes executing a graphic recognition procedure (e.g., the aforementioned YOLOV3 model). Generally speaking, the feature object includes a plurality of graphic objects, and the graphic recognition procedure may, for example, execute step S28: input the difference into the machine learning model to train the machine learning model to extract the graphic objects at the difference as object strings and store them in the memory.

[0061] Specifically, when the closed frame cannot be detected, the difference can be directly taken out as a learning object. For example, the entire screenshot can be reduced to a fixed size, such as 400x400, and input into the machine learning model to train the screenshot, so that when the closed frame cannot be detected in the subsequent application stage, the image can be directly compared.

[0062] On the other hand, if the amount of computation and storage space are taken into consideration, when using a machine learning model, feature detection can be further performed on the differences, and the detected feature objects, such as houses, cars, and people (in specific embodiments, buttons may also be included) are recorded as object strings. For example, each object is stored as a video object (VideoObject) in the MPEG-4 standard to form an object string.

[0063] After the above-mentioned learning stage, the processing method can enter the application stage. It should be noted that the above-mentioned learning stage refers to online learning, and mainly refers to the user using his own device or platform to learn and establish a database by himself. In contrast, in other embodiments, offline learning can also be adopted, which means that the user can directly use the learned database in the cloud through the network without re-learning, and the present invention is not limited to this.

[0064] On the other hand, see Figure 3 The application phase includes configuring the processor 16 to perform the following steps:

[0065] Step S30: Capture the current screen of the user interface.

[0066] Step S31: Detect whether the key object exists in the current picture through the first recognition program. Similarly, it can be determined whether the key object exists through steps S50 to S52, which will not be repeated here. It should be noted that this step can first determine whether there is a key frame in the current picture, and compare it with the key frame learned in the learning stage to determine whether the key object exists.

[0067] In response to detecting that the key object exists in the current picture, the process proceeds to step S32: executing a second recognition procedure to determine whether the key object has a characteristic object. As mentioned above, different recognition methods can be used according to the content of the key object. In other words, the same principle is also used when applying.

[0068] Therefore, further reference can be made to Figure 8 , which is another flow chart of the application phase according to an embodiment of the present invention. In the application phase, the step of determining whether a key object has a feature object further includes:

[0069] Step S80: performing binarization pre-processing on the button object.

[0070] Step S81: Execute a text recognition program to recognize another text object from the pre-processed key object, and calculate the similarity between the text object and the other text object.

[0071] For example, the probability percentage (e.g., confidence score) of the text object identified in step S81 being similar to the text object in memory 14 can be calculated, or the error distance between the two can be calculated. The higher the confidence score or the lower the error distance, the higher the similarity. In addition, the comparison method can be word-by-word comparison or string-by-string comparison, which will not be described in detail in the present invention.

[0072] Step S82: Determine whether the similarity is greater than a predetermined similarity. The predetermined similarity can be set by the user, and when it is greater than a certain level, it is determined that the text object identified in step S81 is the same as the text object in the memory 14.

[0073] In response to the similarity being greater than the predetermined similarity, the process proceeds to step S83: determining whether a feature object exists in the button object.

[0074] On the other hand, further reference can be made to Fig. 9 , which is another flow chart of the application phase according to an embodiment of the present invention. In this embodiment, the step of determining whether a key object has a feature object in the application phase further includes:

[0075] Step S90: Execute a graphic recognition program to identify another graphic object from the button object, and calculate the similarity between at least one graphic object and another graphic object. In this step, the graphic recognition program can be a machine learning model trained in the aforementioned learning phase, and the button object detected in step S31 is input into the trained machine learning model to determine whether the button object will be classified as the button category established in the previous step.

[0076] Step S91: Determine whether the similarity is greater than a predetermined similarity.

[0077] In detail, the area of ​​the graphic object may be further considered and weighted when calculating the similarity. For example, the coordinates of the key object detected in step S31 may be taken into consideration, for example, by determining the upper left coordinate and the lower right coordinate of the graphic object in the memory to estimate the area of ​​the learned graphic object, and when the graphic object is identified in step 90, the upper left coordinate and the lower right coordinate are determined to estimate the area of ​​the graphic object in the current screen, and the area difference is considered and weighted when calculating the similarity, thereby calculating the similarity.

[0078] In response to the similarity being greater than the predetermined similarity, the process proceeds to step S92: determining whether a feature object exists in the button object.

[0079] In response to the presence of a feature object in the button object, the process proceeds to step S33: performing emphasis processing on the button object in the current screen of the user interface.

[0080] For reference Fig.10 , which are multiple examples of emphasis processing according to the embodiment of the present invention. Fig.10 As shown, the emphasis processing may include enlarging, flashing, coloring or marking the detected feature object in other eye-catching ways, and may also include increasing the sensing range of the closing option, such as Fig.10 In a specific embodiment, it is also possible to set the user to automatically click on the close option. For example, the user input inp recorded in the learning phase (for example, the key code value (Keycode) corresponding to the input and the corresponding input position) can be automatically applied to the close option.

[0081] Please refer to Figure 3 The application stage may further include step S34: using a third recognition program to detect whether the object string exists in the current frame. The third recognition program in this step may include executing the graphic recognition program trained in step 28 (e.g., the aforementioned YOLO V3 model) to determine whether the object string recorded in the previous learning stage exists in the current frame.

[0082] In response to the object string existing in the current frame, the process proceeds to step S35 : performing emphasis processing on the object string in the current frame of the user interface.

[0083] [Beneficial Effects of Embodiments]

[0084] One of the beneficial effects of the present invention is that the processing system and method for user interface provided by the present invention, after establishing a specific model in the learning stage, can automatically find the user interface representing specific meanings such as closing, rejecting, etc. in the application stage for emphasis processing, so as to increase the sensing range or dynamically magnify, color, flash, etc., and then prompt the user to close unnecessary advertisements or windows here, so as to reduce the chance of accidental touch by the user and reduce the waste of the user's time and energy.

[0085] In addition, for different types of buttons, such as button objects with text objects or graphic objects, the processing system and processing method for user interface provided by the present invention can perform targeted learning on the characteristics of the above objects, and can even perform learning on non-button type objects, thereby enhancing the user's freedom in system learning.

[0086] The contents disclosed above are only preferred feasible embodiments of the present invention, and are not intended to limit the scope of the present invention. Therefore, all equivalent technical changes made using the contents of the present invention's specification and drawings are included in the scope of the present invention.

Claims

1. A processing method for a user interface, characterized in that: include: Configuring a processor to enter a learning phase includes: configuring the processor to capture a first screen of a user interface; configuring the processor to detect whether there is a user input from an input module, wherein the user input corresponds to an input position on the user interface; In response to detecting the user input, configuring the processor to capture a second screen of the user interface; The processor is configured to compare the differences between the first image and the second image and store the differences in a memory; The processor is configured to execute a first recognition procedure to detect a closed outer frame from the difference as a key object according to the input position; configuring the processor to execute a second recognition program to recognize a feature object from the key object; and configuring the processor to associate the button object with the feature object and store it in the memory; and Configuring the processor to enter an application phase includes: The processor is configured to capture a current screen of the user interface, and detect whether the button object exists in the current screen by using the first recognition program; In response to detecting that the button object exists in the current frame, configuring the processor to execute the second recognition program to determine whether the button object has the feature object; and In response to the button object having the feature object, the processor is configured to perform an emphasis process on the button object in the current frame of the user interface.

2. The processing method according to claim 1, characterized in that: The first identification procedure includes: According to the input position, a spot detection procedure is executed to search for the closed outer frame outward from the input position as the center; Taking the closed outer frame as a reference, adding a set margin to obtain a key outer frame; and The captured image corresponding to the button frame is used as the button object.

3. The processing method according to claim 1, characterized in that: The second identification procedure includes: Perform a binarization pre-processing on the key object obtained by the first recognition procedure; and A text recognition program is executed to recognize a text object from the pre-processed key object as the feature object.

4. The processing method according to claim 3, characterized in that: In the application stage, the step of determining whether the key object has the feature object further includes: Execute the binarization pre-processing on the button object; Executing the text recognition program to recognize another text object from the pre-processed key object, and calculating the similarity between the text object and the other text object, and determining whether the similarity is greater than a predetermined similarity; and In response to the similarity being greater than the predetermined similarity, it is determined that the key object has the feature object.

5. The processing method according to claim 1, characterized in that: The second identification procedure includes: A graphic recognition program is executed to recognize at least one graphic object from the button object as the feature object.

6. The processing method according to claim 5, characterized in that: In the application stage, the step of determining whether the key object has the feature object further includes: Executing the graphic recognition program to recognize another graphic object from the button object, and calculating the similarity between the at least one graphic object and the another graphic object, and determining whether the similarity is greater than a predetermined similarity; and In response to the similarity being greater than the predetermined similarity, it is determined that the key object has the feature object.

7. The processing method according to claim 5, characterized in that: The graphic recognition program includes inputting the button object obtained by the first recognition program into a machine learning model to train the machine learning model to classify the button object including the graphic object into a button graphic category.

8. The processing method according to claim 7, characterized in that: In the application stage, the step of determining whether the key object has the feature object also includes: Inputting the button object into the trained machine learning model; Identifying another graphic object from the button object by using the trained machine learning model, calculating a similarity between the at least one graphic object and the other graphic object, and determining whether the similarity is greater than a predetermined similarity; and In response to the similarity being greater than the predetermined similarity, it is determined that the key object has the feature object.

9. The processing method according to claim 1, characterized in that: Also includes: In response to not detecting the closed frame from the difference, the processor is configured to execute a third recognition procedure to recognize the feature object from the difference.

10. A processing system for a user interface, characterized in that include: a user interface; an input module; 1. Memory; as well as A processor is configured to enter a learning phase and an application phase, Wherein, in the learning phase, the processor is configured to: Capturing a first screen of the user interface; detecting whether there is a user input from the input module, wherein the user input corresponds to an input position on the user interface; In response to detecting the user input, capturing a second screen of the user interface; Comparing the differences between the first frame and the second frame, and storing the differences in a memory; executing a first recognition procedure to detect a closed frame from the difference as a key object according to the input position; executing a second recognition procedure to recognize a feature object from the key object; and Associating the button object with the feature object and storing it in the memory, Wherein, in the application stage, the processor is configured to: Capturing a current screen of the user interface, and detecting whether the button object exists in the current screen by the first recognition procedure; In response to detecting that the button object exists in the current frame, executing the second recognition procedure to determine whether the button object has the feature object; and In response to the button object having the feature object, an emphasis process is performed on the button object in the current screen of the user interface.

Images