User Interface Guidance and Control System Based on Procedure Semantic Recognition

Abstract

A procedural semantics-based user interface guidance and control system is provided. The system includes: a data storage unit storing data and a program for performing a procedure; and a control unit that executes the program to manage, guide, or support a user interface and the entire procedure displayed on a display.

Description

Technology Field

[0001] The present invention relates to an artificial intelligence assistant system. More specifically, it relates to a procedure meaning recognition-based user interface guidance and control system that recognizes the meaning of a procedure in various platform environments, such as apps, kiosks, and the web, and provides appropriate guidance to the user or controls dangerous situations. Background Technology

[0002] With the advancement of information and communication technology, various daily procedures—such as financial transactions, public services, and medical appointments—are shifting to a digital environment, leading to the widespread use of non-face-to-face user interfaces (UI), including smartphone apps and kiosks. However, this shift acts as a new barrier to entry for the elderly and digitally disadvantaged who are unfamiliar with using digital devices, resulting in input errors and process interruptions caused by complex procedures or small text.

[0003] Furthermore, psychological anxiety arising from users' inability to clearly perceive their current status (e.g., remittance confirmation, terms and conditions agreement) can lead to the abandonment of the service. In particular, as electronic financial fraud such as voice phishing becomes more sophisticated, cases of funds being stolen by exploiting users' anxieties are on the rise, making technical countermeasures urgently needed.

[0004] Conventional UI analysis technologies have primarily relied on analyzing the Document Object Model (DOM), which is the source code structure of web pages, or comparing predefined templates with the screen at the pixel level. These methods have limitations, as recognition accuracy drops significantly when the screen layout changes even slightly due to app updates, device resolutions, or platform changes (Android, iOS, Windows, etc.).

[0005] Furthermore, most existing abnormal financial transaction detection systems rely on server-side log analysis or are limited to simply checking for the installation of malicious apps, making it difficult to identify the specific steps currently being performed by the user in real time. Moreover, even when a risk is detected, they remain passive systems that merely display warning messages, posing a problem in that they cannot physically block further progress by users who are already intimidated.

[0006] In addition, technology has been proposed to detect risks by analyzing user behavioral patterns such as touch speed or hesitation; however, concerns have been raised that personal information leakage and security threats may occur during the process of transmitting sensitive input data to an external server for this purpose. Therefore, there is a need to develop a universal system that recognizes procedure steps robustly even with changes in platform or source code structure, safely analyzes user behavior within the terminal, blocks input in dangerous situations, and provides appropriate guidance in the event of operational incompetence. Prior art literature

[0007] Korean Patent Publication 10-2019-0109774 (Published September 27, 2019) The problem to be solved

[0008] The problem that the present invention aims to solve is to provide a system capable of robustly recognizing the state of a procedure by analyzing the relative positions and semantic relationships between screen components, even if the object hierarchy of a user interface (UI) changes or the absolute positions differ due to app updates or platform changes, in order to solve the above problem.

[0009] Furthermore, the present invention aims to simultaneously secure security and usability for the digitally marginalized by employing an active and variable control method that analyzes behavioral patterns, such as the user's touch duration and speed changes, in real time within the terminal without communication with an external server, thereby safely detecting dangerous situations without fear of personal information leakage, and physically blocking user input or providing guide messages to assist in the process depending on the situation.

[0010] The problems of the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0011] A procedural semantic recognition-based user interface guidance and control system according to one embodiment for solving the above problem comprises: a data storage unit storing data and programs necessary for performing a procedure; and a control unit that manages, guides, or supports the progress of a user interface and a user procedure displayed on a display by executing the program; wherein the control unit comprises: a structure generation module that extracts screen components from the user interface and generates relationship information between elements based on at least one of relative positions, spatial relationships, and structural attributes among the extracted screen components to form screen structure data for procedure recognition that can identify the visual structure of the screen even if the source code structure or object hierarchy structure of the application is changed; a state determination module that calculates similarity by comparing the screen structure data for procedure recognition with a reference data set stored in advance and determines a technical state variable representing the semantic state of the user procedure according to the result of the similarity calculation; and a risk calculation module that analyzes the user's touch input pattern on the screen of the user interface to extract behavioral feature data and calculates a risk score indicating the possibility of user operational incompetence or phishing damage by calculating in real time within the control unit based on the extracted behavioral feature data. It may include a control execution module that performs a combined operation to comprehensively determine the conditions corresponding to the determined semantic state and the calculated risk score, and provides a guidance message to the user to assist in the procedure according to the result of the combined operation, or generates a control signal to block input to the screen of the user interface if a dangerous situation is determined.

[0012] The above structure generation module can be configured to maintain the structural identity of the screen structure data for procedure recognition even if the screen layout of the user interface is changed or the absolute position of the components changes, by calculating position values ​​indicating the relative distance and direction between the screen components and text correlation scores indicating the contextual association between characters, respectively, and generating relationship information between the elements by reflecting importance in the position values ​​and text correlation scores.

[0013] The state determination module can convert the screen structure data for procedure recognition into a numeric array that can be processed by a computer, and perform an operation to calculate mathematical similarity by comparing it with a reference vector for each semantic state included in the reference data set, thereby determining the technical state variable representing the semantic state corresponding to the reference vector with the highest similarity as the current user procedure state.

[0014] The above risk calculation module can calculate the risk score by utilizing the computational capabilities of the control unit without communication with an external server to analyze in real time the coordinates, pressing time, touch interval, and finger movement speed at the moment the user touches the screen, and by generating behavioral feature data including at least one of the number of times the same location is repeatedly tapped, the touch-down duration or inter-tap interval of pressing the screen, and speed changes due to hand tremor.

[0015] The above control execution module implements input events in an intercept manner using an accessibility interface allowed by the operating system (OS) or a standard mechanism that performs the same input event relay and blocking functions as said accessibility interface, and normally maintains a state in which user input signals are immediately transmitted to said user interface without separate delay, and is variably activated only when the result of the combined operation is judged to be dangerous because it falls below a preset safety standard, thereby blocking the input signal in the middle before it reaches said user interface and invalidating the input, so as to prevent input delay caused by the security process.

[0016] The state determination module checks whether a security procedure, including whether authentication is complete or whether essential data is entered, is complied with when the determined semantic state is a state requiring security, such as password input or transfer execution, and the control execution module can physically forcibly block the screen transition if an attempt to switch screen is detected while the security procedure is not complied with, or if the risk score exceeds a safe range.

[0017] The structure generation module, in forming the screen structure data for procedure recognition, additionally combines sequence information indicating the input order between the screen components and importance information indicating how important each screen component is for the procedure progress, and the state determination module can correct the accuracy of semantic state determination by giving greater weight to whether the sequence information and importance information match the reference data set when calculating the similarity.

[0018] The control execution module described above can variably adjust and apply calculation weights or preset safety thresholds applied to the combined operation for each user based on user characteristic information, including the user's age, proficiency in using the device, or history of past errors.

[0019] The above data storage unit performs a federated learning function by receiving and storing user interface (UI) structure data of a new application or an updated reference data set from an external server, and the procedure recognition process of the above control unit can be performed independently within the terminal using data already stored in the above data storage unit.

[0020] Specific details of other embodiments are included in the detailed description and drawings. Effects of the invention

[0021] According to a procedural semantic recognition-based user interface guidance and control system according to one embodiment, even if the screen layout or source code structure (DOM, View Tree, etc.) changes due to app updates or device changes, technical state variables representing the semantic state of a user procedure can be robustly recognized through structural analysis based on visual-spatial relationship information. This enables the system to minimize malfunctions and provide stable services even in environments with frequent UI changes.

[0022] In addition, it can calculate a risk score by analyzing the user's behavioral patterns in real time within the terminal without communicating with an external server, prevent security incidents by physically blocking input in dangerous situations, and improve accessibility for the digitally marginalized by providing guidance in cases of lack of proficiency in operation.

[0023] The effects according to the embodiments are not limited to those exemplified above, and a wider variety of effects are included in this specification. Brief explanation of the drawing

[0024] FIG. 1 is a block diagram schematically illustrating the overall configuration of a procedural semantic recognition-based user interface guidance and control system according to one embodiment of the present invention. FIG. 2 is a block diagram illustrating the detailed configuration of a control unit according to one embodiment of the present invention. FIG. 3 is a data flowchart illustrating the process in which screen structure data for procedure recognition is formed according to an embodiment of the present invention, and technical state variables representing the semantic state of a user procedure are determined through the calculation of mathematical similarity with a reference data set. FIG. 4 is a conceptual diagram illustrating the process of a risk calculation module according to an embodiment of the present invention extracting behavioral feature data and calculating a risk score. FIG. 5 is a flowchart illustrating a logical process in which a system according to an embodiment of the present invention processes an input signal according to the result of a combined operation. FIG. 6 is a flowchart illustrating, step-by-step, the operation method of a procedural semantic recognition-based user interface guidance and control system according to one embodiment of the present invention. FIG. 7 is an example diagram illustrating a method of processing user input when a system according to an embodiment of the present invention is determined to be safe. FIG. 8 is an example diagram illustrating a method of blocking user input or displaying a warning / guidance message when a system according to an embodiment of the present invention determines that it is dangerous. Specific details for implementing the invention

[0025] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

[0026] All flowcharts, state transition diagrams, and pseudocode should be understood as representing various processes performed by a computer or processor that can be substantially represented on a computer-readable medium, regardless of whether the computer or processor is explicitly depicted.

[0027] The functions of the various elements illustrated in the drawings, including functional blocks represented as processors or similar concepts, may be provided by the use of dedicated hardware as well as hardware capable of executing software in conjunction with appropriate software. When provided by a processor, said functions may be provided by a single dedicated processor, a single shared processor, or a plurality of individual processors, some of which may be shared. Throughout the specification, the same reference numerals refer to the same components.

[0028] The terms "...part," "module," and "block" as described in the specification refer to a unit that processes at least one function or operation, and may be implemented in hardware or software, or a combination of hardware and software.

[0029] Parentheses used in the specification may mean "for example." Specifically, where indicated as "network (Bluetooth)," "Bluetooth" may be proposed as an example of a "network." In this case, the "network" of the present disclosure is not limited to "Bluetooth," but "Bluetooth" is proposed as an example of a "network."

[0030] In this specification, the term "Semantic State" refers to a technical state variable that is computationally calculated based on the structural relationships and time-series interaction patterns between screen components of a user interface, rather than human mental judgment. That is, the semantic state does not refer to the user's subjective judgment or cognitive state, but is a technical state variable as a procedural step-corresponding identification value calculated as a result of computation between screen structure data and a reference data set.

[0031] FIG. 1 is a block diagram schematically illustrating the overall configuration of a procedural semantic recognition-based user interface guidance and control system according to one embodiment of the present invention.

[0032] Referring to FIG. 1, a procedural semantic recognition-based user interface guidance and control system (1) according to one embodiment of the present invention includes a user interface (100), a control unit (200), and a data storage unit (300). The procedural semantic recognition-based user interface guidance and control system (1) can be applied to various electronic devices, such as smartphones, tablet PCs, and kiosks, where a user performs a specific procedure by touching the screen.

[0033] The above user interface (100) is a component that mediates interaction between the user and the system, outputting visual information through a display screen and receiving input signals from the user through a touch panel, etc. The above user interface (100) is not limited to simply displaying images, but includes various screen components (e.g., text, images, icons, buttons, input fields, etc.), and the user performs a series of procedures such as financial transfer, shopping payment, and login through it. At this time, the layout of the screen displayed on the above user interface (100) may change frequently depending on the type of application being executed or the update status of the web page.

[0034] The data storage unit (300) stores various data and programs necessary for operating the system (1), applications, and procedures. In particular, the data storage unit (300) stores a reference data set that the control unit (200), which will be described later, refers to in order to determine technical state variables corresponding to the current screen. The reference data set is a database containing characteristic values ​​of the screen structure appearing in a normal procedure flow and safety standards required corresponding to each semantic state, and is managed to respond to the latest types of phishing patterns by continuously updating it through big data analysis and machine learning processes of the system.

[0035] Meanwhile, the data storage unit (300) is basically located inside the terminal and can selectively perform a federated learning function by receiving and updating user interface (UI) structure data of a new type of phishing app or an updated reference data set from an external server only when necessary. However, in principle, real-time procedure recognition and control operations are performed on-device without external communication.

[0036] The control unit (200) is a core computing device that executes a program stored in the data storage unit (300) to control the user interface (100) displayed on the display and the entire procedure. The control unit (200) can be implemented as a CPU, an AP (Application Processor), or a separate security chipset, and analyzes signals input by the user through the user interface (100) in real time. The control unit (200) does not merely play a passive role of accepting input, but comprehensively calculates the semantic state of the user procedure calculated based on the visual structure of the current screen and the risk score calculated based on the user's input behavior characteristics, and determines whether to intervene conditionally.

[0037] FIG. 2 is a block diagram illustrating the detailed configuration of a control unit according to one embodiment of the present invention.

[0038] Referring to FIG. 2, the control unit (200) includes a structure generation module (210), a state determination module (220), a risk calculation module (230), and a control execution module (240). Each of the modules may be a hardware-separated circuit or a software block executed within the control unit (200).

[0039] The structure generation module (210) extracts screen components, including text, images, icons, buttons, input fields, etc., and their placement information from the user interface (100). In this specification, the screen components are a concept that encompasses not only the Document Object Model (DOM) elements of a web browser, but also View objects of native applications such as Android / iOS, and User Interface (UI) automation elements of Windows. Accordingly, the structure generation module (210) forms screen structure data for procedural recognition that can identify the visual structure of the screen even if the Document Object Model (DOM), which is the source code of the web page, is changed. For example, even if the ID or tag structure of a button changes due to a website update, if the visual placement of the confirmation button being located at the bottom of the text input field is maintained, the structure generation module (210) can recognize this as the same or similar structure.

[0040] Specifically, the structure generation module (210) can convert and store screen components into a graph structure in which the relationships between elements are nodes and the relationships between elements are edges. At this time, even if non-structural attributes such as the button color, border width, or tag type (Div -> Span) are changed due to a web page update, a fuzzy matching algorithm can be applied to determine that the structure is the same if the topology on the graph matches 80% or more.

[0041] Additionally, the structure generation module (210) calculates position values ​​indicating the relative distance and direction between the screen components and text correlation scores indicating the contextual association between characters, respectively. By generating relationship information between the elements by reflecting importance in the position values ​​and text correlation scores, the structural identity of the screen structure data for procedure recognition is maintained even if the screen layout of the user interface (100) changes or the absolute position of the components changes. This is based on the observation that the relative relationship (topology) between components does not change even when absolute coordinates change due to a change in screen resolution or font size, thereby providing robust recognition performance.

[0042] Furthermore, when the structure generation module (210) forms the screen structure data for the procedure recognition, it generates sequence information by analyzing the order of the tab index extending from the top to the bottom and from the left to the right of the screen. At the same time, it tags importance information that assigns a high weight to input fields or buttons containing security-sensitive keywords such as passwords, resident registration numbers, and transfers, and combines this information with the data. This importance information can be determined by referring to a predefined security dictionary.

[0043] In addition, the structure generation module (210) can generate relationship information between elements by considering not only the relative position and spatial arrangement relationship between the screen components, but also structural attributes such as input order, connection type, and logical dependency relationship. At this time, the relationship information between elements is generated based on at least one of the relative position, spatial relationship, and structural attribute, and it is also possible to selectively use only some of these depending on the implementation type.

[0044] The state determination module (220) calculates a similarity by comparing the screen structure data for procedure recognition with a reference data set that has been stored in advance, and determines whether the current screen corresponds to a technical state variable representing the semantic state of the user procedure based on the result of the similarity calculation.

[0045] FIG. 3 is a data flowchart illustrating the process in which screen structure data for procedure recognition is formed according to an embodiment of the present invention, and technical state variables representing the semantic state of a user procedure are determined through the calculation of mathematical similarity with a reference data set.

[0046] Referring to FIG. 3, the state determination module (220) converts the screen structure data for procedure recognition received from the structure generation module (210) into a numeric array (Vector Array) that can be processed by a computer. At this time, the numeric array is composed of multiple feature values ​​including relative position values ​​of screen components, connection relationships between elements, input order information, and importance information, and each reference value included in the reference data set is also stored as a reference vector having the same feature dimension. Subsequently, an operation to calculate mathematical similarity based on Euclidean distance or cosine similarity is performed by comparing with each semantic state-specific reference vector included in the reference data set. The mathematical similarity calculation is performed as an operation to quantitatively calculate the distance or directional similarity between the numeric array and the reference vector, and the result of the operation is output as a numeric value and used in a subsequent state determination operation.

[0047] When calculating the similarity above, the entire screen is converted into a single vector to calculate Cosine Similarity; however, for areas tagged with the aforementioned importance information (e.g., transfer button, account number input field), a weighted Euclidean Distance calculation may be performed in parallel. This is intended to identify fake apps where the location of core security functions has changed, even if the overall layout is similar.

[0048] At this time, the similarity calculation is not limited to simple shape matching, but is calculated by applying differential weights, such as 1.5 to 2 times, depending on whether elements with high importance information (e.g., a remittance button) match the reference data. Through this, the core semantic state can be accurately identified even if non-essential elements, such as background images or advertising banners, are changed.

[0049] The state determination module (220) determines the current user procedure state by establishing a technical state variable representing a semantic state corresponding to the reference value with the highest calculated similarity. Additionally, the state determination module (220) corrects the accuracy of the semantic state determination by giving greater weight to the similarity calculation whether the sequence information and importance information generated by the structure generation module (210) above match the reference data set. That is, by considering not only the appearance of the screen but also the logical sequence in which the user must perform input, it clearly distinguishes whether the current situation corresponds to a semantic state corresponding to remittance confirmation or a semantic state corresponding to a simple advertisement popup.

[0050] The above risk calculation module (230) analyzes the user's touch input pattern on the screen of the user interface (100) to extract behavioral feature data, and calculates a risk score indicating the possibility of user's lack of operation skills or phishing damage by calculating in real time within the control unit (200) based on the extracted behavioral feature data.

[0051] FIG. 4 is a conceptual diagram illustrating the process of a risk calculation module according to an embodiment of the present invention extracting behavioral feature data and calculating a risk score.

[0052] Referring to FIG. 4, the risk calculation module (230) receives inputs such as touch coordinates, movement speed, touch duration, touch interval, hand tremor and acceleration change, and performs internal calculations.

[0053] Specifically, the risk calculation module (230) analyzes the coordinates at the moment the user touches the screen, the pressing time, the touch interval, and the finger movement speed in real time. Through this, it generates behavioral characteristic data including at least one of the number of times the same location is repeatedly tapped, the touch-down duration or inter-tap interval of pressing the screen, and changes in speed due to hand tremors. For example, a user who is a victim of voice phishing tends to tremble their hand in panic or touch the surrounding area multiple times without accurately pressing the buttons on the screen. The risk calculation module (230) quantifies these subtle biological behavioral signals and converts them into a risk score.

[0054] Specifically, the risk calculation module (230) measures the degree of hand tremor by calculating the variance value of touch coordinates in real time, and may consider it an abnormal pattern if the amount of change in touch pressure or area deviates from the average value range of the user by more than 2 standard deviations. In addition, the risk calculation module (230) calculates the jitter value, which is a minute shaking of the touch input coordinates, and simultaneously fuses and analyzes the gyroscope and accelerometer data embedded in the smartphone. For example, an irregular touch that occurs when the device is placed flat on the floor (acceleration Z-axis value close to 1g) is corrected as a simple mistake by the elderly or infirm, whereas an urgent touch input when the device is held in the hand and shaken irregularly (gyroscope value fluctuation) is considered as a state of psychological anxiety caused by voice phishing and a high risk score is applied.

[0055] In addition, by fusing gyroscope values, it is possible to determine whether the device is placed on a desk or held in the hand, and accordingly, dynamically correct the tolerance range for touch errors to reduce false positives.

[0056] In addition, the risk calculation module (230) has a real-time calculation structure within the control unit (200) that, when performing the process of extracting screen components and calculating risk scores, does not transmit the collected data to an external server but processes it in real-time only within the internal computational resources and volatile memory of the control unit (200) and then immediately destroys it. This is for the protection of personal information and fundamentally blocks the risk of the user's touch patterns or screen information being leaked externally. Since all calculations are performed independently (on-device) within the control unit (200), security functions can be maintained even in situations where the network connection is unstable.

[0057] The control execution module (240) performs a combined operation that comprehensively reflects the safety threshold and / or weight set in the reference data set and the calculated risk score in response to the determined semantic state. The combined operation includes an operation that compares the risk score with the safety threshold set in advance in response to the determined semantic state, or determines whether the calculation result obtained by applying a state-specific weight to the risk score exceeds a threshold value.

[0058] FIG. 5 is a flowchart illustrating a logical process in which a system according to an embodiment of the present invention processes an input signal according to the result of a combined operation.

[0059] Referring to FIG. 5, when a user input signal is received, the control execution module (240) determines whether the result of the combined operation does not meet a preset safety standard. If it is determined to be No (safe), the input signal is immediately transmitted via Event Pass-through to the user interface. On the other hand, if it is determined to be Yes (dangerous), the control execution module (240) invalidates the input and generates a control signal that creates a warning popup.

[0060] FIG. 6 is a flowchart illustrating the overall operation method (S100) of a procedure semantic recognition-based user interface guidance and control system according to one embodiment of the present invention.

[0061] Referring to FIG. 6, the operation method (S100) may include the step of extracting screen components and relationship information (S110), the step of forming screen structure data for procedure recognition (S120), and the step of determining a technical state variable representing the semantic state of a user procedure by comparing it with reference data (S130).

[0062] Specifically, in the step of extracting screen components and relationship information (S110), the control unit (200) recognizes the location, size, and text content of graphic elements displayed on the user interface (100) and measures the distance between them. In the subsequent step of forming screen structure data for procedure recognition (S120), structure data that is identifiable even if the layout changes is generated based on the measured information, and in the step of determining technical state variables representing the semantic state of the user procedure by comparing with reference data (S130), the current screen is compared with a pre-learned set of reference data to determine which state among the semantic states of the user procedure it corresponds to.

[0063] Additionally, the above operation method (S100) may include a step of calculating a risk score by analyzing behavioral characteristics (S140), a step of performing a combination operation (S150), and a step of generating a control signal when safety standards are not met (S160).

[0064] In the step of calculating a risk score by analyzing the above behavioral characteristics (S140), hand tremors, repetitive input, abnormal touch speed, etc., are detected in the user's touch input to derive a quantified risk score. In the step of performing the above combination operation (S150), the security importance of the current procedure and the calculated risk score are combined to determine the final risk level. Finally, in the step of generating a control signal when safety standards are not met (S160), if the risk level exceeds a threshold, touch input is blocked to prevent phishing damage or erroneous operation and a warning is sent to the user.

[0065] FIG. 7 is an exemplary diagram illustrating a method of processing user input when a system according to an embodiment of the present invention is determined to be safe, and FIG. 8 is an exemplary diagram illustrating a method of blocking user input or displaying a warning / guidance message when a system according to an embodiment of the present invention is determined to be dangerous.

[0066] First, referring to FIGS. 7 and 8, the control execution module (240) may be implemented using an accessibility interface allowed by the operating system (OS) or a standard mechanism that performs the same input event relay and blocking functions as the accessibility interface. Normally, it maintains a state where user input signals are immediately transmitted to the user interface (100) without any separate delay to ensure comfortable usability. However, if, as a result of the combined operation, it is determined that the user is having difficulty with the operation (e.g., repeated incorrect input) or that a security risk situation is determined, the control execution module (240) performs selective control operations, such as displaying a guide user interface (UI) to help move to the next step depending on the situation, or variably blocking and nullifying the input signal before it reaches the user interface and generating a warning popup.

[0067] The event control of the present invention follows an accessibility interface permitted by the operating system (OS) or a standard mechanism equivalent thereto, and is performed in a non-invasive manner that does not directly modify the internal logic of the system kernel or other applications.

[0068] Furthermore, the above control operation is performed within the scope of input event processing or screen overlay display through accessibility / event relay APIs provided by the operating system, and is not implemented by hooking or patching application executable files, memory areas, or internal function logic.

[0069] Additionally, the control execution module (240) may variably adjust and apply the calculation weights applied to the combined operation or the safety thresholds for each user based on user characteristic information, including the user's age, proficiency in using the device, or history of past mistakes. For example, in the case of an elderly user who is inexperienced in using a smartphone, the allowable range for hand tremors may be widened, while stricter standards may be applied regarding the omission of security procedures (such as password entry), thereby providing a personalized security environment. This user characteristic information may be received during the initial system setup or may be automatically profiled by learning the user's usual touch patterns collected over a certain period using machine learning.

[0070] For example, the control unit (200) can perform auto-tuning linked to system setting values, such as classifying the user as an elderly or digitally excluded group when the font size stored in the device's system settings is set to 'large' or accessibility mode is activated, and applying an initial safety standard that is relaxed by 20% compared to a general user.

[0071] Additionally, the state determination module (220) checks whether the security procedure is complied with, including whether authentication is complete or whether essential data is entered, when the determined semantic state is a state requiring security, such as password input or transfer execution. The control execution module (240) prevents a malicious app from proceeding with the procedure without the user's consent by invalidating the screen switching attempt at the input event stage if a screen switching attempt is detected while the security procedure is not complied with, or if the risk score is outside the safe range, thereby blocking touch input from being transmitted to the user interface, or by displaying an overlay view on the screen to restrict subsequent operations, thereby covering the entire screen with an opaque overlay view to prevent the user from performing subsequent operations.

[0072] Meanwhile, the control operation of the present invention is performed for the purpose of security and safety with user consent and is implemented within the scope of accessibility interfaces or standard mechanisms permitted by the operating system (OS). Furthermore, it is performed in a non-invasive manner that does not directly modify the system kernel or the internal logic of other applications.

[0073] Although embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention may be implemented in other specific forms without changing the technical concept or essential features thereof. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. Explanation of the symbols

[0074] 1: Procedural Semantic Recognition-Based User Interface Guidance and Control System 100: User Interface 200: Control unit 210: Structure Generation Module 220: State determination module 230: Risk Calculation Module 240: Control Execution Module 300: Data storage unit

Claims

Claim 1 A data storage unit storing data and programs necessary for performing a procedure; and a control unit that executes the program to manage, guide, or support the progress of a user interface and a user procedure displayed on a display; wherein the control unit comprises: a structure generation module that extracts screen components from the user interface and generates relationship information between elements based on at least one of the relative position, spatial relationship, and structural attribute between the extracted screen components to form screen structure data for procedure recognition that can identify the visual structure of the screen even if the source code structure or object hierarchy structure of the application is changed; a state determination module that calculates similarity by comparing the screen structure data for procedure recognition with a reference data set stored in advance and determines a technical state variable representing the semantic state of the user procedure according to the result of the similarity calculation; and a risk calculation module that analyzes the user's touch input pattern on the screen of the user interface to extract behavioral feature data and calculates a risk score indicating the possibility of user operational incompetence or phishing damage by calculating in real time within the control unit based on the extracted behavioral feature data. A control execution module that performs a combined operation to comprehensively determine the condition corresponding to the determined semantic state and the calculated risk score, and, based on the result of the combined operation, provides a guidance message to the user to assist in the procedure, or generates a control signal to block input to the screen of the user interface if a dangerous situation is determined;A procedural semantic recognition-based user interface guidance and control system comprising, wherein a risk score indicating the possibility of user operational incompetence or phishing damage is calculated in real time within the control unit based on behavioral feature data extracted from the user's touch input pattern on the screen of the user interface, and is characterized by being calculated by quantifying the behavioral feature data. Claim 2 A procedural semantic recognition-based user interface guidance and control system according to claim 1, wherein the structure generation module calculates a position value indicating the relative distance and direction between the screen components and a text association score indicating the contextual association between characters, and generates relationship information between the elements by reflecting importance in the position value and the text association score, so as to maintain the structural identity of the screen structure data for procedural recognition even if the screen layout of the user interface is changed or the absolute position of the components changes. Claim 3 A procedural semantic recognition-based user interface guidance and control system according to claim 1, wherein the state determination module converts the screen structure data for procedural recognition into a numeric array that can be processed by a computer, performs an operation to calculate mathematical similarity by comparing it with a reference vector for each semantic state included in the reference data set, and determines a technical state variable representing the semantic state corresponding to the reference vector with the highest similarity as the current user procedural state. Claim 4 A procedural semantic recognition-based user interface guidance and control system according to claim 1, wherein the risk calculation module analyzes in real time the coordinates at the moment a user touches the screen, the pressing time, the touch interval, and the finger movement speed using the computational capabilities of the control unit without communication with an external server, and calculates the risk score by generating behavioral feature data including at least one of the number of times the same location is repeatedly tapped, the touch-down duration or inter-tap interval of pressing the screen, and the speed change due to hand tremor, wherein the speed change due to hand tremor is calculated based on at least one of the degree of hand tremor measured by calculating the variance value of the touch coordinates in real time, the jitter value which is a fine shaking of the touch input coordinates, and gyroscope and accelerometer data embedded in the smartphone. Claim 5 A procedural semantic recognition-based user interface guidance and control system according to claim 1, wherein the control execution module implements input events in an intercept manner using an accessibility interface allowed by an operating system (OS) or a standard mechanism that performs the same input event relay and blocking functions as said accessibility interface, and normally maintains a state in which user input signals are immediately transmitted to said user interface without separate delay, and is variably activated only when the result of the combined operation falls below a preset safety standard and is judged to be dangerous, thereby blocking the input signal in the middle before it reaches said user interface to nullify the input, thereby preventing input delay caused by a security process. Claim 6 A procedure semantic recognition-based user interface guidance and control system according to claim 1, wherein the state determination module checks whether a security procedure, including whether authentication is completed or whether essential data is entered, is complied with when the determined semantic state is a state requiring security, such as password input or transfer execution, and the control execution module physically forcibly blocks the screen transition when an attempt to switch screen is detected in a state where the security procedure is not complied with, or when the risk score exceeds a safe range. Claim 7 A procedural semantic recognition-based user interface guidance and control system according to claim 1, wherein the structure generation module, in forming the screen structure data for procedural recognition, additionally combines sequence information indicating the input order between the screen components and importance information indicating how important each screen component is to the progress of the procedural process, and the state determination module reflects more significantly as a weight in the similarity calculation whether the sequence information and importance information match the reference data set, thereby correcting the accuracy of semantic state determination. Claim 8 A procedural semantic recognition-based user interface guidance and control system, wherein, in claim 1, the control execution module variably adjusts and applies calculation weights or preset safety thresholds applied to the combination operation for each user based on user characteristic information including the user's age, proficiency in using the device, or past error history. Claim 9 A procedure semantic recognition-based user interface guidance and control system according to claim 1, wherein the data storage unit performs a federated learning function of receiving and storing user interface (UI) structure data of a new application or an updated reference data set from an external server, and wherein the procedure recognition process of the control unit is performed independently within the terminal using data already stored in the data storage unit.

Images