System and method for providing overlay-based software usage guidance using position registry creation, abstract identifier-based data processing, and geometric alignment

Abstract

An overlay-based software usage guide method may include: acquiring a user screen displaying a target application; identifying the on-screen location of at least one UI element included in the target application by comparing a template image corresponding to each of a plurality of pre-registered UI elements with the user screen; creating a location registry by mapping an abstract identifier corresponding to each of at least one UI element with coordinates representing the on-screen location; acquiring lecture-related data including an abstract identifier, visual guidance data, a text token, and locale data; identifying a string corresponding to the abstract identifier included in the text token and the target language set for the user from the locale data; replacing the text token with the identified string; identifying target coordinates corresponding to the abstract identifier included in the visual guidance data from the location registry; and displaying visual guidance aligned with the UI element corresponding to the abstract identifier based on the target coordinates through a transparent overlay placed on the target application.

Description

Technology Field

[0001] The present invention relates to an overlay-based software usage guidance system and method. Background Technology

[0002] Recently, as the scope of application software utilization has expanded across various fields such as graphic design, engineering, document creation, and video editing, the demand for educational content to effectively learn how to use such software is also increasing. Since this software provides complex user interfaces containing numerous menus, tools, panels, and settings, users need to learn not only simple function descriptions but also the location of specific functions and the sequence of operations on the actual screen. Accordingly, various forms of software usage training methods are being utilized, including video lectures, image-based tutorials, and step-by-step instruction manuals.

[0003] However, in conventional software usage training, learners often have to alternate between checking the screen displaying lecture content and the application screen where actual tasks are performed. For example, after checking the operation method on the lecture screen, a learner switches to the target application to perform the task; if they forget the sequence of operations or cannot locate the target function during the process, they must return to the lecture screen to check the content. Such repetitive screen switching can reduce usability by interrupting the learning flow and requiring learners to independently determine the correspondence between the lecture content and the actual work screen.

[0004] Furthermore, even for the same application software, menu names, shortcuts, and the placement or on-screen position of tools may vary depending on the operating system, software version, display language, screen resolution, display scale, and user interface settings. Consequently, instructional screens created for a specific environment or guidance provided based on fixed coordinates may not match actual UI elements in other usage environments. Moreover, creating separate instructional content tailored to each language and execution environment can increase the burden of content creation and maintenance. Accordingly, there is a need for software usage guidance technology that reflects the user's actual execution environment while enabling the utilization of a single educational content across various settings. The problem to be solved

[0005] One objective of the present invention is to provide a software usage guidance method that allows a user learning software usage to check the necessary operation locations and operation details on the screen of the target application currently in use, without having to repeatedly switch between a separate lecture screen and a work screen. Through this, the burden of having to personally determine the spatial correspondence between the lecture content and the actual work target is reduced, and the actual work flow and the training process can proceed continuously.

[0006] Another objective of the present invention is to enable the determination of a guidance position based on the actual execution screen, even when the user interface of a target application is displayed in different locations or forms for each user. In particular, instead of fixedly assigning specific screen coordinates during the content creation stage, the invention aims to provide a guidance structure capable of responding to changes in screen resolution, display scale, interface layout, or the application's display environment by identifying target UI elements at the time of execution and utilizing location information associated with those UI elements.

[0007] Another objective of the present invention is to improve the reusability and scalability of content by reducing the dependency between lecture content explaining software usage and the actual user environment. For example, the invention aims to enable a single lecture content to be utilized in multiple languages ​​or multiple execution environments by managing UI elements representing the same function using commonly referenced identification information rather than specific language names or fixed coordinates, and by determining display strings, locations, and visual guidance suitable for the execution environment.

[0008] The technical problems of the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0009] An overlay-based software usage guidance method performed by an electronic device including a processor according to an embodiment of the present invention may include: the processor acquiring a user screen on which a target application is displayed, and identifying the screen location of at least one UI element included in the target application by comparing a template image corresponding to each of a plurality of pre-registered UI elements with the user screen; the processor creating a location registry by mapping an abstract identifier corresponding to each of the at least one UI element with coordinates representing the screen location; the processor acquiring lecture-related data including the abstract identifier, visual guidance data, a text token, and locale data, identifying a string corresponding to the abstract identifier included in the text token and a target language set for the user from the locale data, and replacing the text token with the identified string; and the processor identifying target coordinates corresponding to the abstract identifier included in the visual guidance data from the location registry, and displaying visual guidance aligned with the UI element corresponding to the abstract identifier based on the target coordinates through a transparent overlay placed on the target application.

[0010] The above guide on how to use the overlay-based software is,

[0011] The above processor calculates a screen re-recognition trigger index based on the user operation state at the current step and the UI element detection result at the time of creation of the location registry, based on the following mathematical formula; and

[0012] The processor further includes the operation of, when the screen re-recognition trigger index is greater than or equal to a set threshold, re-acquiring the user screen to re-identify the on-screen location of at least one UI element included in the target application, and updating at least a portion of the location registry based on the re-identified on-screen location.

[0013] [Mathematical Formula]

[0014]

[0015] The above I_resync is a screen re-recognition trigger index indicating the degree of need for re-recognition of the user screen in the current step; the above T_dwell is the user's dwell time measured from the entry point of the current step; the above T_ref is a reference dwell time set to normalize the dwell time; the above D_mouse is the cumulative movement distance of the mouse pointer measured after the entry of the current step; the above D_ref is a reference mouse movement distance set to normalize the cumulative movement distance; the above N_err is the number of user inputs for an area that does not correspond to the UI element subject to the visual guidance in the current step; the above N_ref is a reference number of incorrect inputs set to normalize the number of user inputs; the above C_init is an initial detection confidence level of 0 or greater and 1 or less calculated for the UI element subject to the visual guidance when the location registry is created; the above α is a first weight for setting the degree of influence that an increase in the dwell time has on the screen re-recognition trigger index; and the above β is the increase in the cumulative movement distance of the mouse pointer that the screen re-recognition A second weight for setting the degree of influence on the activation index, wherein γ is a third weight for setting the degree of influence of an increase in the number of user inputs on the screen re-recognition activation index, wherein p is a first exponential coefficient for setting the degree to which the screen re-recognition activation index increases non-linearly as the number of user inputs increases, wherein δ is a fourth weight for setting the degree to which a decrease in the initial detection reliability affects the screen re-recognition activation index, and wherein q is a second exponential coefficient for setting the degree to which the screen re-recognition activation index increases non-linearly as the initial detection reliability decreases.The above η is an interaction weight for reflecting the combined effect in the screen re-recognition trigger index when the number of user inputs increases while the initial detection reliability decreases simultaneously, and the above I_th is a threshold value set to determine whether the screen re-recognition process is automatically executed, and the above α, β, γ, δ, and η are each set to a value greater than 0, and the above p and q can each be set to a value greater than 1.

[0016] The above guide on how to use the overlay-based software is,

[0017] The above processor determines whether the current position coordinates of the mouse pointer have entered the boundary area of ​​the UI element stored in the position registry; and

[0018] The above processor may further include an operation of generating tactile feedback by transmitting a vibration pattern control signal to a user's wearable device through a near-field wireless communication module when it is determined that the mouse pointer has entered the boundary area of ​​the UI element.

[0019] The above guide on how to use the overlay-based software is,

[0020] The above processor detects a window coordinate change event in which the execution window of the target application moves from a first display to a second display; and

[0021] The processor may further include an operation of obtaining the DPI scale and monitor start offset coordinates of the second display, and reordering the display position of the transparent overlay by arithmetic applying the DPI scale and monitor start offset coordinates to the target coordinates stored in the position registry without executing a full screen re-recognition process.

[0022] The above guide on how to use the overlay-based software is,

[0023] The above processor performs the operation of measuring the amount of change in the movement coordinates of the mouse pointer per unit time and calculating the instantaneous acceleration and the number of direction changes of the mouse pointer; and

[0024] If the processor determines that the instantaneous acceleration exceeds a set reference acceleration and simultaneously the number of direction changes exceeds a set threshold number, resulting in a panic state, it may further include an operation of immediately enlarging and rendering the border thickness of the visual guide and the diameter of the alignment circle displayed on the transparent overlay by a set ratio.

[0025] The action of displaying the above visual guidance is,

[0026] The above processor performs the operation of creating a black alpha layer with a set opacity that covers the entire user screen on the transparent overlay; and

[0027] The processor may include an operation of inverting only a specific area on the black alpha layer corresponding to the target coordinates identified in the location registry and the size area of ​​the UI element to an alpha value of 0, thereby exposing only the UI element area in a spotlight form.

[0028] The above guide on how to use the overlay-based software is,

[0029] The above processor defines a graphic object area on the transparent overlay where the visual guide and subtitle data are displayed as a first area, and defines the remaining area excluding the first area as a second area; and

[0030] The processor may include an operation of receiving and processing mouse events generated in the first area at the overlay processor, and dynamically resetting window area properties so that mouse focus and click events generated in the second area are directly transmitted to the target application placed underneath by clicking through the transparent overlay. Effects of the invention

[0031] The effects of the overlay-based software usage guidance method according to embodiments of the present invention are described as follows.

[0032] According to an embodiment of the present invention, instructions regarding software usage can be provided on the actual user screen where the target application is displayed, so the user can perform the corresponding task while checking the instructions without repeatedly switching between the lecture screen and the work screen. Accordingly, the process of memorizing lecture content and reproducing it on a separate screen is reduced, and the time spent navigating the screen to find specific menus or tools is decreased, thereby improving the continuity and convenience of software usage education.

[0033] According to an embodiment of the present invention, since visual guidance can be provided based on the location of UI elements actually identified on the user screen, the guidance position can be adjusted even when the screen configuration at the time of content creation differs from the user's actual screen configuration. Accordingly, the phenomenon where arrows, highlighted areas, or guidance indicators are misaligned with actual UI elements due to differences in screen resolution, display scale, tool placement, etc., can be reduced, and the adaptability of lecture content to changes in the user environment can be improved.

[0034] According to an embodiment of the present invention, since lecture content, language-specific display information, and on-screen location information can be managed in conjunction with identification information corresponding to UI elements, educational content can be configured in a form that is not dependent on a specific language or screen environment. Accordingly, the burden of having to repeatedly produce the same lecture content for each language or environment can be reduced, and even when the language, version, or user environment of the target application changes, it can be responded to by updating a portion of the relevant data, thereby improving the efficiency of content maintenance and expansion.

[0035] In addition, various effects identified directly or indirectly through this document may be provided. Brief explanation of the drawing

[0036] FIG. 1 is a block diagram showing the components of an electronic device according to one embodiment of the present invention. FIG. 2 is a block diagram showing the components of an overlay-based software usage guide system according to one embodiment of the present invention. FIG. 3 is a flowchart of the operation of an overlay-based software usage guide method according to one embodiment of the present invention. FIG. 4 is a configuration diagram showing the connection relationships between a plurality of applications, a screen recognition engine, and a database according to one embodiment of the present invention. FIG. 5 is a flowchart illustrating the process of querying, sorting, and assembling lecture-related data according to one embodiment of the present invention. FIG. 6 is an example diagram showing a state in which visual guidance is displayed on a user screen according to one embodiment of the present invention. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Specific details for implementing the invention

[0037] Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. It should be noted that in assigning reference numerals to the components of each drawing, the same components are given the same reference numeral whenever possible, even if they are shown in different drawings. Furthermore, in describing the embodiments of the present invention, if it is determined that a detailed description of related known components or functions would hinder understanding of the embodiments of the present invention, such detailed description is omitted.

[0038] In describing the components of the embodiments of the present invention, terms such as first, second, A, B, (a), (b), etc., may be used. These terms are intended merely to distinguish the components from other components, and the essence, order, or sequence of the components is not limited by the terms. Furthermore, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0039] Hereinafter, embodiments of the present invention will be described in detail with reference to FIGS. 1 to 3.

[0041] FIG. 1 is a block diagram showing the components of an electronic device according to one embodiment of the present invention.

[0042] According to one embodiment, the electronic device (100) may include a memory (110), a processor (120), a communication interface (130), and / or a display device (140). The configuration of the electronic device (100) illustrated in FIG. 1 is exemplary and the embodiments of the present invention are not limited thereto. For example, the electronic device (100) may further include components not illustrated in FIG. 1 (e.g., a user interface, an input device, a notification unit, a sensor unit, or at least one of any combination thereof).

[0043] According to one embodiment, the memory (110) may store instructions or data. For example, the memory (110) may store one or more instructions that cause the electronic device (100) to perform various operations when executed by the processor (120).

[0044] For example, the memory (110) may be implemented as a single chipset with the processor (120). The processor (120) may include at least one of a communication processor or a modem.

[0045] For example, the memory (110) can store various information related to the electronic device (100). For example, the memory (110) can store information regarding the operation history of the processor (120). For example, the memory (110) can store input data acquired by the electronic device (100), output data output by the electronic device (100), data acquired from an external server (220), a user terminal (210), and / or a target application.

[0046] For example, the memory (110) may include multiple storage devices of different types. For example, the memory (110) may include volatile and / or non-volatile storage media. For example, the memory (110) may include at least one of RAM (random-access memory), ROM (read only memory), eMMC (Embedded Multi-Media Card), or any combination thereof.

[0047] The steps of the method or algorithm described in connection with the embodiments of the present disclosure may be directly implemented in hardware, software modules, or a combination of both, executed by a processor (120). The software modules may reside in a storage medium (i.e., memory (110)) such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, or a CD-ROM.

[0048] For example, the memory (110) is coupled to a processor (120), and the processor (120) can read information from a storage medium and write information to a storage medium. Alternatively, the memory (110) may be integrated with the processor (120). The memory (110) and the processor (120) may reside within an application-specific integrated circuit (ASIC). The ASIC may reside within a user terminal. Alternatively, the memory (110) and the processor (120) may reside as separate components within the user terminal.

[0049] According to one embodiment, the processor (120) may be operatively connected to the memory (110), the communication interface (130), and / or the display device (140). For example, the processor (120) may control the operation of the memory (110), the communication interface (130), and / or the display device (140).

[0050] According to one embodiment, the communication interface (130) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (100) and an external device (e.g., user terminal (210) of FIG. 2, external server (220), etc.), and the performance of communication through the established communication channel. The communication interface (130) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication interface (130) may include a wireless communication module (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device through a first network (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a long-range communication network such as a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module can identify or authenticate the electronic device (100) within a communication network, such as the first network or the second network, using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in a subscriber identification module.

[0051] According to one embodiment, the display device (140) may include at least one output device that provides various information and a user interface to the user.

[0052] For example, the display device (140) may include a display device, an audio output device, a virtual reality output device, etc.

[0053] For example, the display device (140) can provide the user with various types of user interfaces described in the present disclosure visually and / or audibly.

[0054] The components of the electronic device (100) illustrated in FIG. 1 are exemplary, and the embodiments of the present disclosure are not limited thereto.

[0055] At least some of the embodiments of the present disclosure may be implemented as artificial intelligence (AI) through a processor (120) and memory (110) of an electronic device (100). The processor (120) may be composed of one or more processors, and the one or more processors may be general-purpose processors such as a CPU, AP, DSP (digital signal processor), etc., graphics-dedicated processors such as a GPU, VPU (vision processing unit), or artificial intelligence-dedicated processors such as an NPU. The one or more processors may be controlled to process input data according to predefined operation rules or artificial intelligence models stored in memory (110). Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0056] The predefined operation rules or artificial intelligence model are characterized by being created through learning. Here, being created through learning means that a predefined operation rules or artificial intelligence model configured to perform a desired characteristic (or purpose) is created by a basic artificial intelligence model being trained using a number of learning data by a learning algorithm. Such learning may be performed within the electronic device (100) itself where the artificial intelligence according to the present disclosure is performed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0057] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and can perform neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights can be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. Artificial neural networks may include, but are not limited to, deep neural networks (DNN), convolutional neural networks (CNN), recurrent neural networks (RNN), restricted Boltzmann machines (RBM), deep belief networks (DBN), bidirectional recurrent deep neural networks (BRDNN), or deep Q-networks.

[0058] According to one embodiment, the electronic device (100) may use at least one artificial intelligence model to identify UI elements from a user screen of a target application, process lecture-related data, and determine visual guidance to be provided to the user screen.

[0059] For example, the artificial intelligence model may be stored in the memory (110) of the electronic device (100) and executed by the processor (120), or it may be deployed on an external server to receive input data from the electronic device (100) and return an inference result. The artificial intelligence model may be implemented as a single integrated model, or it may include multiple models separated by functions such as screen recognition, language processing, guidance location determination, and user state analysis.

[0060] In one embodiment, the electronic device (100) can identify the location of at least one UI element by inputting a user screen displaying a target application into an artificial intelligence-based screen recognition model.

[0061] For example, the screen recognition model may include at least one of an object detection model, an image classification model, a feature point extraction model, or a vision transformer-based model, and can identify candidate regions corresponding to menus, buttons, icons, panels, input areas, or tool areas within the user screen. The electronic device (100) may further calculate the similarity between the candidate regions identified by the screen recognition model and a pre-registered template image to determine the final UI element, or conversely, provide the template matching result as input to an artificial intelligence model to correct the accuracy of the detection result. Accordingly, the location of the UI element can be identified even when the appearance of the UI element changes partially depending on the screen resolution, display scale, UI theme, or application version.

[0062] In another embodiment, the electronic device (100) may use an artificial intelligence model in the process of mapping an abstract identifier corresponding to each of a plurality of UI elements to a position on the screen.

[0063] For example, when multiple UI elements of the same or similar form exist simultaneously on the screen, the electronic device (100) inputs strings displayed around each UI element, the type of adjacent UI element, the position on the menu hierarchy, the arrangement relationship within the screen, and the current lecture step into an artificial intelligence model and can calculate the probability that each candidate UI element corresponds to a specific abstract identifier. The electronic device (100) determines the candidate UI element whose probability is greater than or equal to a set threshold value as the UI element corresponding to the abstract identifier, and can store the coordinates of the determined UI element in a location registry.

[0064] In another embodiment, the electronic device (100) may use a natural language processing-based artificial intelligence model in the process of replacing text tokens with strings of the target language.

[0065] For example, if multiple candidate strings correspond to the same abstract identifier, the electronic device (100) can determine a target string suitable for the current context among the multiple candidate strings by inputting the subtitle data of the current step, the content of the previous step and the next step, the type of target application, the target language, and the context in which the corresponding UI element is used into an artificial intelligence model. Additionally, if there is no corresponding string in the locale data, the artificial intelligence model may be configured to generate or recommend an alternative string of the target language based on strings of other languages, functional information of UI elements, and surrounding text.

[0066] In another embodiment, the electronic device (100) can look up data corresponding to lectures, chapters, and steps, and then use an artificial intelligence model to adjust the order or availability of steps to be provided to the user.

[0067] For example, the electronic device (100) inputs the user's previous lecture history, the time taken to perform each step, the number of incorrect inputs, the number of repeated searches for the same UI element, and the lecture progress rate into an artificial intelligence model, and can identify the proficiency or learning status of each user. Subsequently, the electronic device (100) can adjust the lecture progression order so as to skip some basic steps for users whose proficiency is higher than a set standard, or to provide additional relevant steps for users who repeatedly make errors in operating specific UI elements.

[0068] In another embodiment, the electronic device (100) may use an artificial intelligence model to determine the display position, size, or shape of the visual guidance.

[0069] For example, when multiple windows, popups, menus, or other UI elements are nested on a user screen, the electronic device (100) inputs the location, size, surrounding margins, location of adjacent objects, and complexity of the current screen of the target UI element into an artificial intelligence model and can determine a display location so that an arrow, highlight box, or highlight area does not obscure other important UI elements. Additionally, it may calculate a visibility score or an interference score for each of the multiple candidate display locations and select the candidate location with the lowest interference score and a visibility score above a set standard as the final display location.

[0070] In another embodiment, the electronic device (100) can analyze the user's operation state and adjust the timing or intensity of the visual guidance provided.

[0071] For example, the electronic device (100) inputs the mouse movement trajectory, click location, repetitive pointer movement in the same area, dwell time at a specific step, and user input after the previous guidance into an artificial intelligence model and can determine whether the user is currently perceiving the UI element normally or if additional guidance is needed.

[0072] For example, if it is determined that the user fails to select a target UI element for more than a set amount of time or repeatedly selects an incorrect UI element, the electronic device (100) may provide a highlight box, enlargement, or auxiliary text in addition to the existing arrow guidance.

[0073] Meanwhile, the artificial intelligence model described above may be learned in at least one of supervised learning, unsupervised learning, self-supervised learning, or reinforcement learning, and may include a neural network-based model, a convolutional neural network, a recurrent neural network, a transformer, a vision transformer, or a combination thereof. However, the specific types and learning methods of the artificial intelligence model are exemplary and are not limited thereto, and the electronic device (100) may use various forms of learning models or inference models to assist or enhance at least one of screen recognition, location registry creation, lecture-related data processing, string substitution, and visual guidance display according to the present disclosure.

[0075] FIG. 2 is a block diagram showing the components of an overlay-based software usage guide system according to one embodiment of the present invention.

[0076] Referring to FIG. 2, an overlay-based software usage guide system may include an electronic device (100), a user terminal (210), and at least one external server (220). The electronic device (100), the user terminal (210), and the external server (220) may transmit and receive data to and from each other via a wired communication network or a wireless communication network.

[0077] According to one embodiment, the electronic device (100) may be a computing device for performing the overlay-based software usage guidance method of the present invention.

[0078] The electronic device (100) may be implemented as a desktop computer, a notebook computer, a workstation, a tablet computer, or a device having equivalent computational processing capabilities. In particular, the electronic device (100) may run a target application that the user actually operates together with a guidance application according to the present invention, and may provide the execution screen of the target application and the information provided by the guidance application together in a single display environment.

[0079] The electronic device (100) may include a memory (110), a processor (120), a communication interface (130), and a display device (140). The memory (110) may store an operating system, a target application, a program for providing an overlay, user setting information, data regarding UI elements, and various data required for program execution. The processor (120) may control the overall operation of the electronic device (100) by executing at least one instruction stored in the memory (110). The communication interface (130) may transmit and receive data with a user terminal (210), an external server (220), or other external devices, and the display device (140) may provide the user interface of the target application and visual guidance superimposed thereon to the user.

[0080] The electronic device (100) may include various software modules for checking the execution status and screen status of a target application. For example, the electronic device (100) may include a module for capturing the screen of a target application, a module for recognizing UI elements displayed on the screen, a module for managing recognition results, a module for playing lecture data, and a module for generating a transparent overlay. These modules may be executed in a single process, but they may also be separated into multiple independent processes to consider processing load, stability, and the lifecycle of the program.

[0081] The electronic device (100) may check inputs that occur during the process of the user actually manipulating the target application. For example, the electronic device (100) may check at least one of the position of the mouse pointer, click input, keyboard input, the activation status of the target application, and the currently displayed lecture step. Accordingly, the electronic device (100) may determine whether the user has selected a specific UI element, completed the currently guided task, or whether the next guidance should be provided, and may change the progress status of the lecture in conjunction with the user's task progress.

[0082] The electronic device (100) is not necessarily limited to a device that performs all processing on its own. The electronic device (100) may perform processing requiring response speed locally, such as screen recognition or visual guidance rendering, and may perform processing such as storing lecture content, user authentication, learning history management, and data backup in conjunction with an external server (220). Additionally, some functions of the electronic device (100) may be deployed in a cloud computing environment, and the electronic device (100) may be implemented to receive processing results performed externally and reflect them on the user screen.

[0083] According to one embodiment, the user terminal (210) may be an electronic device that receives information related to a user using the service of the present invention or provides information necessary for using the service to the user.

[0084] The user terminal (210) may be implemented as a smartphone, tablet computer, laptop computer, desktop computer, smart display, or other electronic device capable of running a web browser. The user terminal (210) may be implemented by the same physical device as the electronic device (100), or it may be implemented as a separate device from the electronic device (100).

[0085] The user terminal (210) can receive various inputs related to the use of the service from the user. For example, the user can log in to a user account through the user terminal (210), select a target lecture to study among multiple lectures, or input a target application, desired language, learning stage, and user preference settings. Additionally, the user terminal (210) can provide information such as the title of the selected lecture, estimated time required, lecture progress rate, completed chapters, or a list of available lectures.

[0086] The user terminal (210) may provide an auxiliary interface regarding the progress of the lecture. For example, while the target application and transparent overlay are displayed on the display device (140) of the electronic device (100), the user terminal (210) may provide the progress status of the current lecture, additional explanation of the current step, an input interface for moving to the next step, or a control interface for pausing and ending the lecture. Accordingly, additional information necessary for the user can be provided on a separate screen while reducing the amount of information displayed on the work screen of the target application.

[0087] The user terminal (210) may also be used to collect user learning results or feedback. For example, the user terminal (210) may receive user input regarding whether a specific step is understood, the difficulty of the lecture, the appropriateness of the visual guidance, or whether an error occurred. The collected information may be transmitted to an electronic device (100) or an external server (220) and used for improving lecture content, analyzing the learning status of each user, or recommending subsequent lectures.

[0088] The user terminal (210) can communicate with the electronic device (100) in real-time or non-real-time. For example, when a user selects a target lecture from the user terminal (210), the selection result is transmitted to the electronic device (100) so that the lecture can be executed, and when a specific step of the lecture is completed in the electronic device (100), the progress information can be reflected in the user terminal (210). However, the user terminal (210) is not limited to a separate physical device essential to the present invention, and may be implemented in the form of a user interface module or a web-based interface installed in the electronic device (100).

[0089] According to one embodiment, the external server (220) may be a server device for providing a software usage guide service of the present invention to an electronic device (100) and a user terminal (210).

[0090] The external server (220) can be implemented as a single server, a server group composed of multiple physical servers, a cloud server, or a distributed computing environment. The external server (220) can store, manage, and provide data necessary for service provision while communicating with the electronic device (100) and the user terminal (210) through a network.

[0091] The external server (220) may include a database that stores various data related to lectures or may be connected to a database. For example, the external server (220) may store basic information per lecture, chapter information, step information, subtitle data, visual guidance data, UI element identification information, locale data, user account information, and lecture progress information. Depending on the type of data and management purpose, the data may be stored in a single integrated database or may be stored separately in multiple different databases or tables.

[0092] The external server (220) can provide data necessary for a target lecture in response to a request received from the electronic device (100). For example, when the electronic device (100) transmits a lookup request including a lecture identifier, the external server (220) can identify the chapters and steps belonging to the lecture and transmit corresponding data to the electronic device (100). At this time, the external server (220) may determine the range of data provided differently depending on the lecture access rights of the user account, the type of target application, the software version, or the user's language settings.

[0093] The external server (220) can manage the service usage history for each user. For example, the external server (220) can store lectures taken by the user, steps completed, the time taken for each step, the last place learned, and user feedback in conjunction with the user account. Accordingly, even if the user changes the electronic device (100) or stops and restarts the lecture, learning can resume from the point where it was stopped based on the previously saved progress information. In addition, this usage history can be used as basic data for analyzing the usage status of lecture content or for personalized lecture recommendations.

[0094] The external server (220) can additionally perform functions for the operation and maintenance of the service. For example, when a version update or UI change of the target application is detected, the external server (220) can update and provide the template image, UI element information, or locale data corresponding to the version to the electronic device (100). Accordingly, it is possible to respond to a new execution environment by updating server-side data without changing the entire program stored on each user device. In addition, the external server (220) can perform functions necessary for the stable operation of the service, such as user authentication, access rights management, data integrity verification, version management, and log storage.

[0095] As described above, an overlay-based software usage guidance system according to one embodiment of the present invention may be configured to identify the location of UI elements based on the screen of a target application actually used by the user, manage an abstract identifier and screen coordinates corresponding to the identified UI elements in conjunction, process subtitle data and visual guidance data included in lecture content according to the user's execution environment, and then provide guidance corresponding to the target UI elements by overlaying it on the actual work screen.

[0096] In addition, lecture-related data can be managed by classifying it into lecture, chapter, and step units, and necessary data can be retrieved, sorted, and assembled according to the target lecture selected by the user, and the locations of UI elements, strings included in subtitles, and display targets of visual guidance can be interconnected using the same abstract identifier.

[0097] Accordingly, guidance corresponding to the actual user screen can be provided without being dependent on specific screen coordinates, specific languages, or specific user environments, and the continuity and convenience of software usage education can be improved while reducing interruptions in the learning flow caused by screen transitions.

[0098] Furthermore, the electronic device (100) can perform user authentication, lecture selection, lecture progress information management, content updates, and user-specific learning status management by linking with a user terminal (210) and an external server (220), and may selectively use an artificial intelligence model for screen recognition, language processing, guidance location determination, or user behavior analysis as needed.

[0099] Meanwhile, the embodiments described above are exemplary for explaining the technical concept of the present invention, and the scope of the present invention is not limited to a specific application, a specific UI form, a specific lecture data structure, a specific communication method, or a specific visual guidance form; various modifications, substitutions, and combinations are possible within the scope of the technical concept of the present invention.

[0101] FIG. 3 is a flowchart of the operation of an overlay-based software usage guide method according to one embodiment of the present invention.

[0102] According to one embodiment, the components of an overlay-based software usage guide system can perform the operations disclosed in FIG. 3. For example, among the components included in the overlay-based software usage guide system, at least some of the components included in the electronic device (100) (e.g., memory (110), processor (120), communication interface (130), and display device (140) of FIG. 1) may be configured to perform the operations of FIG. 3.

[0103] In the following embodiments, the operations S310 to S340 may be performed sequentially, but are not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. Additionally, content corresponding to or overlapping with the above description in relation to FIG. 3 may be briefly explained or omitted.

[0104] According to one embodiment, the electronic device (100) acquires a user screen displaying a target application and can identify the location on the screen of at least one UI element included in the target application by comparing the user screen with a template image corresponding to each of a plurality of pre-registered UI elements (S310).

[0105] First, the processor (120) can obtain a user screen displaying a target application. Here, the target application may refer to an application that needs to be guided on how to use by the user, and may be, for example, a graphic editing program, a document creation program, a design program, a video editing program, a data analysis program, or another GUI (Graphical User Interface) based application that includes multiple menus and tools.

[0106] In one embodiment, the target application may be a graphic editing program such as Adobe Illustrator, but the present invention is not limited thereto. The user screen may be the entire execution screen of the target application, or a single window, monitor area, or a region of interest set within the screen where the target application is displayed.

[0107] In the present disclosure, "acquiring a user screen" may mean obtaining a display screen on which a target application is running as electronic data.

[0108] For example, if the user is using multiple monitors, the electronic device (100) may identify the monitor on which the target application is displayed and acquire the screen displayed on the monitor in the form of image data. In another example, the electronic device (100) may selectively acquire only the screen area where the execution window of the target application is located based on identification information of the active window.

[0109] By limiting the screen acquisition range in this way, the influence of screens displayed on other programs or other monitors unrelated to the target application on the UI element recognition process can be reduced.

[0110] Multiple pre-registered UI elements may refer to screen components that can be the subject of software usage guidance in the target application.

[0111] For example, UI elements may include menus, menu items, icons, buttons, tools, toolbar areas, input fields, checkboxes, dropdown menus, panels, tabs, selection objects, or other graphic objects capable of receiving user interaction.

[0112] For each UI element, at least one template image may be pre-registered along with an abstract identifier to identify the UI element.

[0113] For example, a template image including part or all of the tool icon corresponding to a specific tool in Illustrator may be registered, and in the case of a menu, a template image including an area displaying the menu name may be registered.

[0114] In one embodiment, multiple template images may be registered for a single UI element.

[0115] For example, even if it is the same "selection tool," the shape of the icon or the surrounding background may vary depending on the version of the target application, the language used, the operating system, the UI theme, or the display scale, so multiple template images corresponding to each environment can be associated with and stored under the same abstract identifier.

[0116] For example, the processor (120) can prioritize selecting a template image corresponding to the current execution environment. Accordingly, the function called "selection tool" itself can be maintained as a single abstract identifier, while distinguishing and applying images used in Korean versions, English versions, other software versions, or different display environments.

[0117] The processor (120) can compare the acquired user screen with a pre-registered template image to search for candidate regions within the user screen that correspond to the template image. Here, comparison does not mean merely determining whether the two images are completely identical, but may include a series of image comparison processes to determine whether a portion of the user screen has correspondence with the template image greater than or equal to a set standard.

[0118] For example, the processor (120) can calculate a normalized similarity value with a template image for each of the multiple candidate locations within the user screen, and identify a location where the similarity value is greater than or equal to a set threshold as a candidate location where the corresponding UI element exists.

[0119] At this time, the processor (120) can adjust the size of the template image and compare it after considering the user's display environment.

[0120] For example, in a lecture production environment, a template image is generated on a high-resolution display, but if the actual user uses a standard-scale display, the pixel size on the screen may differ even for the same UI elements.

[0121] Accordingly, the processor (120) can compare the display scale of the current user device with the reference scale at which the template image was created to enlarge or reduce the template image, and compare the resized template image with the user screen.

[0122] This is intended to reduce recognition failures that may occur when simply using the stored pixel size as is.

[0123] For example, it can be assumed that a user selects a lesson on using the "Rectangle Tool" in Illustrator. At the start of the lesson, the processor (120) obtains an image of the screen where Illustrator is currently displayed and can verify a pre-registered template image of the Rectangle Tool. Subsequently, the processor can compare a plurality of candidate locations, including the tool area on the left side of the user screen, with the template image of the Rectangle Tool and identify the screen area that exhibits the highest correspondence. As a result, if a specific area located at the bottom left of the user screen has a similarity greater than or equal to a set standard, the processor (120) can determine that area as the area where the UI element corresponding to the Rectangle Tool is located.

[0124] The "on-screen position" of a UI element is not limited to meaning only a single point. In one embodiment, the on-screen position may include the center coordinates of the UI element, and in another embodiment, it may include the top-left and bottom-right coordinates, center coordinates, width, height, or a combination thereof. For example, if a rectangular UI element occupies an area of ​​30 pixels in the x-direction and 30 pixels in the y-direction, not only the center coordinates of the element but also the width and height information can be identified together. This size information can subsequently be used to determine to which boundary of the UI element an arrow should approach or to determine the size of a highlight box.

[0125] Additionally, the processor (120) can convert pixel-unit coordinates obtained from a screen capture into logical coordinates used for actual overlay rendering. For example, if the ratio of actual physical pixels to logical coordinates used by the application differs due to the operating system or display settings, simply using the coordinates on the captured image as they are may cause the visual guidance to be misaligned with the actual UI elements. Therefore, the processor (120) can calculate logical coordinates usable in the user interface by correcting the physical coordinates according to the display scale and, if a system area such as a menu bar exists, additionally considering the height of that area.

[0126] Meanwhile, it is not necessary to recognize all UI elements every time a lecture starts. In one embodiment, the processor (120) checks a list of abstract identifiers corresponding to UI elements actually used in the currently selected target lecture and sets only the UI elements corresponding to that list as targets for recognition. For example, even if there are hundreds of menus and tools in the target application, if the current lecture uses only "File menu," "Export menu," and "OK button," only those three elements or related elements can be selectively explored. Accordingly, the amount of computation required for screen recognition can be reduced, and the possibility of misrecognizing UI elements unrelated to the current lecture can be reduced.

[0127] Screen recognition does not need to be performed continuously while the lecture is running. In one embodiment, screen recognition may be performed at the start of the lecture and may be performed again thereafter if a request for screen re-recognition is entered by the user or if the display environment changes.

[0128] For example, if a user moves the Illustrator window to another monitor, changes the display scale, or changes the position of a panel during a lecture, the coordinates of previously identified UI elements may differ from their actual positions.

[0129] In this case, the processor (120) can re-acquire the user screen and update the position of the UI element according to the user request or the set re-recognition condition.

[0130] According to one embodiment, the electronic device (100) can create a location registry by mapping an abstract identifier corresponding to each of at least one UI element and coordinates representing a location on the screen (S320).

[0131] Next, the processor (120) can create a location registry by mapping an abstract identifier corresponding to each of at least one UI element whose location is identified through the first operation (S310) and coordinates representing the on-screen location of the UI element.

[0132] Here, the location registry can be understood as intermediate mapping information that allows lecture content to reference UI elements on the actual user screen without directly storing specific screen coordinates.

[0133] In other words, instead of the lecture content directly specifying the location of a specific device, such as "point to x-coordinate 150, y-coordinate 90," it is made to have only an abstract identifier corresponding to "point to the file menu," and the location of the actual file menu on the current screen is determined through the location registry.

[0134] In the present disclosure, "abstract identifier" may be identification information assigned to identify the same function or UI element independently of the environment, rather than the UI name of a specific language or specific screen coordinates itself.

[0135] For example, instead of using the Korean string "파일" or the English string "File" itself as an identifier, identification information such as topmenu_file, which corresponds to both, can be used. Accordingly, the lecture content can maintain the same abstract identifier whether the target application is executed in Korean or in English.

[0136] A location registry can be implemented as a data structure containing, for example, a correspondence relationship of "abstract identifier → screen location information". For example, coordinates x=95, y=42 can be stored to correspond to an abstract identifier called topmenu_file, and coordinates x=34, y=420 can be stored to correspond to another abstract identifier called tool_rectangle. If necessary, the location information may include not only center coordinates but also the width, height, and perception confidence of UI elements.

[0137] For example, it can be assumed that a user is running Illustrator on a screen with a resolution of 1920X1080. If, in the first operation, it is determined that the "File menu" is located at coordinates (80, 35) in the upper left corner of the screen and the "Rectangle tool" is located at coordinates (42, 360), the processor (120) can map the abstract identifier of the file menu to (80, 35) and the abstract identifier of the rectangle tool to (42, 360). Subsequently, if it is necessary to point to the file menu at any step of the lesson, the coordinates corresponding to the abstract identifier of the file menu can be looked up in the location registry without analyzing the entire screen again.

[0138] By utilizing such a location registry, the creation of lecture content can be separated from the actual execution of the lecture. The creator of the lecture content does not need to know the user's monitor resolution, actual menu locations, panel layouts, or display scales in advance. The lecture content merely records abstract identifiers for the UI elements to be guided, and the actual locations can be obtained from the location registry generated at the time of lecture execution. Consequently, even if the same lecture content is provided to users with different screen environments, guidance can be displayed at locations appropriate for each user's actual screen.

[0139] In one embodiment, the location registry may be initially created at the start of a lecture and maintained throughout the lecture session. In this case, the coordinates of the abstract identifier referenced by the visual guidance data of the current step can be directly queried from the location registry without the need to repeat screen recognition every time each step changes. For example, even if there are 50 steps in a single lecture and the same "selection tool" is used repeatedly in multiple steps, the information from the previously created location registry can be used repeatedly without detecting the location of the selection tool anew for each step.

[0140] In another embodiment, the processor (120) may update at least a portion of the location registry when screen re-recognition is performed. For example, if a user requests screen re-recognition after moving a tool panel to the right side of the screen during a lecture, the coordinates of the tool panel and its internal UI elements stored in the existing location registry may be changed based on the new screen recognition result. On the other hand, the information of UI elements whose positions have not changed may be maintained. Accordingly, it is possible to selectively update only the positions of the changed UI elements without reconfiguring the data of the entire screen every time.

[0141] Additionally, the location registry may include or be merged with default location information to handle screen recognition failures. For example, if a specific UI element is obscured by another window and cannot be properly identified through comparison with a template image, pre-set default coordinates for that UI element may be used. In another example, actual detected coordinates may be used for UI elements that successfully recognize the screen, while default coordinates may be retained for UI elements that failed.

[0142] However, an implementation utilizing a default location may be suitable for environments where the UI structure of the target application is maintained within a certain range, and it may be implemented in a way that does not display the corresponding visual guidance if providing guidance to the default location is likely to point to an incorrect location. That is, if valid coordinates corresponding to a specific abstract identifier do not exist in the location registry, the rendering of the visual guidance referencing that identifier may be omitted. This approach may be used in situations where it is appropriate not to provide guidance to the user rather than displaying incorrect guidance at an uncertain location.

[0143] The location registry does not necessarily need to be permanently stored. In one embodiment, the location registry may be temporarily maintained in the memory of the electronic device (100) during the current lecture session and deleted when the lecture ends. This is because the user's screen layout may change when the next lecture is executed, even if the UI elements are the same. In another embodiment, recent recognition results may be temporarily stored and used as initial values ​​when the next lecture is started in the same execution environment. Therefore, the storage location and retention period of the location registry can be set in various ways according to the purpose of the present invention.

[0144] According to one embodiment, the electronic device (100) acquires lecture-related data including an abstract identifier, visual guidance data, a text token, and locale data, identifies a string corresponding to the abstract identifier included in the target language set for the user and the text token from the locale data, and can replace the text token with the identified string (S330).

[0145] Next, the processor (120) can obtain lecture-related data used for the progress of the lecture and user guidance.

[0146] Lecture-related data does not refer merely to simple subtitle strings, but can comprehensively include data that organizes the overall progression of the lecture and enables visual guidance and multilingual display.

[0147] For example, lecture-related data may include at least one of lecture content, chapter information, step information, subtitle data, visual guidance data, abstract identifier, text token, locale data, identification information of the target application, lecture sequence information, and step transition conditions.

[0148] In one embodiment, the lecture content is composed of multiple chapters, and each chapter may include multiple steps. Here, a step may be the minimum unit of a lecture for a user to perform a single continuous task or guidance. For example, a "lecture on creating a new document" may be composed of multiple steps such as "selecting the file menu," "selecting New," "setting the document size," and "selecting the create button." Each step may include caption data to be displayed to the user and visual guidance data that defines which UI element to highlight or point to in the current step.

[0149] In the present disclosure, a "text token" may be substitution target information used to represent a specific UI element, term, or other variable representation of a target application in subtitle data.

[0150] For example, subtitle data may not simply be stored as a complete Korean string like "Click the file menu," but may include specific parts in the form of tokens, such as "Click the {{menu:topmenu_file}} menu." In this case, topmenu_file can be an abstract identifier corresponding to the file menu, and the entire token can be replaced with an actual string suitable for the user's language environment at runtime.

[0151] The reason for using this text token structure is to separate the lecture content itself from the UI names of specific languages. For example, if a single lecture is provided to Korean, English, and Japanese users, it is possible to create separate full subtitle files for each language; however, managing recurring parts, such as UI menu names, as abstract identifiers allows for more efficient sharing of the same lecture structure. In particular, if the official UI names of the target application change or translations are modified, the changes can be reflected across multiple lectures by updating the locale data corresponding to the abstract identifiers, instead of individually modifying every lecture sentence.

[0152] In the present disclosure, "locale data" may be data that defines a correspondence relationship between each of a plurality of abstract identifiers and one or more language-specific strings. For example, for topmenu_file, the Korean string "File", the English string "File", and the string corresponding to Japanese may be mapped, respectively. The processor (120) may select the Korean string corresponding to topmenu_file when the target language set for the user is Korean, and select the English string corresponding to the same abstract identifier when the target language is English.

[0153] Here, the "target language set for the user" may be determined based on at least one of the service language registered in the user account, the UI language of the target application, the operating system language of the electronic device (100), or the language directly selected by the user when running the lecture. If necessary, the language used for the lecture explanation and the actual UI language of the target application may be set differently. For example, in the case of a user who uses the English version of the target application while providing Korean explanations, general subtitles may be displayed in Korean, but menu tokens may be replaced with actual English UI names. This is intended to allow the user to receive guidance that accurately corresponds to the menu names of the application they are actually viewing, while listening to the explanation in a language that is easy for the user to understand the lecture content.

[0154] For a specific example, it can be assumed that the subtitle data included in the current step of the lecture is configured as "Select {{menu:topmenu_file}} and then select Export". If the user uses an English UI, the processor (120) can extract an abstract identifier called topmenu_file included in the text token and identify a string "File" corresponding to the target language English in the locale data. Then, the original text token can be replaced with "File" to finally generate a guidance string corresponding to "Select File and then select Export". On the other hand, if the user uses a Korean UI, the same token in the same lecture content can be replaced with "File".

[0155] As such, a substitution structure using a single abstract identifier is not required to be used only for string conversion in subtitles. The same topmenu_file identifier may be used to select a template image for finding the file menu in the first operation (S310), to use as a key to identify the actual coordinates of the file menu in the location registry in the second operation (S320), and to determine a language-specific string in the third operation (S330).

[0156] In other words, a single semantic object called "File Menu" can be linked through the same abstract identifier at different processing stages of screen recognition, location management, and caption display.

[0157] In one embodiment, text tokens can be applied not only to menus but also to technical terms or shortcuts. For example, a token in the form of {{term:...}} can be used to determine the display method or supplementary explanation of technical terms, or a token in the form of {{shortcut:...}} can be used to display shortcuts suitable for the current operating system.

[0158] The processor (120) may perform an exception action to handle cases where a string corresponding to a specific abstract identifier cannot be found in the locale data. For example, when the corresponding string does not exist, the abstract identifier itself may be temporarily displayed, or a string of a pre-set standard language may be used instead. In another embodiment, the corresponding token may be displayed in its original state without being replaced so that the lecture itself is not interrupted. This method of processing is intended to prevent the entire lecture from not running due to the omission of specific translation data.

[0159] Additionally, locale data can be acquired once at the start of the lecture and maintained throughout the lecture session. Instead of having to repeatedly request the same locale data from an external server (220) for each step of the lecture, locale data corresponding to the language of the target lecture or target application can be maintained in memory (110) and referenced whenever the subtitles change. Accordingly, string replacement can be performed quickly even when multiple text tokens are used repeatedly.

[0160] According to one embodiment, the electronic device (100) can identify target coordinates corresponding to the abstract identifier included in the visual guidance data from the location registry and display visual guidance that matches the UI element corresponding to the abstract identifier based on the target coordinates through a transparent overlay placed on the target application (S340).

[0161] Next, the processor (120) can identify an abstract identifier included in the visual guidance data to be provided now, and identify the target coordinates corresponding to the abstract identifier in the location registry.

[0162] Here, visual guidance data may refer to display information intended to enable the user to intuitively identify the location of the task currently to be performed. Visual guidance may be implemented, for example, as arrows, circular highlights, rectangular highlight boxes, highlight borders, translucent or opaque masks, flashing indicators, pulse-shaped highlight effects, guidance text, or a combination thereof.

[0163] For example, if the current lecture step is a step that instructs the user to "select the rectangle tool," the visual guidance data for that step may include an abstract identifier representing the rectangle tool. The processor (120) can use this abstract identifier as a key in the location registry to determine the target coordinates where the rectangle tool is located on the current user screen. Then, by placing an arrow or highlight box based on those coordinates, the location where the user actually needs to click can be visually indicated.

[0164] "Matched visual guidance" may mean that guidance is not displayed at a fixed screen location in advance during the content creation stage, but rather is placed based on the location of the UI element actually detected on the user screen. For example, if a rectangle tool is located at x=40, y=300 on the lecture creator's screen but is located at x=55, y=430 on the user's screen, the processor (120) may display visual guidance based on the actual coordinates of the user screen stored in the location registry, rather than using the creator's original coordinates. Therefore, even if the screen layout differs for each user, the guidance can correspond to the actual UI element.

[0165] A transparent overlay may be a separate display layer superimposed on the screen where the target application is displayed. The background area of ​​the overlay may be configured to be transparent or translucent so that the user can continue to view the content of the target application, and visual objects such as arrows, circular markers, and explanatory text may be displayed only where actual guidance is required. Accordingly, the user can check the operation location on the current work screen itself without switching to another screen within the target application to view a separate lecture video.

[0166] In one embodiment, the overlay may be positioned to have a higher display priority than the target application. Therefore, even if the user opens another menu or panel of the target application, the visual guidance provided in the lecture can remain recognizable to the user. However, the entire area of ​​the overlay does not need to block user input. For example, mouse input through a transparent area where no guidance object exists can be transmitted to the target application, and the user can be configured to directly operate buttons, menus, or tools of the actual target application even while the overlay is displayed. Thus, the parallel execution of actual work and guidance on the same screen can have significant meaning in the educational method of the present invention.

[0167] Visual guidance data may include at least one of the following: an abstract identifier of the guidance target, the size of the guidance, an offset for adjusting the position relative to the guidance target, and the form or display state of the emphasis effect, along with the type of visual guidance. For example, even when pointing to the same UI element, depending on the purpose of the lecture, a circle may be displayed around the UI element, an arrow pointing from the center of the screen toward the element may be displayed, or a rectangular border enclosing the entire UI element may be displayed.

[0168] In one embodiment, when the visual guide is an arrow, the processor (120) may not simply set the center coordinates of the UI element as the endpoint of the arrow. This is because drawing the arrow to the center of the UI element may cause the arrow to obscure icons or menu text, or penetrate the UI element itself, thereby reducing visibility. Therefore, the processor (120) can determine the boundary of the UI element based on the target coordinates and the width and height of the UI element, and determine the direction in which the arrow faces the UI element and the point where the boundary meets.

[0169] For example, a target UI element can be considered as a single rectangular area, and a direction can be set from the center area of ​​the screen toward the center of the UI element. The processor (120) can determine which boundary among the upper, lower, left, or right boundary of the rectangle the direction first meets, and determine the position of the arrow endpoint based on the intersection point. At this time, the arrow endpoint may be set to exactly coincide with the intersection point, but may also be set to be located a certain distance outward from the intersection point for visibility.

[0170] For example, if the File menu is located at the top-left of the screen, an arrow pointing from the center of the screen toward the File menu can be positioned to point toward the right or bottom boundary of the menu. Conversely, if the target tool is located in the right panel of the screen, the arrow can be automatically positioned in the opposite direction. Therefore, even if the lecture creator does not individually specify positions for each UI element, such as "the arrow should appear from the left" or "the arrow should appear from the right," the appropriate direction can be determined based on the actual position of the UI element at the time of execution.

[0171] The processor (120) may also handle the case where there are no coordinates corresponding to the abstract identifier in the location registry so as not to display a visual guide at an incorrect location. For example, if the current step is set to point to a "specific button on the property panel" but the user closes the panel and the UI element is not detected, an arrow appearing at an arbitrary coordinate may confuse the user. Therefore, if valid coordinates are not confirmed, the processor (120) may omit the display of the visual guide or provide a separate guide requesting screen re-recognition.

[0172] A single step does not necessarily require only one visual guide. In one embodiment, a single step may include multiple visual guides, which may be provided simultaneously or sequentially. For example, when guiding a single action such as "Select the File menu and move to the Export menu," a first arrow may be displayed on the File menu first, and a second arrow may be displayed on the Export item once the user's operation is confirmed. In another example, a highlight circle may be displayed around a target button while simultaneously showing the name of the corresponding button in a subtitle area. Accordingly, step-by-step guidance corresponding to the user's actual sequence of operations is possible even within a single step.

[0173] Furthermore, the display of visual guidance and the display of lecture subtitles can be linked. For example, in the third action, the abstract identifier 'topmenu_file' is used to replace the text token of the subtitle with the string "File," while in the fourth action, the same abstract identifier 'topmenu_file' is used to query the coordinates of the actual file menu from the location registry. Therefore, the "File" in the text "Click the file menu" displayed to the user and the target pointed to by the arrow on the screen can be logically linked by the same abstract identifier. This reduces inconsistencies that can occur when different data are managed independently, such as "the subtitle describes the file menu, but the arrow points to a different menu."

[0174] The above-mentioned operation scenario is explained as follows.

[0175] When a user selects a target lesson on how to use Illustrator on an electronic device (100), the processor (120) can first obtain the screen where the current Illustrator is displayed. Then, the actual location of each can be verified by comparing the user screen with a template image corresponding to the file menu, rectangle tool, alignment panel, etc. used in the lesson. The processor (120) can create a location registry by linking the location of each verified UI element to the abstract identifier of that UI element. Next, the processor (120) obtains the lesson-related data of the target lesson and can replace text tokens included in subtitles with actual menu names according to the target language of the current user. Subsequently, if the use of the rectangle tool is required at the current step, the processor (120) can identify the abstract identifier of the rectangle tool in the visual guidance data, look up its actual coordinates in the location registry, and then display an arrow or highlight corresponding to the coordinates on a transparent overlay on the Illustrator screen. The user can proceed with the lesson while checking "which tool to press" on their actual Illustrator screen without moving to a separate lesson screen.

[0176] As described above, the four operations can form a series of interconnected structures, rather than being a simple set of functions listed independently of one another. This structure involves verifying the UI location of the actual user environment in the first operation (S310), connecting that location to an environment-independent abstract identifier in the second operation (S320), constructing a lecture expression corresponding to the user's language using the same abstract identifier in the third operation (S330), and providing guidance at the correct location on the actual screen again using the same abstract identifier in the fourth operation (S340). Accordingly, during the authoring stage of the lecture content, dependency on specific screen coordinates or specific UI languages ​​can be reduced, while during the actual execution stage, visual guidance and language-specific lecture content aligned with the user's screen environment can be provided.

[0177] Meanwhile, lecture-related data may include lecture content comprising multiple steps, and each of the multiple steps may include subtitle data and visual guidance data to guide a task to be performed using a target application, the subtitle data may include a text token containing an abstract identifier, and the visual guidance data may include an abstract identifier to specify a UI element where the visual guidance will be displayed.

[0178] In one embodiment, the lecture-related data may include lecture content comprising multiple steps. Here, the lecture content may refer to a set of data for providing the usage of a target application as a single continuous training course, and a single lecture content may include multiple work units that a user must perform sequentially.

[0179] In the present disclosure, "step" may refer to a unit for which guidance is provided in correspondence with a single task or process to be performed by a user using a target application. Accordingly, a step does not necessarily correspond only to a single mouse click, but may correspond to an task of selecting a menu, activating a specific tool, entering a setting value, executing a specific function, or a task that groups multiple related user operations into a single guidance unit.

[0180] For example, it can be assumed that the target lesson is about "how to create a new document and generate shapes in Illustrator." In this case, the lesson content may be composed of multiple steps, such as "a first step of selecting the file menu," "a second step of selecting the new document creation function," "a third step of setting the document size," "a fourth step of selecting the create button," "a fifth step of selecting the rectangle tool," and "a sixth step of creating a rectangle in the workspace." By dividing the lesson content into multiple steps corresponding to the sequence of tasks performed by the user in the actual software, the electronic device (100) can selectively provide guidance only on the tasks that the user currently needs to perform.

[0181] Each step may include at least subtitle data and visual guidance data. Subtitle data may be data intended to explain, in text form, the task the user must perform in that step. For example, content such as "Select the File menu," "Select the Rectangle tool and drag the work area," or "Select the OK button to apply the settings" may be provided to the user through subtitle data. Subtitle data may include not only simple explanatory sentences but also the purpose of the task, the sequence of operations, precautions, supplementary explanations, or conditions for proceeding to the next task.

[0182] Visual guidance data may be data intended to visually indicate where on the user screen the task described by the subtitles should be performed. For example, if the subtitle data includes the content "Select the rectangle tool," the visual guidance data included in the same step may designate the UI element corresponding to the rectangle tool as the guidance target. Accordingly, within a single step, the user can confirm "what to do" via text, while simultaneously confirming "where to do it" through arrows or highlights displayed on the user screen.

[0183] Such subtitle data and visual guidance data can be managed in a related state corresponding to the same step. That is, rather than the subtitle data and visual guidance data being provided independently at any point in time, they can be selected together based on a single step currently being executed. For example, if the fifth step is a step that guides the selection of a rectangle tool, the electronic device (100) can check the subtitle data of the fifth step and the visual guidance data of the fifth step together and provide them to the user at the same point in time of the lecture. Through this, a correspondence relationship can be maintained between the task explained in the subtitles and the UI elements highlighted on the screen.

[0184] In one embodiment, one subtitle data and one visual guidance data may correspond to a single step, but are not limited thereto. For example, if a single step includes multiple detailed operations, a single step may include multiple subtitle data or multiple visual guidance data. For example, in a single step such as "Select the File menu and then select the Export item," a first visual guidance corresponding to the File menu may be displayed first, and once the user's selection of the File menu is confirmed, a second visual guidance corresponding to the Export item may be displayed sequentially. In this case, each guidance may be provided according to an order set within a single step.

[0185] In addition, steps may include other data used for the progression of the lecture, in addition to subtitle data and visual guidance data. For example, a step may include step identification information, the display order of the steps, identification information of the chapter to which the lecture belongs, estimated execution time, video or audio materials, conditions for determining whether user operation has been completed, trigger information for transitioning to the next step, or auxiliary data to explain the operation of a specific UI. Therefore, the description that a step includes subtitle data and visual guidance data is not limited to the meaning that the step includes only these two types of data.

[0186] In one embodiment, multiple steps may be played according to a preset order within the lecture. For example, if steps 1 through 10 are defined sequentially, the user may proceed to Step 2 after performing the task in Step 1. However, not all steps necessarily proceed in a uniform sequential manner. A specific step may be switched to the next step by user input, another step may be switched automatically after a set time has elapsed, and yet another step may be switched to the next step when the selection of a specific UI element or the completion of a specific user operation is confirmed. Accordingly, a step-unit data structure can be used to provide the lecture not as a simple static document, but as step-by-step content linked to the user's actual work progress.

[0187] Additionally, multiple steps can be divided into one or more chapters. For example, there may be three chapters titled "Mastering Basic Tools," "Creating Shapes," and "Applying Colors," with multiple steps included within each chapter. This hierarchical structure not only clarifies the management and playback order of lecture content but can also be used to indicate how far a user has progressed in the entire course or to resume learning from a specific chapter.

[0188] The subtitle data of the present disclosure may include a text token containing an abstract identifier. The basic substitution principle of this part is as described in the description of FIG. 3 above and is therefore not repeated. However, it is significant that it clearly defines an inclusion relationship in which the text token is not independent information separated from the lecture content, but specifically can be included within the subtitle data of each step.

[0189] For example, if the subtitle data for Step 1 includes the data "Please select the {{menu:topmenu_file}} menu," then {{menu:topmenu_file}} can correspond to a text token. In this case, the abstract identifier 'topmenu_file' can be replaced with an actual string corresponding to the user's language at runtime. Therefore, rather than creating subtitles by fixing UI elements to specific languages ​​such as "File" or "File," the lecture creator can construct the subtitles for a step using an abstract identifier that represents the meaning of the corresponding UI element.

[0190] Additionally, visual guidance data of the same step may include an abstract identifier to specify the UI element where the visual guidance will be displayed. In the previous example, if the text token included in the subtitle data contains topmenu_file, the visual guidance data may also include an abstract identifier corresponding to topmenu_file. Then, the electronic device (100) can convert the identifier in the subtitles into a string such as "File" or "File" that is suitable for the user's language, and in the visual guidance, use the same identifier as a lookup key in the location registry to check the coordinates of the actual file menu.

[0191] In this way, since subtitle data and visual guidance data are included together in a single step and both can utilize the same abstract identifier, the linguistic and spatial guidance of a lecture can be linked to a single semantic object. For instance, this reduces inconsistencies that occur when subtitle data describes a file menu while visual guidance data specifies an edit menu. Furthermore, since the abstract identifier referenced by the lecture content can be maintained even if the display language or on-screen position of UI elements changes, the burden of having to repeatedly modify the lecture content itself for each environment can be reduced.

[0192] For example, it can be assumed that the user is using the English version of Illustrator and the current step is to guide the user to select the File menu. The subtitle data for that step may include a text token containing an abstract identifier corresponding to the File menu, and the visual guidance data may also include the same abstract identifier corresponding to the File menu. The electronic device (100) can replace the text token of the subtitle data with the string "File", while checking the location of the File menu on the actual user screen by looking up the abstract identifier of the visual guidance data in the location registry. Subsequently, it can provide a subtitle with the meaning "Please select the File menu" and an arrow pointing to the File menu.

[0193] Even if another user runs the same lecture in the Korean version of Illustrator, there is no need to change the lecture steps themselves. The electronic device (100) uses the same abstract identifier, but determines the string "file" on the subtitle side, and on the visual guide side, uses the actual coordinates of the file menu detected on the current user screen where the Korean UI is displayed. In this way, a single step structure can be reused in different execution environments.

[0194] Furthermore, locale data may include data in which multiple language-specific strings are mapped and stored corresponding to each of multiple abstract identifiers.

[0195] In one embodiment, locale data may include data in which multiple language-specific strings are mapped and stored corresponding to each of a plurality of abstract identifiers. Here, the locale data may be data for managing menu names, tool names, function names, or other strings that must be displayed to the user in a target application, by language. As described above, locale data may be used for substitution of text tokens, and the present embodiment defines a specific data structure in which such locale data has a correspondence relationship between a plurality of abstract identifiers and a plurality of language-specific strings.

[0196] For example, locale data may include a first abstract identifier corresponding to the File menu, a second abstract identifier corresponding to the Edit menu, and a third abstract identifier corresponding to the Object menu. Each abstract identifier may correspond to a string specific to the language, such as Korean, English, or Japanese. Accordingly, the same abstract identifier may correspond to the first string when the target language is Korean, to the second string when the target language is English, and to the third string when the target language is Japanese.

[0197] For example, assuming that the abstract identifier corresponding to the file menu is topmenu_file, the locale data may conceptually store strings corresponding to topmenu_file, such as “File” in Korean, “File” in English, and other supported languages. Similarly, the abstract identifier topmenu_edit corresponding to the edit menu may correspond to “Edit” in Korean, “Edit” in English, etc. Here, the specific identifier names or data storage formats are exemplary, and the locale data of the present invention is not limited to specific names or specific data formats.

[0198] Managing multiple language-specific strings by associating them with a single abstract identifier in locale data is intended to avoid treating UI elements with the same meaning as separate objects for each language. For example, if “File” and “File” are registered as separate UI elements, different identifiers may need to be used for each language in the lecture content as well. On the other hand, in the present embodiment, a single abstract identifier corresponding to the meaning of the UI element is maintained, and only the actual display string can be selected differently depending on the target language. Therefore, it is possible to respond to different UI language environments without duplicating the structure of the lecture content for each language.

[0199] In one embodiment, the processor (120) can identify a string in locale data using the target language set for the user and an abstract identifier included in the text token. For example, if the target language is English and the text token contains an abstract identifier called topmenu_file, the processor (120) can identify a string of “File” corresponding to the combination of topmenu_file and English in the locale data. Then, the string can be inserted into the subtitle area where the text token is located to construct the final subtitle to be provided to the user.

[0200] Locale data may be managed separately by target application. This is because identical or similar functions may be expressed by different official names in different applications. For example, if an abstract identifier refers to a function of a specific graphics editing program, the language-specific UI name officially used by that application may be stored. Since different target applications may use different strings even if they have the same functional meaning, the final string may be determined by using application identification information, abstract identifiers, and language information together.

[0201] In addition, locale data can be distinguished based on the version of the target application. For example, if the name of an existing menu changes as specific software is updated, different strings can be assigned to the same abstract identifier for each version. Accordingly, the abstract identifier used in the lecture content can be maintained while displaying the official menu name suitable for the application version used by the actual user.

[0202] In one embodiment, locale data is stored in an external server (220) and can be provided to an electronic device (100) at the time when the target lecture starts. For example, the electronic device (100) transmits information regarding the language set for the user, the type of the target application, or the version of the target application to the external server (220), and the external server (220) can return locale data corresponding thereto. In another embodiment, all locale data corresponding to multiple languages ​​is stored in advance in the electronic device (100), and the processor (120) can select and use only the information of the language required when executing the lecture.

[0203] In addition, some or all of the locale data may be retained in memory (110) during the lecture session. Since the same UI element may appear repeatedly in multiple steps of a single lecture, it may be unnecessary to repeatedly query the same locale data from an external server (220) whenever a step changes. Therefore, the processor (120) may obtain the locale data required at the start of the lecture, store it in memory (110), and then refer to the stored locale data when a text token is identified in each step.

[0204] For example, it can be assumed that the file menu is used repeatedly in 7 steps out of 30 steps of the target lecture. After the processor (120) first obtains the locale data, it can repeatedly check the corresponding string in the locale data maintained in memory (110) without requesting the server anew for “what the English name of the file menu is” at every step. Accordingly, network traffic can be reduced and subtitle display delay can be reduced.

[0205] Situations where the target language string of a specific abstract identifier does not exist in the locale data can also be considered. For example, there may be cases where a new UI element is added to the lecture content, but the Japanese string of that UI element has not yet been registered in the locale data. In this case, the processor (120) can use the string of the set alternative language, display a default string available corresponding to the abstract identifier, or maintain the text token in its original form. Through such alternative processing, the execution of the entire lecture can be prevented even if some locale information is missing.

[0206] In one embodiment, locale data may be configured to be updateable. When a UI name or a translation expression changes due to an update of the target application, the corresponding string stored in the external server (220) may be changed. At this time, the changes can be applied to multiple lectures and multiple steps using the abstract identifier by updating the locale data corresponding to the specific abstract identifier without having to directly modify all subtitles included in each lecture content itself.

[0207] For example, it can be assumed that a menu name has been changed from “Export Settings” to “Export Options” due to a software update. If the corresponding English string is directly written in the lecture content, it may be necessary to find and modify all lectures that reference that menu. On the other hand, if the function is managed by a single abstract identifier as in the present embodiment, the changed name can be reflected in multiple lecture contents by changing the English string corresponding to that identifier to “Export Options.”

[0208] Furthermore, the information stored in the locale data is not necessarily limited to only one display string. In one embodiment, a basic display string, an abbreviated display string, a descriptive string, or auxiliary term information may be stored together in correspondence with one abstract identifier. The electronic device (100) may select and use at least one of these according to the display area of ​​the subtitle, the form of guidance, or user settings.

[0209] Ultimately, according to the above-described embodiment, subtitle data regarding “what task is explained” and visual guidance data regarding “which UI element is guided” can be linked on a step-by-step basis in each step of the lecture content, and locale data can be managed separately by mapping multiple language-specific strings to abstract identifiers used in these data. Accordingly, UI names suitable for different display languages ​​for each user can be provided while maintaining the logical task structure of the lecture, and changes or additions to strings of a specific language can be handled without remaking the entire lecture content.

[0211] Meanwhile, the operation of acquiring lecture-related data may include the operation of verifying a lecture identifier corresponding to a target lecture selected by a user, the operation of querying data corresponding to lectures, chapters, and steps stored in a remote database using the lecture identifier, and the operation of sorting the steps based on the order of the retrieved chapters and the order of the steps included in each chapter, and configuring and acquiring the lecture-related data to include the sorted steps.

[0212] In one embodiment, the electronic device (100) can identify a lecture identifier corresponding to a target lecture selected by a user, query data corresponding to a lecture, chapter, and step stored in a remote database using the lecture identifier, and then construct and obtain lecture-related data based on the queried data. Here, the lecture identifier is information for distinguishing a specific lecture selected by the user from other lectures among a plurality of lectures, and may include numbers, strings, combinations of characters and numbers, or other forms of identification information that can uniquely distinguish a lecture on a server.

[0213] For example, it can be assumed that multiple courses such as “Basic Shape Creation,” “Logo Creation,” “Path Editing,” and “Color Application” of Illustrator are registered on an external server (220). When a user selects the “Basic Shape Creation” course through the user interface, the electronic device (100) can identify the course identifier assigned to the selected course. Subsequently, the electronic device (100) can request related data stored on the external server (220) or a remote database by using the course identifier as a search condition.

[0214] Course identifiers can be used as reference information to link a user's course selection with actual course data in a remote database. Accordingly, even if strings displayed to the user, such as the course title, remain the same or change, the correct course data can be identified using the course identifier. For example, even if the course title changes during operation, maintaining the same course identifier internally allows the connection with the existing user's enrollment history or course data to be preserved.

[0215] In a remote database, information regarding a single target lecture may be stored as a single data object, but in one embodiment, lectures, chapters, and steps may be stored in a data structure that is separated from one another. For example, lecture data may include the title of the target lecture, the target application, the applicable version, the estimated learning time, and setting information regarding the lecture as a whole; chapter data may include identification information for each chapter, the chapter title, and the order of the chapters; and step data may include identification information for each step, the corresponding chapter, the order within the chapter, subtitle data, and visual guidance data.

[0216] Storing lecture data by dividing it into multiple layers in this manner makes it easier to add, modify, and change the order of content compared to managing the entire lecture as a single large-capacity dataset. For example, if it is necessary to modify only the subtitles included in Step 5 of Chapter 3 within a single lecture, the data corresponding to that step can be modified without recreating the entire lecture. Furthermore, an extended form can be considered where the same chapter structure or parts of the steps are utilized in the composition of other lecture content.

[0217] The electronic device (100) can use a lecture identifier to query not only lecture data corresponding to a target lecture, but also multiple chapter and step data belonging to the lecture. For example, if the lecture identifier is COURSE_A, it can query chapters set with the lecture identifier COURSE_A, and then query steps corresponding to the lecture or each chapter. Such querying may be performed sequentially, or data that can be queried independently may be queried in parallel to reduce data acquisition time.

[0218] At this time, the order retrieved from the database does not necessarily match the order to be used in the actual lecture. This is because the database may return the results based on the storage time, data identifier, or internal search method. Therefore, the electronic device (100) can determine the actual lecture progression order using order information separately set for the retrieved chapters and steps.

[0219] In one embodiment, each chapter may include chapter order information indicating the chapter order in the target lecture, and each step may include step order information indicating the step order within the corresponding chapter. For example, if there is a first chapter and a second chapter, there may be Step 1, Step 2, and Step 3 in the first chapter, and there may again be Step 1, Step 2, and Step 3 in the second chapter. Therefore, if all steps are sorted using only step order information, it may be difficult to accurately determine the overall lecture order of steps included in different chapters.

[0220] Accordingly, the electronic device (100) can first check the order of the chapter to which each step belongs, and among steps belonging to the same chapter, additionally check the internal chapter order of the step to arrange multiple steps. That is, the order of the chapter can be applied as a first criterion, and the order of the step within the chapter can be applied as a second criterion. Through this two-stage arrangement, the overall progression order of the lecture can be stably determined.

[0221] For example, it can be assumed that the order of Chapter A is 1 and the order of Chapter B is 2, and that each chapter contains steps 1 through 3. The electronic device (100) may first arrange steps 1, 2, and 3 of Chapter A, and then arrange steps 1, 2, and 3 of Chapter B. Alternatively, if sorting is performed based only on the step order information of each step, steps 1 of Chapter A and steps 1 of Chapter B may be placed adjacent to each other, which may distort the actual order of the lectures.

[0222] Therefore, the sorting process according to the present embodiment can be understood not merely as listing data retrieved from a database, but as a process of converting separately stored hierarchical data into an order suitable for actual lecture playback.

[0223] The electronic device (100) can organize lecture-related data to include a plurality of aligned steps. Here, “organizing” is not limited to listing each raw data obtained from a remote database as is, but may include organizing the data into a single lecture execution data by linking the data in a form necessary for the execution of the target lecture.

[0224] For example, the electronic device (100) may include basic information of the target lecture, sorted chapter information, and sorted steps in a single lecture-related data. Additionally, each step may include subtitle data and visual guidance data, as well as a list of UI elements requiring screen recognition in the current target lecture, lecture progress conditions, or information to be displayed after the lecture is completed.

[0225] In one embodiment, lecture-related data can be configured entirely at the start of the lecture. For example, when a user selects a target lecture, the electronic device (100) retrieves and sorts all chapters and steps required for the target lecture, and the sorted steps can be used in sequence during the course of the lecture. According to this method, pre-configured lecture-related data can be used without requesting data again from a remote database whenever each step changes.

[0226] In other embodiments, only a portion of the lecture-related data may be acquired first, and the remaining data may be acquired additionally as the lecture progresses. For example, only the basic information of the entire lecture and the steps of the current chapter may be acquired first, and the steps of the corresponding chapter may be acquired additionally before the user accesses the next chapter. Therefore, the description "acquiring lecture-related data by configuring" is not necessarily limited to a method of downloading all lecture data at a single point in time.

[0227] Additionally, the electronic device (100) may configure lecture-related data by taking into account the user's access rights or lecture usage status. For example, if the steps provided to free users and the steps provided to paid users are different, only the steps available to the user may be included in the lecture-related data based on permission information provided by an external server (220). In another example, the device may be configured to check information on chapters previously completed by the user and start the lecture from the step after the last learning point.

[0228] In one embodiment, if a specific step is deleted or set to an inactive state, that step may be excluded from the lecture-related data. For example, if a content manager temporarily disables a specific step for which an error was found, that step may not be included in the lecture-related data provided to actual users, even if the data for that step remains in the database.

[0229] This method of organizing lecture-related data can also be advantageous for separating the lecture production environment from the user execution environment. The content creator creates each lecture, chapter, and step in manageable units, and the actual user-side electronic device (100) can retrieve them at the time of execution and assemble them according to the actual lecture playback order. Therefore, even if content modification or a change in the chapter order occurs, the changes can be reflected through data updates from an external server (220) without reinstalling the entire lecture program on the user terminal.

[0230] To illustrate a specific scenario, if a user selects “Illustrator Basics Course,” the electronic device (100) can identify the course identifier of the corresponding course. The electronic device (100) can retrieve basic course information corresponding to the course identifier, as well as chapter data such as “Understanding Basic Screens,” “Creating Shapes,” and “Applying Colors.” Additionally, it can retrieve multiple steps included in each chapter together.

[0231] The electronic device (100) confirms that the retrieved chapter is set in the order of “understanding the basic screen → creating shapes → applying colors” and can arrange multiple steps according to the set step order within each chapter. Subsequently, lecture-related data is configured to include basic information, chapter information, and arranged steps of the target lecture, and when the lecture starts, the arranged steps are called sequentially to provide subtitle data and visual guidance data for each step to the user.

[0232] According to this embodiment, instead of providing raw lecture data stored in a database to the user as is, it can be reconstructed into actual executable lecture data based on the hierarchical structure and progression order of the target lecture, so that multiple lectures and multiple steps can be managed consistently while providing content to the actual user according to a predetermined lecture flow.

[0233] Additionally, the visual guidance according to the present disclosure may include an arrow pointing to a UI element. For example, the operation of displaying the visual guidance may include identifying a point where the direction of the arrow toward the UI element intersects at the boundary of the UI element based on target coordinates and the size of the UI element, and determining a position spaced apart from the intersecting point by a set interval as the endpoint of the arrow.

[0234] In one embodiment, the visual guide may include an arrow pointing to a target UI element. The arrow is a visual object intended to allow the user to intuitively recognize the location of the UI element they currently need to select or manipulate, and it may be displayed on a transparent overlay placed on the screen of the target application. The location of the target UI element can be verified through a location registry, and this embodiment describes a specific method for determining the actual endpoint of the arrow using the verified location and the size of the UI element.

[0235] Simply extending an arrow to the center coordinates of a UI element can cause visual problems in the actual user environment. For example, if the target UI element is a small icon, the tip of the arrow may obscure the icon itself, making it difficult for the user to identify its shape; if the target UI element is a menu or a long button, the arrow may pass through the interior of the UI element and obscure text or other information. Therefore, in this embodiment, the position of the arrow endpoint can be determined by considering not only the center of the target UI element but also the size of the area actually occupied by the UI element.

[0236] The target coordinates may be coordinates representing the center position of the target UI element or the reference position of the target UI element. Additionally, the size of the UI element may include the width and height of the UI element. The electronic device (100) can determine the boundary area occupied by the target UI element on the user screen based on the target coordinates, width, and height.

[0237] For example, it can be assumed that the center coordinates of the target UI element are (500, 300), the width of the UI element is 40, and the height is 30. In this case, the electronic device (100) can determine that the target UI element occupies a range of 20 each to the left and right and 15 each to the top and bottom relative to the center coordinates. Therefore, the boundary of the target UI element can be conceptually represented as a rectangular area defined by x=480 to x=520 and y=285 to y=315.

[0238] However, the boundary of the target UI element does not necessarily have to be exactly the same as the visual shape of a rectangle. In one embodiment, the rectangular area surrounding the UI element can be defined as the boundary using the bounding box of the UI element obtained during the screen recognition process. For example, even if it is a circular button, a complex icon, or a menu consisting of text, the arrow position can be calculated based on the bounding box surrounding the area occupied by the UI element on the screen.

[0239] The electronic device (100) can determine the direction in which an arrow is directed toward a target UI element. Here, the direction can be determined based on the relative positional relationship between the reference position where the arrow starts and the target UI element. For example, the center area of ​​the screen can be used as a reference point to determine the placement direction of the arrow, and the direction from the reference point toward the center of the target UI element can be set as the direction in which the arrow approaches the UI element.

[0240] For example, if the target UI element is located at the top right of the screen center, the electronic device (100) can determine the direction toward the target UI element from the screen center toward the top right. Conversely, if the target UI element is located at the bottom left of the screen center, an arrow can be positioned in the corresponding opposite direction. Accordingly, the direction of the arrow can be determined based on the actual position of the target UI element on the screen, even without the instructor individually setting which direction the arrow should approach for every UI element.

[0241] The electronic device (100) can identify the point where the direction of the arrow intersects the boundary of the target UI element. Here, the intersecting point can be understood as the location where the arrow first meets the boundary of the target UI element, assuming that the arrow is approaching the target UI element.

[0242] For example, if a target UI element is positioned on the right side of the screen and an arrow approaches the target UI element from the left to the right, an intersection point may be formed at the left boundary of the target UI element. If a target UI element is located at the bottom of the screen and an arrow approaches from the top to the bottom, an intersection point may be formed at the top boundary of the target UI element. If a target UI element is positioned diagonally from the center of the screen, an intersection point with one of the top, bottom, left, or right boundaries of the target UI element can be determined based on the direction of the arrow's approach.

[0243] Accordingly, even if the screen position and size of UI elements change, the intersection position can be determined by the same principle.

[0244] However, the electronic device (100) may not set the endpoint of the arrow exactly the same as the intersection point. This is because if the endpoint of the arrow touches the boundary of the UI element, the arrowhead may obscure the boundary of the UI element, or the user may mistake the arrow and the UI element for a single graphic object. Therefore, in this embodiment, a position spaced apart from the intersection point by a set interval may be determined as the endpoint of the arrow.

[0245] Here, "set spacing" may refer to the visual margin to be secured between the endpoint of the arrow and the target UI element. The set spacing may be defined, for example, as a distance in pixels, a ratio to the screen size, or a distance determined in correspondence with the size of the UI element. In one embodiment, the same spacing may be applied to all UI elements, while in another embodiment, different spacings may be applied depending on the size of the UI element or the size of the arrow.

[0246] For example, if the intersection point is located at (480, 300) on the left boundary of the target UI element and the set spacing is 5 pixels, and the arrow points from left to right toward the target UI element, the arrow endpoint can be set to a location 5 pixels outward from the intersection point toward the UI element, near (475, 300). Accordingly, the arrowhead can clearly point toward the UI element without directly covering it.

[0247] The electronic device (100) can also determine the starting point of the arrow based on the determined endpoint of the arrow. For example, if the length of the arrow is pre-set, the starting point can be determined by moving the set length from the endpoint in the opposite direction of the arrow's progress. In this case, even if the position on the screen of the target UI element changes, the arrow can be positioned to face the target UI element while maintaining the length of the arrow itself at a constant level.

[0248] In another embodiment, the length of the arrow may be adjusted according to the available space on the screen. For example, if the target UI element is located close to the edge of the screen and it is difficult to apply a constant arrow length, the arrow length may be reduced. Conversely, if there is sufficient display space around the target UI element, a relatively long arrow may be provided.

[0249] Additionally, when determining the display position of the arrow, overlap with other visual guides or user interfaces may be considered. For example, although the arrow is generated by default from the center of the screen toward the target UI element, the starting position or approach direction of the arrow may be changed if an important subtitle area or other guide object exists along that path. However, such additional avoidance processing is not strictly required for the configuration of this embodiment and may be applied as an optional embodiment.

[0250] As an example of a specific usage scenario, the user may enter a lecture step in which they must select the rectangle tool located in the left tool panel in Illustrator. The electronic device (100) can check the target coordinates corresponding to the rectangle tool and the width and height of the tool icon in the location registry.

[0251] The electronic device (100) can determine the approach direction of the arrow based on the center of the rectangular tool and a reference point on the screen, and can identify the location where the direction intersects the boundary of the rectangular tool. Then, a location spaced outward from the rectangular tool by a set distance from the intersection point can be set as the end point of the arrow, and a location spaced in the opposite direction by a set length can be set as the starting point of the arrow.

[0252] As a result, the arrow can be displayed pointing toward the rectangle tool, terminating at the outer edge with a certain margin rather than covering the icon or penetrating its center. Users can intuitively identify the tool they need to select by following the direction the arrow points.

[0253] As another example, consider a case where the same lecture content is provided to users with different screen resolutions. Even if the first user's rectangle tool is located at the top-left and the second user's rectangle tool is located at the bottom-left, the start and end positions of the arrows can be re-determined based on the target coordinates identified on each user's screen and the size of the UI element, rather than being fixed during the content creation phase. Therefore, it is possible to provide arrow placement suitable for different user screens while utilizing the same visual guidance data.

[0254] In addition, the arrow can automatically adapt even when the size of the UI element is changed. For example, if the size of a specific button increases from 30x30 to 45x45 due to a change in the display scale, the arrow may go inside the button if the arrow endpoint is fixed based only on the center coordinates. On the other hand, in the method according to the present embodiment, a new boundary is identified from the size of the changed UI element and the intersection point with that boundary is re-determined, so the arrow can be displayed at an appropriate distance from the outside of the UI element even after the size change.

[0255] In such an embodiment, the method goes beyond merely "displaying an arrow at the target coordinates" and identifies the boundaries of the UI elements based on the actual location and size of the detected UI elements. It then calculates the point where the approach direction of the arrow intersects the corresponding boundary, and determines a location spaced apart from that intersection point by a set interval as the endpoint of the arrow. Accordingly, even if the screen configuration, size, or location of the UI elements varies from user to user, the arrow can be aligned to point from the outside of the corresponding UI element toward the target UI element, thereby reducing the phenomenon where the visual guide obscures the UI element itself.

[0257] In another embodiment, the processor (120) can analyze the user's operation status during the lecture and determine whether to automatically perform screen re-sync for the user screen.

[0258] Below, embodiments related to screen re-recognition will be described in detail.

[0259] The above guide on how to use the overlay-based software is,

[0260] The operation of the processor to calculate a screen re-recognition trigger index based on the user operation state at the current step and the UI element detection result at the time of creation of the location registry, based on the following mathematical formula 1; and

[0261] The processor further includes the operation of, when the screen re-recognition trigger index is greater than or equal to a set threshold, re-acquiring the user screen to re-identify the on-screen location of at least one UI element included in the target application, and updating at least a portion of the location registry based on the re-identified on-screen location.

[0262]

[0263] The above I_resync is a screen re-recognition trigger index indicating the degree of need for re-recognition of the user screen in the current step; the above T_dwell is the user's dwell time measured from the entry point of the current step; the above T_ref is a reference dwell time set to normalize the dwell time; the above D_mouse is the cumulative movement distance of the mouse pointer measured after the entry of the current step; the above D_ref is a reference mouse movement distance set to normalize the cumulative movement distance; the above N_err is the number of user inputs for an area that does not correspond to the UI element subject to the visual guidance in the current step; the above N_ref is a reference number of incorrect inputs set to normalize the number of user inputs; the above C_init is an initial detection confidence level of 0 or greater and 1 or less calculated for the UI element subject to the visual guidance when the location registry is created; the above α is a first weight for setting the degree of influence that an increase in the dwell time has on the screen re-recognition trigger index; and the above β is the increase in the cumulative movement distance of the mouse pointer that the screen re-recognition A second weight for setting the degree of influence on the activation index, wherein γ is a third weight for setting the degree of influence of an increase in the number of user inputs on the screen re-recognition activation index, wherein p is a first exponential coefficient for setting the degree to which the screen re-recognition activation index increases non-linearly as the number of user inputs increases, wherein δ is a fourth weight for setting the degree to which a decrease in the initial detection reliability affects the screen re-recognition activation index, and wherein q is a second exponential coefficient for setting the degree to which the screen re-recognition activation index increases non-linearly as the initial detection reliability decreases.The above η is an interaction weight for reflecting the combined effect in the screen re-recognition trigger index when the number of user inputs increases while the initial detection reliability decreases simultaneously, the above I_th is a threshold value set to determine whether the screen re-recognition process is automatically executed, and the above α, β, γ, δ, and η are each set to a value greater than 0, and the above p and q are each set to a value greater than 1.

[0264] The processor (120) can quantitatively determine whether screen re-recognition is necessary for the current user screen by analyzing the user's operation state during the lecture and the detection state of UI elements identified at the time of creation of the location registry. In this embodiment, the screen re-recognition trigger index is not determined solely based on whether the user stays at a specific step for a long time, but can be calculated by using the dwell time at the current step, the cumulative movement distance of the mouse pointer, the number of user inputs in areas outside the target UI element, and the initial detection reliability together. Accordingly, it is possible to distinguish and evaluate situations where the dwell time temporarily increases or the mouse is moved a lot while the user is performing the lecture normally, and situations where the user cannot actually find the UI element to be guided or where there is a possibility that the coordinates stored in the location registry are inconsistent with the current user screen.

[0265] In the present disclosure, the screen re-recognition trigger index may be a value that quantitatively indicates the degree of need for re-recognition of the user screen at the current step. An increase in the value of the screen re-recognition trigger index may mean that the need to re-recognize the user screen and update the location information is increasing, rather than continuing to use the location information of UI elements stored in the current location registry. Conversely, if the screen re-recognition trigger index is lower than a set threshold, it may be determined that the user's current operation behavior falls within the range that can occur during a normal learning process, or that the location information of UI elements stored in the existing location registry remains available.

[0266] The processor (120) can determine the user's dwell time measured from the time of entry into the current step in order to calculate the screen re-recognition trigger index. Here, the dwell time may refer to the time elapsed from the time the user entered the current step to the present time. For example, the dwell time can be obtained by starting a timer at the time when the second application (421) activates a new step and calculating the difference between the current time and the step start time. In another embodiment, the time during which no user input occurs may be separately excluded, or the time during which the target application is inactive may be excluded, so that only the time during which the actual user was performing the step is calculated as the dwell time.

[0267] Dwell time can be used as behavioral information to indicate whether the user is having difficulty understanding the current guidance and navigating target UI elements. For example, under normal circumstances, it is expected that the user will select a target UI element within a set time after entering a specific step; however, if the user remains on the same step for a significant amount of time, there is an increased possibility that the location of the currently displayed visual guidance does not match the actual UI element, or that the user is unable to find the UI element. However, since dwell time can also increase when the user works slowly to familiarize themselves with the lecture content, in this embodiment, screen re-recognition is not performed based solely on dwell time, but can be used in conjunction with other behavioral information.

[0268] The processor (120) can check a reference dwell time to prevent the absolute value of the dwell time from being interpreted differently for each step. The reference dwell time may be the time expected to be required to perform the corresponding step normally, or a time pre-set to normalize the dwell time. For example, the reference dwell time may be set short for a step requiring a simple button click, and a relatively long reference dwell time may be set for a step that sets multiple input values ​​or performs an editing operation.

[0269] The reference dwell time may be pre-set by the content creator during the production process of the lecture content, or it may be determined based on the past execution times of multiple users who have performed the same or similar steps. For example, an external server (220) may store statistical information regarding the dwell times of users who have successfully completed a specific step, and determine the reference dwell time based on an average value, a median value, or a set percentile value. The electronic device (100) may receive the reference dwell time corresponding to the step together during the process of acquiring lecture-related data.

[0270] The processor (120) may use a value normalized by dividing the dwell time by the reference dwell time, and a logarithmic function may be applied to this normalized dwell time in the mathematical formula. By using a logarithmic function, the screen re-recognition trigger index increases as the dwell time increases above the reference dwell time, but it is possible to prevent the effect from increasing indefinitely with the same slope as the dwell time continues to increase. For example, since even a normal user may stay at a certain time step to reread an explanation or check the results of a task, the need for screen re-recognition can be prevented from increasing excessively merely because the dwell time has increased slightly.

[0271] The processor (120) can additionally check the cumulative distance the mouse pointer has moved since entering the current step. The cumulative distance can be calculated using continuous pointer position information obtained from a user input device. For example, the processor (120) can calculate the distance between a first mouse coordinate and a second mouse coordinate that are temporally consecutive, and obtain the cumulative distance of the mouse pointer by accumulating multiple distances calculated during the current step.

[0272] The cumulative movement distance of the mouse pointer can indicate the extent to which a user is exploring various areas of the screen to find a specific UI element. For example, if the target UI element is normally located where visual guidance is displayed and the user easily recognizes that location, the mouse pointer may move relatively directly toward that location. On the other hand, if the user searches for the target UI element by repeatedly moving across the left, right, top, and bottom areas of the screen, the cumulative movement distance may increase.

[0273] However, the cumulative movement distance of the mouse pointer may also vary depending on the screen size, the user's pointer operation habits, and the characteristics of the step's operation. Accordingly, the processor (120) can check the reference mouse movement distance and normalize the cumulative movement distance of the mouse pointer measured in the current step by dividing it by the reference mouse movement distance. The reference mouse movement distance may be determined based on at least one of the distance between the guide target UI element of the current step and the mouse pointer at the time of entering the step, the display resolution, the average movement distance of normal users, or a reference value set by the content creator.

[0274] In mathematical formulas, a logarithmic function can also be applied to the accumulated movement distance of a normalized mouse pointer. This is intended to ensure that while it is normal for a user to move the mouse to a certain extent to navigate the screen, the need for re-recognition also increases gradually as the accumulated movement distance grows. By applying the logarithmic function, it is possible to respond sensitively to small increases in movement while preventing very large movements from excessively dominating the screen re-recognition trigger index.

[0275] The processor (120) can additionally check the number of user inputs for an area that does not correspond to a target UI element. Here, the number of user inputs may refer to the number of clicks, taps, selections, or corresponding inputs performed on a screen area other than the target UI element that the user must select in the current step. For example, if the target UI element of the current step is a rectangle tool on the left side of the screen, the processor (120) can set a target input area based on the position and size of the rectangle tool stored in the position registry and determine whether the coordinates of the user input are included in the target input area.

[0276] If the user's click coordinates fall within the boundaries of the target UI element or within a set allowed area, it can be determined as a valid input; clicks in other areas can be determined as incorrect inputs. If necessary, by setting the valid input range to an area extended by a certain distance beyond the actual boundaries of the target UI element, it is possible to prevent cases where the user selects areas near the edges of the UI element from being classified as incorrect inputs.

[0277] The number of user inputs in areas outside the target UI element can directly indicate the possibility of a discrepancy in the current guidance position, more so than simple mouse movements. For example, if a user repeatedly clicks near the guidance arrow but the target UI element is not selected, one can consider the possibility that there is a discrepancy between the coordinates stored in the current location registry and the actual location of the UI element. Furthermore, if a user repeatedly selects multiple different screen areas, it can be determined that the user is not properly recognizing the target UI element.

[0278] The processor (120) can normalize the number of user inputs based on a reference number of incorrect inputs. The reference number of incorrect inputs may be a value for setting the level of incorrect inputs that can be allowed during normal user operation in the current step. For example, the reference number of incorrect inputs may be set low in a step requiring a simple single button selection, and may be set to a relatively high value in a complex step requiring the manipulation of multiple screen elements.

[0279] In the mathematical formula, a first exponential coefficient may be applied to the number of normalized user inputs. Since the first exponential coefficient can be set to a value greater than 1, the impact on the screen re-recognition trigger index may increase faster than linearly as the number of incorrect inputs increases. This is intended to evaluate that while a single incorrect input can be viewed as a typical user error, if multiple incorrect inputs are repeated in the same step, it is more likely that the current guidance position is inconsistent with the actual UI element rather than a simple user error.

[0280] For example, instead of evaluating a single click on an area outside the target UI element and five repeated clicks at the same rate, the need for screen re-recognition can be significantly increased as repeated input errors accumulate. Accordingly, this prevents unnecessary screen re-recognition from occurring frequently due to simple one-time mistakes, while enabling active screen re-recognition in situations of repeated failure.

[0281] The processor (120) can additionally check the initial detection confidence calculated for the target UI element at the time of creating the location registry. The initial detection confidence may be a value indicating how high the correspondence is between the user screen and the template image of the target UI element during the screen recognition process. The initial detection confidence may be expressed as a value between 0 and 1, for example, and the closer it is to 1, the more likely it is that the UI element has been identified with high confidence.

[0282] The initial detection reliability can be stored along with the coordinates of the UI element during the location registry creation process. For example, one entry in the location registry can store an abstract identifier, on-screen coordinates, the width of the UI element, the height of the UI element, and the initial detection reliability in correspondence with each other. Accordingly, when the visual guidance of the current step refers to a specific abstract identifier, the processor (120) can check not only the target coordinates but also the initial detection reliability at the time the coordinates were first calculated in the same location registry.

[0283] A low initial detection reliability does not mean that the coordinates stored in the location registry are completely incorrect, but rather indicates that the uncertainty of the location information is relatively high. For example, initial detection reliability may decrease if multiple similar icons exist on the screen, if UI elements are partially obscured, or if the display state of the template image differs from that of the actual UI element.

[0284] In the mathematical formula, a second exponential coefficient can be applied to the complement value of the initial detection reliability. That is, when the initial detection reliability is close to 1, the influence on the screen re-recognition trigger index is kept low, and as the initial detection reliability decreases, the influence can be made to increase non-linearly. The second exponential coefficient can be set to a value greater than 1.

[0285] According to this configuration, UI elements identified with high confidence during the initial screen recognition stage can continue to use the existing location registry even if user behavior information fluctuates to a certain degree, whereas UI elements with low initial detection confidence may be more easily identified as targets for re-recognition if abnormal signs are detected in user behavior. In other words, even for the same user behavior, the necessity for screen re-recognition can be evaluated differently depending on the reliability of the existing location information.

[0286] The first weight may be a value used to adjust the degree of influence that an increase in dwell time has on the screen re-recognition trigger index. For example, if it is important for the user to perform each step quickly in a lecture, the first weight may be set relatively high, whereas in a lecture where the task itself requires a long time, the first weight may be set low.

[0287] The second weight may be a value used to set the degree of influence that the cumulative movement distance of the mouse pointer has on the screen re-recognition trigger index. For example, in lectures where screen navigation behaviors to find target UI elements are frequently included in the normal learning process, the second weight may be set relatively low, while in steps where the location of the target UI element is clear and pointer movement occurs briefly in normal cases, the second weight may be set high.

[0288] The third weight may be a value for controlling the influence of the number of user inputs in areas other than the target UI element. Since incorrect input can more directly indicate the inability to find the target UI element than simple dwell time or mouse movement, in one embodiment, the third weight may be set to be greater than the first or second weight. However, the relationship between specific weights is not limited thereto.

[0289] The fourth weight may be a value for setting the degree of influence that a decrease in initial detection reliability has on the screen re-recognition trigger index. For example, in an environment where the template matching result of the screen recognition engine (430) is highly reliable, the fourth weight may be set low, and in an environment where the variability of the UI recognition result is high due to various application versions or screen themes, the fourth weight may be set high.

[0290] The interaction weight may be a value intended to reflect the combined influence of the number of user inputs and the initial detection reliability on the need for screen re-recognition. In this embodiment, when a situation where incorrect inputs are repeated and a situation where the initial detection reliability is low occur simultaneously, it can be assessed that the coordinates of the location registry are more likely to be inconsistent with the actual user screen than when each element occurs individually.

[0291] For example, if a user makes one or two incorrect inputs regarding a UI element with very high initial detection confidence, it is relatively likely that this is a simple user operation error. On the other hand, if a user repeatedly clicks outside the target area regarding a UI element with low initial detection confidence, one can consider the possibility that the coordinates of the UI element stored at the time of initial recognition differ from the actual UI element. The interaction term is intended to further increase the screen re-recognition trigger index in such complex situations.

[0292] The mathematical formula of this embodiment may be characterized in that it does not simply weight and sum four types of information of different natures in the same manner, but applies different function forms according to the technical meaning of each piece of information. A logarithmic function is applied to dwell time and the cumulative movement distance of the mouse pointer to allow for a certain degree of value increase due to normal user behavior while suppressing an excessive increase in influence. On the other hand, an exponential form is applied to repetitive erroneous inputs and low initial detection reliability to increase the influence non-linearly as conditions strongly indicating the need for screen re-recognition accumulate.

[0293] Furthermore, by adding an interaction term between the erroneous input and the initial detection reliability, it is possible to reflect complex situations that are difficult to express solely by evaluating each parameter independently. In other words, rather than a rule that simply states "re-recognizes because the user stayed for a long time," this mathematical formula can provide a structure that evaluates the necessity of maintaining or updating the current location registry by combining the user's actual search and selection behaviors with the reliability of existing location information.

[0294] The processor (120) can compare the calculated screen re-recognition trigger index with a set threshold. The set threshold may be a reference value for determining whether it is necessary to automatically perform screen re-recognition. The set threshold may be applied commonly to the entire lecture, or may be set differently for each lecture, chapter, or step.

[0295] For example, in steps where the position of UI elements may frequently vary by user, the configured threshold can be set relatively low to actively perform re-recognition. Conversely, in steps where the position of the target UI element remains fixed, or in environments where it is necessary to minimize unnecessary re-recognition considering the computational cost of screen recognition processing, the configured threshold can be set relatively high.

[0296] When the screen re-recognition trigger index is greater than or equal to a set threshold, the processor (120) determines that there is a possibility of a discrepancy between the location information of UI elements stored in the current location registry and the actual user screen, and may perform screen re-recognition. At this time, screen re-recognition may be performed by re-recognizing all UI elements included in the target application, but may also be performed by selectively re-recognizing only the target UI elements used in the current step or multiple UI elements related to the current lecture.

[0297] The processor (120) can call the screen recognition engine (430) to re-acquire the user screen and compare the re-acquired user screen with a template image corresponding to the current target UI element. If re-recognition is successful, the screen location of the newly identified UI element can be reflected in the existing location registry. Accordingly, the previous coordinates corresponding to the abstract identifier can be updated to the new coordinates.

[0298] If necessary, the processor (120) may update the location registry only when the reliability of the re-recognition result is higher than the existing initial detection reliability. For example, if the existing detection reliability is 0.82 and the reliability of the re-recognition result is 0.95, new coordinates may be adopted, and conversely, if the reliability of the re-recognition result is below the threshold value, the existing coordinates may be maintained or the display of visual guidance may be temporarily suspended.

[0299] When the location registry is updated, the processor (120) can reposition the visual guidance of the current step based on the new target coordinates. For example, if the existing arrow points to an incorrect location as a result of the user moving the tool panel from left to right on the screen during a lecture, the screen re-recognition trigger index based on the user action can be increased above a set threshold. Subsequently, when the new location of the tool panel is confirmed by screen re-recognition and the location registry is updated, the arrow can also be repositioned to point toward the location of the newly confirmed UI element.

[0300] To illustrate a specific scenario, it can be assumed that the user is being instructed to "select the alignment button on the properties panel" during the current lecture step. At the start of the lecture, the screen recognition engine (430) can identify the alignment button at a specific coordinate on the right side of the screen and store that coordinate in the location registry. Subsequently, if the user moves the properties panel to a different location during the lecture, the actual location of the alignment button changes, but the location registry may still contain the previous coordinate.

[0301] In this case, the arrow for the current step is displayed based on the coordinates stored in the previous location registry, so it may point to a location different from the actual alignment button. Users may not be able to find the alignment button in the area pointed to by the arrow, causing them to stay on that step longer than the standard dwell time, repeatedly move the mouse pointer to various areas of the screen, and click multiple times in areas different from the target UI element.

[0302] The processor (120) can continuously check these dwell times, the cumulative movement distance of the mouse pointer, and the number of incorrect inputs at a set interval, and can also check the initial detection reliability stored in the location registry corresponding to the sort button. Subsequently, a screen re-recognition trigger index can be calculated according to a mathematical formula.

[0303] For example, if dwell time and mouse movement have increased but no input errors occur and the initial detection reliability remains high, the increase in the screen re-recognition trigger index can be limited by a logarithmic function. Therefore, it is possible to prevent screen re-recognition from being performed immediately simply because the user has read the explanation for a long time.

[0304] Conversely, if the standard dwell time is exceeded, mouse movement increases, erroneous inputs in areas outside the target UI element are repeated, and the initial detection reliability is low, the index term regarding erroneous input, the index term regarding initial detection reliability, and the interaction term between the two elements may all increase together. As a result, the likelihood of the screen re-recognition trigger index exceeding the set threshold may increase.

[0305] In this case, the processor (120) can automatically call the screen recognition engine (430) and re-identify the new on-screen position of the alignment button. Then, the corresponding coordinates in the position registry can be updated, and the arrow or highlight can be repositioned based on the updated coordinates. The user can receive visual guidance corresponding to the changed actual UI position again without having to directly input a separate "screen re-recognition" command.

[0306] In another scenario, it can be assumed that the user is a learner who simply reads the lecture content slowly. In this case, while the dwell time may be long, the amount of mouse pointer movement is small, there are no erroneous inputs in areas outside the target UI elements, and the reliability of initial detection can be high. In such cases, since only the log term for dwell time increases to a certain level while the influence of the other terms remains low, the screen re-recognition trigger index may not reach the set threshold. Accordingly, unnecessary repeated re-recognition can be prevented for users with a normally slow learning speed.

[0307] In another scenario, the cumulative movement distance of the mouse pointer may increase due to the user's habit of actively exploring the screen, yet the target UI element is successfully selected and no input errors occur. Even in this case, since a logarithmic function is applied to the cumulative movement distance, it is possible to suppress a rapid increase in the screen re-recognition trigger index based solely on the amount of mouse movement.

[0308] Conversely, regarding UI elements with low initial detection reliability, if a user repeatedly selects areas outside the target UI element, the need for screen re-recognition may be assessed as high, even if other behavioral information is relatively insignificant. This is because the uncertainty of the initial recognition result and the actual user's operational failures serve as evidence that align with each other.

[0309] According to the present embodiment, whether to perform screen re-recognition does not depend solely on a simple fixed time or an explicit user request for re-recognition, but can be dynamically determined by considering user behavior information obtained during the actual lecture process together with the reliability of existing screen recognition results. Accordingly, the problem of incorrect guidance continuing due to the continued use of the existing location registry even when the UI layout of the user screen has changed can be reduced.

[0310] Furthermore, since normal user behavior and behavior caused by location information discrepancies can be distinguished and evaluated, the frequency of unnecessary screen re-recognition can be reduced. In particular, by combining logarithmic, exponential, and interaction terms, non-linear influence relationships tailored to the nature of each parameter can be reflected, and the necessity for screen re-recognition can be assessed relatively more strongly when multiple abnormal signs occur simultaneously compared to when a single parameter increases temporarily.

[0311] Consequently, the present embodiment has technical significance in that it not only utilizes user behavior as simple learning statistics but also utilizes user behavior as feedback information to control the operation of an actual screen recognition system. That is, a closed feedback structure can be formed by generating a location registry based on screen recognition results, displaying visual guidance using it, analyzing the user's subsequent behavior to evaluate the validity of the location registry, and performing screen re-recognition and location registry updates if necessary. Accordingly, even if the UI location of the target application changes during a lecture, the visual guidance can be continuously aligned with the actual state of the user screen.

[0313] The above guide on how to use the overlay-based software is,

[0314] The above processor determines whether the current position coordinates of the mouse pointer have entered the boundary area of ​​the UI element stored in the position registry; and

[0315] The above processor may further include an operation of generating tactile feedback by transmitting a vibration pattern control signal to a user's wearable device through a near-field wireless communication module when it is determined that the mouse pointer has entered the boundary area of ​​the UI element.

[0316] The processor (120) can check the positional relationship between the UI element that is the target of visual guidance in the current lecture step and the user's mouse pointer in real time, and provide haptic feedback to the user when the mouse pointer enters the boundary area of ​​the target UI element.

[0317] Here, the boundary area of ​​the UI element can be set based on the coordinates and size of the target UI element stored in the location registry. For example, if the center coordinates, width, and height of the target UI element are stored in the location registry, the processor (120) can determine the rectangular boundary area occupied by the UI element on the user screen based on the information. In another embodiment, the boundary area may also be set by directly using the top-left and bottom-right coordinates of the UI element.

[0318] The processor (120) can obtain the current position coordinates of the mouse pointer from the user's mouse input device at set intervals. For example, it can receive the x-axis and y-axis coordinates of the current mouse pointer from an operating system or an input event processing module, and determine whether the received coordinates are included in the boundary area of ​​a target UI element stored in a position registry.

[0319] For example, it can be assumed that the boundary area of ​​a target UI element is set to 500 to 540 in the x-axis direction and 300 to 340 in the y-axis direction. When the current position of the mouse pointer is (480, 320), it can be determined that it is located outside the UI element, and when the position of the mouse pointer changes to (515, 325), it can be determined that it has entered inside the boundary area.

[0320] In one embodiment, the processor (120) can identify an actual "entry" event by comparing not only whether the mouse pointer is simply inside the boundary area, but also whether it was outside the boundary area at a previous time and is inside the boundary area at a current time. Accordingly, haptic feedback can be prevented from occurring repeatedly while the mouse pointer remains over the same UI element.

[0321] When the processor (120) determines that the mouse pointer has entered the boundary area of ​​a target UI element, it can transmit a vibration pattern control signal to the user's wearable device using a short-range wireless communication module included in the communication interface (130). Here, the short-range wireless communication may include, for example, Bluetooth, Bluetooth Low Energy, or an equivalent short-range wireless communication method, but is not limited thereto.

[0322] The wearable device may be a smartwatch, smart band, haptic band, smart ring, or an electronic device that can be worn on a user's body to generate vibrations. The wearable device may include a vibration motor, a linear vibration actuator, or other forms of tactile output devices and may generate vibrations in response to a vibration pattern control signal received from the electronic device (100).

[0323] The vibration pattern control signal may simply be information indicating whether vibration occurs, or it may include at least one of the vibration intensity, duration, number of repetitions, or vibration interval. For example, a short single vibration may be generated when the mouse pointer first enters a target UI element, and two short vibrations may be generated in succession when the element is a core UI element that the user must select at the current step.

[0324] In another embodiment, the vibration pattern may be changed according to the distance to the target UI element. For example, a weak vibration may be generated when the mouse pointer approaches within a set distance of the target UI element, and a stronger vibration may be generated when it enters the actual boundary area. Accordingly, haptic information can be provided step-by-step along with visual guidance.

[0325] This haptic feedback allows the user to recognize that the mouse pointer has reached a target UI element without having to continuously watch only the arrow or highlighted area on the screen. In particular, when multiple icons are densely packed on the screen or when selecting a small UI element, the user can immediately confirm that they have entered the target area through the haptic signal.

[0326] For example, it can be assumed that the current step provides a "Select pen tool" instruction. The processor (120) checks the coordinates and size of the pen tool in the location registry and can send a short vibration signal to the wearable device the moment the mouse pointer enters the corresponding boundary area. The user moves the mouse while checking the arrow on the screen and can additionally confirm whether they have reached the target UI element through the vibration occurring in the wrist the moment they reach it.

[0327] Accordingly, the present embodiment can provide user guidance using multiple sensory channels by combining tactile feedback with existing visual overlay guidance, and can enable the user to more immediately recognize when they have accessed or reached a target UI element.

[0329] The above guide on how to use the overlay-based software is,

[0330] The above processor detects a window coordinate change event in which the execution window of the target application moves from a first display to a second display; and

[0331] The processor may further include an operation of obtaining the DPI scale and monitor start offset coordinates of the second display, and reordering the display position of the transparent overlay by arithmetic applying the DPI scale and monitor start offset coordinates to the target coordinates stored in the position registry without executing a full screen re-recognition process.

[0332] The processor (120) can detect when the execution window of a target application moves from one display to another and can convert the target coordinates stored in the location registry to fit the new display environment without performing the full screen recognition process again.

[0333] In one embodiment, the electronic device (100) may include a plurality of displays or be connected to a plurality of external monitors. Each display may have a different resolution, DPI scaling, or screen start coordinates. For example, the first display may be set to a 100% scaling and the second display may be set to a 150% scaling, and the screen coordinate system start point of the second display may be located outside the right side of the first display.

[0334] The processor (120) can check for a window coordinate change event of the operating system indicating the location of the execution window of the target application. Here, the window coordinate change event may include a change in at least one of the top-left coordinates of the window, size, identification information of the connected display, or information of the monitor to which the window belongs.

[0335] For example, if it is confirmed that the window of the target application has moved from the coordinate area of ​​the first display to the coordinate area of ​​the second display, the processor (120) may determine that the display on which the target application is displayed has changed.

[0336] The processor (120) can obtain a DPI scaling factor and monitor start offset coordinates set on the second display. The DPI scaling factor may represent the ratio between physical display pixels and logical coordinates of an application or overlay, and the monitor start offset coordinates may represent the reference start position of the second display when multiple displays are arranged in a single virtual desktop coordinate system.

[0337] For example, one can assume a case where the starting coordinates of the first display are (0, 0), the second display is placed to the right of the first display and has starting coordinates of (1920, 0), and the DPI scale of the second display is 1.5. In this case, if the coordinates of the UI element stored based on the first display are used as is on the second display, the actual UI element and the visual guide may be misaligned.

[0338] Accordingly, the processor (120) can calculate new coordinates by arithmetically applying the DPI scaling of the second display and the monitor start offset coordinates to the target coordinates stored in the location registry. For example, the existing coordinates can be enlarged or reduced in proportion to the scaling, and the start offset of the second display can be added to determine the transformed position of the target UI element in the virtual desktop overall coordinate system.

[0339] In this process, the entire user screen may not be recaptured, and template matching may not be performed repeatedly for multiple UI elements. That is, if the processor (120) determines that the UI of the target application itself has not changed and only the display has changed, it may perform only the display coordinate system transformation while maintaining the relative positional relationship of the existing location registry.

[0340] In another embodiment, the width and height information of the UI element, as well as the coordinates, can be converted by taking into account the difference in DPI scaling between the first display and the second display. For example, if the width of the existing UI element is 40 and the scaling of the second display is 1.5 times larger than that of the first display, the converted size of the UI element can be set to a value corresponding to 60.

[0341] The processor (120) can use the converted position information to realign the display positions of arrows, highlight areas, spotlights, or subtitle areas within the transparent overlay. In this case, the relative direction of the visual guide and the alignment relationship with UI elements can be maintained in their existing state while only the overall position is moved to fit the new display coordinate system.

[0342] For example, it can be assumed that a user moves the Illustrator execution window from the laptop's built-in screen to an external monitor during a lecture. The electronic device (100) detects the window's display movement event, checks the DPI scale and start offset of the external monitor, and then converts the coordinates of each UI element stored in the existing position registry. Subsequently, the display positions of the arrows and highlight areas can also be moved immediately based on the converted coordinates.

[0343] Accordingly, since there is no need to re-execute the full screen recognition engine (430) with just a simple display movement, the computational load and delay required for screen re-recognition can be reduced. In particular, since existing location information can be reused even in lectures where template matching for multiple UI elements is required, the guidance screen can be quickly rearranged after the display movement.

[0344] However, if the processor (120) determines that the internal UI layout of the target application has changed along with the display movement, it may perform screen re-recognition without applying only coordinate transformation. For example, if the window size is significantly changed and the panel is repositioned or some UI elements are hidden, it may be difficult to accurately reflect the actual location using only simple arithmetic transformation of the existing location registry, so the screen recognition engine (430) may be called again.

[0345] Accordingly, the present embodiment distinguishes between "structural changes to the UI itself" and "changes to the display coordinate system" and, in the case of the latter, performs coordinate transformation that omits full re-recognition, thereby enabling more efficient maintenance of visual guidance in a multi-monitor environment.

[0347] The above guide on how to use the overlay-based software is,

[0348] The above processor performs the operation of measuring the amount of change in the movement coordinates of the mouse pointer per unit time and calculating the instantaneous acceleration and the number of direction changes of the mouse pointer; and

[0349] If the processor determines that the instantaneous acceleration exceeds a set reference acceleration and simultaneously the number of direction changes exceeds a set threshold number, resulting in a panic state, it may further include an operation of immediately enlarging and rendering the border thickness of the visual guide and the diameter of the alignment circle displayed on the transparent overlay by a set ratio.

[0350] The processor (120) analyzes the user's mouse pointer movement to identify a state in which the user cannot find a target UI element in the current step and rapidly navigates the screen, and if such a state is confirmed, it can automatically increase the display intensity of the visual guidance.

[0351] In the present disclosure, a panic state does not necessarily mean determining the user's psychological state itself by directly measuring it, but may mean an operational state in which rapid movement of the mouse pointer and repetitive changes in direction satisfy the set conditions.

[0352] The processor (120) can continuously acquire the position coordinates of the mouse pointer at set time intervals while the current step is in progress. For example, it can sequentially check the first coordinate at time t1, the second coordinate at time t2, and the third coordinate at time t3, and calculate the distance traveled and the direction of travel for each time interval.

[0353] The processor (120) can calculate the movement speed of the mouse pointer based on the amount of change in consecutive coordinates and time intervals, and can calculate the instantaneous acceleration based on the amount of change in consecutive movement speeds. Here, the instantaneous acceleration can be used as an indicator of how rapidly the user changes the movement speed of the pointer within a short period of time.

[0354] Additionally, the processor (120) can calculate the number of times the mouse pointer changes direction by comparing the directions of consecutive movement vectors. For example, if the difference in direction angle between the first movement vector and the second movement vector is greater than or equal to a set reference angle, it can be determined as one change in direction, and the number of times such a change in direction occurs during a set analysis time can be accumulated.

[0355] Accordingly, when a user naturally moves from one side of the screen toward a target UI element, the number of direction changes may be calculated to be low, and when a user rapidly moves back and forth across multiple areas of the screen to explore the target UI element, the number of direction changes may increase.

[0356] The processor (120) can determine the current user operation state as a panic state if the instantaneous acceleration exceeds a set reference acceleration and at the same time the number of direction changes exceeds a set threshold number. By using both conditions simultaneously, the likelihood of mistaking a situation where the mouse is simply moved quickly or the direction is changed once or twice for a panic state can be reduced.

[0357] For example, even if a user rapidly moves a pointer to select a UI element located far away on a large screen, it may not be judged as a panic state if the direction of movement is consistent. Conversely, if rapid changes in speed and multiple changes in direction occur simultaneously over a short period, it can be assessed that the user is highly likely to be repeatedly navigating to find a target UI element on the current screen.

[0358] The processor (120) can increase the border thickness of the visual guide displayed on the transparent overlay when a panic state is detected. For example, if the thickness of the existing highlight border was displayed as a first value, it can be changed to a second value based on a set magnification ratio.

[0359] Additionally, the diameter of the alignment circle or circular highlight area displayed around the target UI element can be expanded by a set ratio. For example, if the existing circular highlight was displayed with a size expanded by 10 pixels beyond the outline of the UI element, after a panic state is detected, it can be displayed with a range expanded by 20 pixels or a set ratio.

[0360] If necessary, at least one of the arrow size, blinking frequency, opacity, or subtitle font size may also be increased together. However, these additional display changes may be applied optionally.

[0361] The processor (120) may also return the enlarged visual guidance to its original display state when the panic state is released. For example, if the instantaneous acceleration and the number of turns decrease to a normal range for a set period of time, the thickness of the visual guidance and the size of the circular highlight area may be returned to their original values.

[0362] As a specific example, let's assume the user is currently performing the step "Select the New Layer button in the Layers panel." If the user fails to find the New Layer button and rapidly moves the pointer repeatedly to the left, right, top, and bottom, both the instantaneous acceleration and the number of direction changes can increase.

[0363] The processor (120) can determine that the action is in a panic state when the set conditions are met, and can enlarge the diameter of the circular highlight area displayed around the new layer button and make the border thicker. Accordingly, the target UI element can be more clearly highlighted compared to other UI elements on the screen.

[0364] This embodiment is significant in that it does not determine whether a user is performing the lecture normally based solely on the success or failure of a click, but dynamically changes the display intensity of visual guidance using actual mouse movement patterns. Therefore, guidance can be automatically enhanced without a separate request when the user's navigation behavior becomes complex.

[0367] The action of displaying the above visual guidance is,

[0368] The above processor performs the operation of creating a black alpha layer with a set opacity that covers the entire user screen on the transparent overlay; and

[0369] The processor may include an operation of inverting only a specific area on the black alpha layer corresponding to the target coordinates identified in the location registry and the size area of ​​the UI element to an alpha value of 0, thereby exposing only the UI element area in a spotlight form.

[0370] The processor (120) can provide a spotlight-type visual guide that displays the remaining area of ​​the user screen, excluding the target UI element, relatively darkly in order to more clearly highlight the target UI element of the current lecture step on the user screen.

[0371] The processor (120) can create a black alpha layer with an opacity set to correspond to the entire user screen of the target application or the entire execution window of the target application on a transparent overlay. Here, the alpha layer is a graphic layer additionally superimposed on the screen of the target application, and can be configured so that a portion of the screen of the target application located underneath is visible according to the set alpha value.

[0372] For example, the color of the alpha layer can be set to black, and the opacity can be set to 30%, 50%, or other values ​​according to user settings. Accordingly, the overall brightness can be lowered while keeping the original screen recognizable, rather than completely obscuring the entire user screen.

[0373] The processor (120) can determine the target coordinates and the size of the UI element corresponding to the target UI element of the current step in the location registry. For example, if the center coordinates, width, and height of the target UI element are stored in the location registry, the area to form the spotlight can be determined based on the information.

[0374] Afterward, the processor (120) can change the alpha value of a specific area of ​​the alpha layer corresponding to the target UI element to 0 or a value close to it. Accordingly, the black alpha layer is effectively removed in that area, and the actual UI element located underneath can be exposed at its original brightness.

[0375] For example, if a black layer with 50% opacity is superimposed on the entire user screen, setting the alpha value to 0 only for a rectangular area of ​​the target UI element can make that area appear brighter than its surroundings. Users can perceive the target UI element as being clearly exposed amidst a darkened background.

[0376] The spotlight area can be set to exactly match the actual boundaries of the UI element, or it can be set as an area extended from the UI element's boundaries by a set margin. For example, when targeting a small button, an area larger than the actual button by a certain distance in all directions can be made transparent to allow users to more easily recognize the target UI element.

[0377] The shape of the spotlight area does not necessarily have to be limited to a rectangle. For example, it can be implemented as a circle, an ellipse, a rectangle with rounded corners, or a mask corresponding to the actual outline of the UI element.

[0378] The processor (120) may simultaneously display an arrow, an accent border, or subtitles along with a spotlight. For example, the entire screen may be made semi-transparent and darkened, while only the target button portion is brightly exposed, and an arrow may be additionally displayed on the outside of the button.

[0379] Additionally, when the user moves to the next step, the processor (120) can remove the existing spotlight area and create a new spotlight area based on the location registry coordinates of the newly designated UI element. Accordingly, the position highlighted on the screen can be automatically changed according to the order of the lecture.

[0380] For example, if the prompt "Select the File menu" is provided in the current step, only the File menu area at the top can be highlighted while the rest of the screen is displayed in a dark state. When the user selects the File menu to move to the next step, the display can be changed to highlight only the area where the "Export" menu item is located.

[0381] According to the present embodiment, the burden of having to directly navigate to a target UI element in a complex screen where multiple menus and panels are displayed simultaneously can be reduced. In particular, since only the target UI element can be selectively exposed while relatively reducing the visual stimulation of surrounding UI elements, it is possible to provide the effect of focusing the user's gaze on the target location.

[0383] The above guide on how to use the overlay-based software is,

[0384] The above processor defines a graphic object area on the transparent overlay where the visual guide and subtitle data are displayed as a first area, and defines the remaining area excluding the first area as a second area; and

[0385] The processor may include an operation of receiving and processing mouse events generated in the first area at the overlay processor, and dynamically resetting window area properties so that mouse focus and click events generated in the second area are directly transmitted to the target application placed underneath by clicking through the transparent overlay.

[0386] The processor (120) can divide the areas inside the overlay by function and apply different mouse event processing methods to each area so that the user can continue to operate the actual target application even if the transparent overlay is placed on the target application.

[0387] The processor (120) may define a first area as an area where graphic objects requiring direct interaction with the user are displayed on a transparent overlay. For example, the first area may include a subtitle display area, a next step button, a previous step button, a lecture pause button, a visual guide object, or an area of ​​other graphic objects that need to receive input directly from the user.

[0388] The processor (120) may define the remaining area of ​​the entire transparent overlay area, excluding the first area, as the second area. The second area may be an area where separate user input processing is not required in the overlay itself, but where user input is required to be transmitted to the target application displayed underneath.

[0389] The processor (120) can determine in real time whether the current position of the mouse pointer is included in the first area or the second area. For example, if the mouse pointer is positioned over the subtitle area or the lecture control button area, it is determined to correspond to the first area, and if it is positioned over the tool or work area of ​​the target application, it is determined to correspond to the second area.

[0390] When the mouse pointer is located in the first area, the processor (120) can directly receive and process the mouse event occurring at that location from the process providing the overlay or the overlay-related user interface module. For example, when the user clicks the "Next" button, the click is not transmitted to the target application, but is processed as an input to move to the next step of the lecture.

[0391] On the other hand, when the mouse pointer is located in the second area, the processor (120) can change the input processing properties of the window area so that the area does not block mouse input. Accordingly, mouse focus, mouse movement, or click events can be delivered to the target application placed underneath without being consumed in the transparent overlay.

[0392] In the present disclosure, click-through may refer to an action in which a transparent overlay is visually present over the target application, but in a configured area, it is not a target for receiving mouse input, and a sub-window receives user input directly.

[0393] The processor (120) can dynamically change the input processing properties between the first area and the second area according to the movement of the mouse pointer. For example, when the mouse pointer moves from the work area of ​​the target application to the subtitle area, it can change the overlay to receive mouse events, and when it moves back to the work area, it can change the input to be transmitted to the target application.

[0394] This behavior can be distinguished from the method of uniformly setting the entire transparent overlay to a click-through state. If the entire overlay is set to a click-through state, user input regarding lecture control buttons or subtitle areas is transmitted to the target application, which may prevent the user interface of the overlay itself from being manipulated. Conversely, if the entire overlay receives all mouse events, the user may be unable to select menus and buttons within the actual target application.

[0395] Therefore, in this embodiment, the area where user interface objects exist and the area where actual target application manipulation is required are distinguished, and by setting different input processing methods for each area, both functions can be satisfied simultaneously.

[0396] For example, it can be assumed that an overlay UI containing lecture subtitles and "Previous," "Next," and "Pause" buttons is displayed at the bottom of the screen. In this area, click events can be received by the second application (421) and used to control the progress of the lecture.

[0397] On the other hand, the target application tool panel on the left side of the screen may display only an arrow via a transparent overlay. When a user clicks the actual tool pointed to by the arrow, the click is not blocked by the overlay and can be transmitted to the underlying target application. Therefore, the user can directly manipulate the actual target application even while the guide object is displayed.

[0398] In another embodiment, the visual guidance itself may be classified into a first area or a second area depending on whether it requires user input. For example, an arrow that simply shows a location can be set as a second area that does not receive mouse input, and a guidance icon that displays additional explanations when clicked by the user can be set as a first area.

[0399] Additionally, the scope of the first and second areas can be dynamically changed according to the lecture steps. For example, at a specific step, an additional explanation panel may be displayed in the center of the screen, increasing the area of ​​the first area, and when that panel disappears in the next step, that area can be changed back to the second area.

[0400] According to the present embodiment, even when a transparent overlay is continuously displayed over the target application, the lecture control interface required by the user operates normally, and mouse operation on the target application itself is not interfered with. Accordingly, the user experience of the present invention, which provides overlay-based guidance while directly manipulating the actual software without separate window switching, can be stably implemented.

[0401] Furthermore, this embodiment has technical significance in that, unlike simply adjusting the transparency of the overlay, it controls the visual display layer and the input event transmission layer independently of each other. That is, while the overlay continues to exist on the screen, the recipient of input events can be dynamically switched between the overlay and the target application depending on the area, thereby simultaneously ensuring the continuity of visual guidance and the possibility of actual software manipulation.

[0403] FIG. 4 is a configuration diagram showing the connection relationships between a plurality of applications, a screen recognition engine, and a database according to one embodiment of the present invention.

[0404] Referring to FIG. 4, the software usage guidance system according to the present invention may include a first application (410), a second application (421), an IPC (Inter-Process Communication, 423), a screen recognition engine (430), and a database (440). Here, each component does not necessarily need to be placed on a different physical device and may be implemented as a plurality of software processes or functional modules running on a single electronic device (100), and some components may be placed on a separate computing device such as an external server (220).

[0405] The first application (410) may be an application that provides a basic user interface for searching, selecting, and executing a lecture to the user. For example, through the first application (410), the user can select one of a plurality of lectures corresponding to the target application they wish to study, and check the title, lecture description, estimated time required, or learning progress status of the selected lecture. When the user selects the target lecture and requests the start of the lecture, the first application (410) may transmit lecture identification information, user identification information, authentication information, language setting information, or information regarding the target application necessary to execute the lecture to the second application (421).

[0406] The second application (421) may be an application that actually conducts the lecture selected by the user and manages the guidance displayed on the target application. The second application (421) may create a lecture session based on lecture-related information transmitted from the first application (410), obtain data necessary for the lecture from the database (440), and provide subtitles or visual guidance to the user according to the progress of each step. Therefore, if the first application (410) functions as an interface for selecting and entering the lecture, the second application (421) can be understood as a component responsible for the actual execution and progress of the selected lecture.

[0407] The area (420) indicated by the dotted line in FIG. 4 can conceptually represent a configuration in which the second application (421) and the IPC (423) are interconnected. The IPC (423) may be a means of communication for transmitting data between processes or modules that are executed separately within an electronic device (100). For example, if the processing area responsible for the user interface of the second application (421) and the processing area responsible for managing the status of the lecture, calling external functions, or controlling the system are separated from each other, the current lecture step, user input, screen re-recognition request, visual guidance display request, or screen recognition result can be transmitted through the IPC (423).

[0408] IPC (423) is not limited to a specific communication standard. That is, message passing between processes, event calling, request and response communication, shared data structures, or equivalent inter-process communication methods may be used. Importantly, functions related to displaying a lecture screen to a user and screen recognition or system control do not necessarily have to be performed in a single processing flow; instead, they can be configured to exchange necessary information between processing areas separated by function.

[0409] The screen recognition engine (430) may be configured to analyze the user screen on which the target application is displayed and to identify one or more UI elements present on the current user screen. For example, the second application (421) may provide abstract identifiers corresponding to UI elements used in the current lecture to the screen recognition engine (430), and the screen recognition engine (430) may identify each actual location on the screen by comparing it with a template image corresponding to the UI element after acquiring the user screen.

[0410] The screen recognition engine (430) can provide not only the coordinates of the identified UI elements but also, if necessary, the width, height, or reliability information regarding the recognition results of the UI elements. The second application (421) can use these results to create or update a location registry that maps abstract identifiers to actual screen coordinates. Accordingly, visual guidance can be placed corresponding to the actual user screen at the time of lecture execution without storing fixed coordinates at a specific resolution in the lecture content itself.

[0411] For example, it can be assumed that the user has started a lecture with the tool panel of the target application moved to a location different from its default position. In this case, the second application (421) transmits an abstract identifier of the tool used in the lecture to the screen recognition engine (430), and the screen recognition engine (430) can detect the current location of the tool on the user's screen. Subsequently, the second application (421) can place an arrow or highlight based on the detected location, so that the guidance location can be adjusted to the actual environment even if the lecture creator's original screen layout and the user's actual screen layout are different.

[0412] The database (440) can store various data constituting the lecture content according to the present invention. For example, the database (440) can store at least one of basic information per lecture, chapter information, step information, subtitle data, visual guidance data, step order, abstract identifier, locale data, and UI element information required for screen recognition. Additionally, the database (440) may additionally store lecture progress rate per user, completed steps, lecture usage history, or user setting information.

[0413] The second application (421) can transmit and receive data bidirectionally with the database (440). For example, when a user selects a target lecture, the configuration data of the lecture can be retrieved from the database (440), and during the course, information that the user has completed a specific chapter or step can be stored in the database (440). Accordingly, even if the user runs the course again after ending it, it is possible to check the previous progress status and continue providing the lecture.

[0414] According to the configuration of Fig. 4, lecture selection and service entry, actual lecture execution, information transfer between processes, user screen recognition, and lecture data management can each be functionally separated. Accordingly, each function can be managed or updated independently, and processing that requires a relatively large amount of computation, such as screen recognition, can be performed by a separate screen recognition engine (430), while the results can be linked to the actual guidance display process of the second application (421).

[0416] FIG. 5 is a flowchart illustrating the process of querying, sorting, and assembling lecture-related data according to one embodiment of the present invention.

[0417] Referring to FIG. 5, data regarding a target lecture can be managed in a separated form, such as courses, which stores data regarding the lecture itself; courses_chapters, which stores data regarding multiple chapters constituting the lecture; and courses_steps, which stores data regarding multiple steps belonging to each chapter. Unlike a method of storing a lecture as a single integrated file, this corresponds to a form in which the overall information of the lecture, chapter information, and detailed step information to be provided to the actual user are managed separately.

[0418] Lecture data corresponding to courses may include, for example, lecture identification information, lecture title, type of target application or tool, applicable version, estimated learning time, lecture usage format, information regarding UI elements requiring screen recognition, or information to be provided upon completion of the lecture. Such data can be used as meta-information that applies commonly to an entire target lecture, rather than being information dependent on a single specific step.

[0419] The chapter data corresponding to courses_chapters may include chapter identification information, chapter title, identification information of the course to which the chapter belongs, and chapter order information indicating the order in which the chapter should be placed within the target course. For example, if a target course includes three chapters such as "Mastering the Basic Interface," "Creating Shapes," and "Editing Shapes," information corresponding to the first, second, and third order may be assigned to each chapter.

[0420] The step data corresponding to courses_steps may include identification information for each step, a title, identification information for the chapter to which the step belongs, the order of steps within the chapter, subtitle data, visual guidance data, and additional information necessary for the progress of the lecture. If necessary, the step data may further include the location of supplementary videos, information on whether the step will proceed automatically, estimated execution time, conditions for transitioning to the next step, or supplementary guidance regarding specific panels.

[0421] The electronic device (100) can retrieve the lecture data, chapter data, and step data using the lecture identifier of the target lecture selected by the user. At this time, the three types of data do not necessarily need to be retrieved sequentially one by one, and may be requested in parallel if they can be retrieved independently of each other. Accordingly, the time required to obtain multiple types of data necessary to compose the entire lecture can be reduced.

[0422] When data regarding lectures, chapters, and steps is retrieved, the electronic device (100) can arrange and assemble them in a form suitable for the actual lecture progression. In FIG. 5, this processing process may be illustrated as an arrangement and assembly step delineated by a dotted line. Since the order in which data is returned from a remote database cannot be guaranteed to match the actual lecture progression order, the electronic device (100) can determine the actual playback order of the lecture based on a separately stored chapter order and step order.

[0423] First, the electronic device (100) can generate a chapter order map that represents the correspondence between the identification information of each chapter and the chapter order based on the retrieved chapter data. For example, if the identifier of the first chapter is chapter_A and its order is 1, and the identifier of the second chapter is chapter_B and its order is 2, the electronic device (100) can configure the correspondence so that the order within the entire lecture of that chapter can be immediately checked using each chapter identifier.

[0424] Subsequently, the electronic device (100) can align multiple steps based on a two-stage alignment criterion. The first alignment criterion may be the order of the chapter to which the step belongs, and the second alignment criterion may be the order of steps within that chapter. This two-stage alignment is intended to accurately handle a structure in which the step order in different chapters can each start again from 1.

[0425] For example, it can be assumed that Chapter 1 includes Step 1, Step 2, and Step 3, and Chapter 2 also includes Step 1, Step 2, and Step 3. If the entire data is sorted simply based on the step order, there is a possibility that Step 1 of Chapter 1 and Step 1 of Chapter 2 will be placed adjacent to each other. On the other hand, in the present embodiment, the chapter order is checked first, and the step order is applied again within the same chapter, so a normal lecture flow can be constructed in which the steps of Chapter 2 proceed after all steps of Chapter 1 are completed.

[0426] Next, the electronic device (100) can convert each sorted step data into a step-unit data object that can be used for actual lecture execution. In this process, information necessary for actual lecture execution can be selected from among several fields stored in the database, or some data can be converted and included to suit the user's environment. For example, if subtitle data includes strings corresponding to multiple languages, the subtitle corresponding to the user's target language can be selected first.

[0427] In one embodiment, if subtitles corresponding to the target language do not exist, subtitles of a preset standard language may be selected. For example, if subtitles corresponding to the user's service language do not exist, subtitles of a default language, such as Korean, may be used. This alternative processing is intended to prevent the execution of the entire lecture from being interrupted even if some language data is missing.

[0428] When the conversion for each step is completed, the electronic device (100) can integrate the meta-information of the target lecture, the chapter arrangement, and the sorted step arrangement to form a single lecture script or corresponding lecture-related data. Here, the lecture script can be understood as data of a lecture execution unit that is referenced while the user actually executes the lecture, and is not necessarily limited to a specific programming object format.

[0429] The configured lecture-related data may be provided to the second application (421). The second application (421) may proceed with the lecture according to the order of the arranged steps and guide the user on the task to be performed at present using the subtitle data and visual guidance data included in each step.

[0430] Additionally, the lecture-related data may include information for specifying UI elements requiring screen recognition in the target lecture. By providing this information to the screen recognition engine (430), the second application (421) can enable screen recognition to be performed based on the UI elements actually used in the current lecture, rather than all UI elements present in the target application.

[0431] For example, if the target lecture only explains rectangle creation and color change, even if there are hundreds of menus and tools in the target application, only the UI elements necessary for the lecture, such as rectangle tools, color panels, and confirmation buttons, can be set as screen recognition targets. Therefore, the lecture data assembly according to FIG. 5 is not limited to simply determining the order of subtitles to be displayed to the user, but can also be utilized to determine the scope of screen recognition to be performed thereafter.

[0432] Consequently, according to the embodiment of FIG. 5, lecture, chapter, and step data stored separately externally can be retrieved at the time of execution in correspondence with the target lecture, and the actual lecture sequence can be constructed by applying the chapter order and the step order within the chapter step by step, and then each step can be converted into a form corresponding to the user environment and assembled into a single executable lecture-related data. Accordingly, the storage and management stage of the lecture content and the actual lecture execution stage can be separated, while changes to the content can be flexibly reflected at the time of execution.

[0434] FIG. 6 is an example diagram showing a state in which visual guidance is displayed on a user screen according to one embodiment of the present invention.

[0435] Referring to FIG. 6, the actual user interface of the target application may be displayed on the user screen. For example, the target application may include a plurality of menus at the top, a plurality of tools or icons that the user can select during the process in one side area, and a work area in the center area where the user performs editing or work may be displayed.

[0436] The user screen illustrated in FIG. 6 is not intended to limit the specific UI of a particular application, but rather to exemplify how the visual guidance according to the present invention can be provided to the user on an actual target application. Accordingly, the number of menus, the arrangement of icons, the shape of the work area, and the design of each UI element of the target application may vary depending on the type, version, operating system, or user's UI settings of the target application.

[0437] In one embodiment, a specific UI element of the user screen may be designated as a guidance target for a currently ongoing lecture step. For example, if the current step requires the selection of a specific rectangular object or rectangular tool in the work area, an abstract identifier corresponding to the UI element may be included in the visual guidance data. The electronic device (100) may check the actual coordinates corresponding to the abstract identifier in the location registry and provide visual guidance such as that shown in FIG. 6 based on the checked coordinates.

[0438] In FIG. 6, a dotted line highlighting area may be displayed on the outer edge of the UI element or work target area that is the subject of guidance. This dotted line area may be an example of a highlighting indicator used to visually distinguish an object that the user should pay attention to from other screen areas. In actual implementation, not only dotted line borders but also solid line borders, circular borders, translucent areas, blinking indicators, magnified indicators, or other forms of highlighting effects using brightness differences may be used.

[0439] In addition, in FIG. 6, a visual guide in the form of an arrow may be displayed at a location adjacent to the highlighted object. The arrow is intended to allow the user to intuitively recognize the direction of the UI element that they need to currently operate or check. In particular, when many menus and icons are displayed simultaneously in the target application, the user's navigation burden can be reduced by providing an arrow that directly points to the element on the actual screen, rather than explaining the location of a specific UI element with simple subtitles alone.

[0440] The position of the arrow is not fixed to a single absolute coordinate during the lecture content creation stage, but can be determined based on the coordinates of the target UI element actually identified on the current user screen. For example, if the guidance target is located in the center of the screen in the lecture creator's environment but is located on the right side of the screen in the actual user's environment, the electronic device (100) can move the arrow to the right side of the screen and display it based on the coordinates stored in the user's actual location registry.

[0441] Furthermore, the endpoint of the arrow may not simply be set to be the same as the center point of the target UI element. In order to display the arrow pointing toward the target element from the outside as in FIG. 6, the electronic device (100) may consider the width and height of the UI element along with the center coordinates of the target UI element. Then, after determining the point where the direction of the arrow toward the target UI element meets the boundary of the UI element, the endpoint of the arrow may be determined to be a position spaced outward by a set distance from that point.

[0442] For example, in FIG. 6, it can be assumed that the object to be highlighted is a rectangular area. The electronic device (100) can determine the upper, lower, left, and right boundaries of the rectangle based on the center coordinates, width, and height of the rectangle. Then, if an arrow points toward the rectangle from the bottom right, it can determine which point the direction first intersects with the lower or right boundary of the rectangle.

[0443] Even if an intersection point is determined, the endpoint of the arrow can be set to be located a certain distance outside of that intersection point. Accordingly, this prevents the arrow head from encroaching upon the interior of the target UI element or obscuring part of its content, while allowing the user to clearly recognize which target the arrow is pointing to.

[0444] Additionally, dotted line highlights and arrows can be used simultaneously. For example, a highlight border serves to show the user the entire scope of a target UI element, while an arrow can directionally indicate which element among multiple screen elements the user should focus on. Therefore, combining different types of visual guidance allows users to more easily identify the location and scope of the object they are currently manipulating.

[0445] The visual guidance shown in Fig. 6 is not inserted into the UI of the target application itself, but can be displayed through a transparent overlay placed above the target application. Since the area other than the guidance in the transparent overlay can remain substantially transparent, the user can view the original screen of the target application as is. At the same time, guidance objects such as highlight borders and arrows can be additionally displayed only where necessary.

[0446] In this case, the system can be configured so that the user can continue to interact with the target application even when a transparent overlay is displayed. For example, mouse input generated in the transparent area other than the guidance can be transmitted to the target application, allowing the user to directly click the actual button or tool indicated by the arrow. Consequently, the user can perform actual tasks while viewing guidance content on a single screen, without having to switch between the lecture screen and the work screen separately.

[0447] As a specific example, it can be assumed that the subtitle of the current step includes the content "Select the rectangle tool and drag in the workspace." The electronic device (100) can first provide the subtitle by replacing the text tokens included in the subtitle data with strings corresponding to the user's target language. At the same time, it can display a first visual guide at the actual location of the rectangle tool by looking up an abstract identifier corresponding to the rectangle tool in the location registry.

[0448] After it is confirmed that the user has selected the rectangle tool, the process proceeds to the next guidance step, and a dotted line highlight area and an arrow, as shown in Fig. 6, may be displayed at the target location of the work area or the location of the object to be created. In this way, whenever the guidance target changes in a single step or in a series of consecutive steps, the visual guidance can be dynamically changed using an abstract identifier and location information corresponding to the current target.

[0449] In addition, if the actual location of a UI element changes because the user moves the window of the target application or changes the screen scale, a screen re-recognition process is performed and the location registry is updated, after which the visual guide can be repositioned based on the updated coordinates. Therefore, the visual guide illustrated in FIG. 6 is not a static image applied only to a specific screen layout at the time of lecture production, but rather a guide whose location is determined in correspondence with the user screen state at the time of lecture execution.

[0450] As a result, according to the embodiment of FIG. 6, visual guidance such as highlights and arrows linked to the current lecture step can be directly superimposed and provided on the user screen of the target application that the user actually operates. Accordingly, the user can immediately check the correspondence between the target of work explained in the subtitles and the UI elements that must be operated on the actual screen, and learn how to use the software while maintaining the workflow of the actual target application without repeatedly checking a separate lecture video or explanation screen.

[0452] As described above, the overlay-based software usage guidance method according to one embodiment of the present invention is based on the fundamental technical concept of placing a transparent overlay on the screen of a target application that the user actually operates, and aligning the actual location of a UI element identified on the user screen with the lecture content at the time of execution to provide guidance.

[0453] Specifically, UI elements of a target application can be identified by comparing the user screen with pre-registered template images for each UI element, and a location registry can be generated by mapping abstract identifiers of the identified UI elements to on-screen coordinates. Subsequently, instead of directly specifying a specific resolution or screen coordinates, the lecture content can reference the guidance target using the aforementioned abstract identifiers, and at the time the actual guidance is provided, look up the corresponding coordinates from the location registry to display visual guidance, such as arrows, circular or rectangular highlight areas, or spotlights, corresponding to the actual location of the UI elements.

[0454] This structure separates the results of detecting UI elements on the actual screen from the logical reference information used during content authoring, thereby enabling guidance to be provided using the same lecture content even when the screen layout at the time of content creation differs from the actual user's screen layout.

[0455] In addition, the abstract identifier of the present invention is not limited to a key for simply querying the coordinates of UI elements, but can be used as a common connection standard to correspond lecture content to various user environments.

[0456] For example, a single abstract identifier can be used as a criterion for determining language-specific strings of text tokens included in subtitle data, and at the same time, can be commonly used as a criterion for selecting a template image corresponding to the language or execution environment in a screen recognition engine and as a criterion for verifying the actual display location of visual guidance in a location registry.

[0457] Accordingly, while maintaining semantic criteria for representing specific UI elements within the same lecture content, the actual display string, recognition image, and on-screen location can be determined at runtime based on the user's target language, operating system, version of the target application, display environment, etc.

[0458] Through this common usage structure of the same abstract identifier, subtitles, screen recognition, and guidance rendering can operate independently of each other while consistently referencing the same UI elements, and the reusability and maintainability of the content can be improved compared to cases where specific languages ​​or specific coordinates are directly fixed and stored in the lecture content.

[0459] The lecture content of the present invention is stored by being divided into multiple layers, such as lectures, chapters, and steps, and can be configured to retrieve, sort, and assemble necessary data based on the identification information of the target lecture at the time of actual lecture execution.

[0460] Each step may include subtitle data, visual guidance data, optional video or panel demo data, progress conditions, etc., and the actual lecture progression order may be determined based on the order of chapters and the order of steps within each chapter.

[0461] In addition, by providing only the list of identifiers of UI elements actually used in the target lecture as the screen recognition target, it is possible to selectively recognize UI elements necessary for the current lecture without detecting all UI elements present in the target application every time.

[0462] Accordingly, by separating the storage and management phase of lecture content from the execution phase in the actual user environment, it is possible to provide a structure that facilitates the addition, modification, language change, or UI element change of content.

[0463] In addition, the visual guidance of the present invention is not limited to a method of simply displaying graphics fixed at the center coordinates of a UI element, but its shape and display position can be dynamically determined based on the coordinates and size of the actually detected UI element.

[0464] For example, in the case of an arrow-type guide, the point where the direction from a reference position on the screen toward a target UI element intersects with the boundary area of ​​the target UI element can be identified, and a position spaced apart from that intersection point by a set interval can be determined as the endpoint of the arrow.

[0465] Accordingly, even if the size, position, or display scale of the target UI element changes, the arrow can maintain a form that points to the target UI element from the outside while reducing the phenomenon where the arrow passes inside the UI element or obscures the UI element itself.

[0466] In addition, a spotlight method that forms a translucent alpha layer over the entire or part of the user screen and makes only the area corresponding to the target UI element transparent, a method that generates haptic feedback through a wearable device when the mouse pointer enters the boundary of the target UI element, and a method that increases the size or emphasis of the visual guidance when the user's mouse movement pattern satisfies a set condition may also be optionally applied.

[0467] Furthermore, the present invention is not limited to a method of simply fixing and using a location registry initially created at the start of a lecture until the end of the lecture, but can evaluate the validity of the location registry based on changes in the user screen or user behavior and update it if necessary.

[0468] For example, the processor (120) calculates a screen re-recognition trigger index based on the current step's dwell time, the cumulative movement distance of the mouse pointer, the number of incorrect inputs for areas outside the target UI element, and the initial detection reliability at the time of creating the location registry, and if the index is greater than or equal to a set threshold, the screen recognition engine (430) can be automatically called to re-identify the location of the target UI element.

[0469] In this case, a logarithmic function is applied to dwell time and pointer movement amount to limit excessive influence from normal behavior, an exponential form is applied to repetitive erroneous inputs and low initial detection reliability, and an interaction term is applied to situations where erroneous inputs and low detection reliability occur simultaneously, thereby allowing multiple anomalies to be reflected non-linearly.

[0470] Accordingly, rather than performing re-recognition solely on the grounds that a set time has elapsed, the decision to re-recognize can be made by evaluating the actual user's search behavior and the reliability of existing location information together.

[0471] Accordingly, the above-described automatic re-recognition embodiment can form a closed feedback structure leading to guidance provision, observation of user behavior, determination of the possibility of location discrepancy, screen re-recognition, location registry update, and re-alignment of visual guidance by adding user behavior-based feedback judgment to this existing structure.

[0472] Furthermore, when the execution window of a target application moves between multiple displays, full-screen re-recognition of all UI elements is not strictly required; instead, the coordinates stored in the position registry may be transformed using the DPI scaling and monitor start offset coordinates of the moved display. The screen recognition engine itself also presents a structure that considers multi-monitor environments, display scaling, and the conversion between physical and logical coordinates.

[0473] Accordingly, if only the display coordinate system changes while the internal UI arrangement of the target application is maintained, the entire template matching can be omitted by arithmetically rearranging the stored coordinates; conversely, if the actual UI arrangement changes due to factors such as changes in window size, screen re-recognition can be performed. Through such selective processing, unnecessary screen recognition operations can be reduced while maintaining guidance consistency in a multi-display environment.

[0474] The transparent overlay of the present invention not only has the function of displaying visual guidance, but can also selectively process input events by region to enable simultaneous manipulation of the target application and the user interface of the overlay itself.

[0475] For example, an area where subtitles, lecture control buttons, or graphic objects requiring interaction with the user are displayed can be set as a first area where the overlay directly receives mouse events, and other parts corresponding to buttons, menus, or work areas of the actual target application can be set as a second area where input events pass through the overlay and are transmitted to the lower target application.

[0476] Accordingly, while the overlay remains visually maintained over the target application, the recipient of user input can be selectively changed between the overlay and the target application depending on the position of the pointer or the configuration of the current step.

[0477] Unlike cases where the entire overlay is simply fixed to either an input-enabled or input-inable state, this structure allows lecture control functions and actual software operation functions to be performed in parallel on the same screen.

[0478] Meanwhile, some UI elements or panels of the target application may be described using separately re-implemented panel demos rather than direct detection via screen recognition. For example, for docking panels whose positions change frequently or UI areas where it is difficult to reliably ensure the reliability of direct screen recognition, the state changes of the corresponding UI can be reconfigured into separate components and linked to lecture steps using common schedule data that includes start state, end state, transition delay time, transition method, and emphasis effects.

[0479] That is, the present invention may be implemented by including a panel demo framework configured so that different panels use a common schedule grammar to play effects such as highlights, pulses, or double-click simulations.

[0480] In addition, the multi-level menu of the target application is managed in a tree structure where each menu item can again contain a submenu, and by traversing the identifier path to the target menu, only the menu items necessary for the current stage can be selectively displayed, or for deterministically calculable submenus, coordinates can be calculated based on the item order and row height, etc.

[0481] Accordingly, different processing methods can be selectively applied to UI elements suitable for screen recognition and UI elements that can be efficiently guided by calculation or reimplementation.

[0482] Consequently, embodiments of the present invention are not limited to a technology for displaying a specific single overlay graphic on a screen, but can perform, in combination, processing for recognizing an actual user screen to determine the current location of a UI element, processing for linking the location of a UI element with multilingual content and visual guidance based on an abstract identifier, processing for hierarchically querying and assembling lecture data, processing for dynamically aligning guidance according to the shape and location of the actual UI element, processing for detecting changes in user behavior and execution environment during a lecture to update the guidance location or guidance intensity, and processing for selectively transmitting user input between the overlay and the target application.

[0483] Accordingly, users can receive step-by-step guidance directly on the screen of the target application they are actually operating without having to repeatedly switch between a separate lecture screen and the actual work screen, and can also respond to changes in the user's language, operating system, application version, display configuration, UI layout, and screen state that occur during the learning process.

[0484] According to the configuration of the present invention, a content author can create lecture content based on an abstract identifier even without knowing the specific user's screen coordinates or specific language environment in advance, and in the actual execution stage, can dynamically determine appropriate strings and visual guidance using location information and execution environment information obtained from the user screen.

[0485] In addition, since different compensatory measures such as default position, selective re-recognition, coordinate transformation, or alternative panel rendering can be applied even when screen recognition fails or the user environment changes, the continuity and stability of software usage guidance can be ensured while minimizing the interruption of the entire lecture due to partial recognition failures or UI changes.

[0486] Meanwhile, each embodiment and additional embodiment described above is illustrative for explaining the technical concept of the present invention more specifically, and the scope of the present invention is not limited to a specific target application, a specific operating system, a specific display configuration, a specific screen recognition algorithm, a specific template matching method, a specific database structure, a specific form of expression of an abstract identifier, a specific form of visual guidance, a specific wireless communication method, a specific wearable device, or a specific user behavior analysis method.

[0487] In addition, at least some of the configurations of the embodiments described above may be applied independently or in combination with one another, and within the scope of the technical concept of the present invention, it is possible to add, delete, substitute, combine, change the order of processing, or change to equivalent technical means of the components. Accordingly, it should be understood that the scope of protection of the present invention is determined not by the specific embodiments described above themselves, but by the technical concept described in the claims and the scope equivalent thereto.

Claims

Claim 1 An overlay-based software usage guidance method performed by an electronic device including a processor, wherein the processor acquires a user screen displaying a target application and identifies the on-screen location of at least one UI element included in the target application by comparing a template image corresponding to each of a plurality of pre-registered UI elements with the user screen; the processor creates a location registry by mapping an abstract identifier corresponding to each of the at least one UI element with coordinates representing the on-screen location; the processor acquires lecture-related data including the abstract identifier, visual guidance data, a text token, and locale data, identifies a string corresponding to the abstract identifier included in the text token and a target language set for the user from the locale data, and replaces the text token with the identified string; and the processor identifies target coordinates corresponding to the abstract identifier included in the visual guidance data from the location registry and displays visual guidance aligned with the UI element corresponding to the abstract identifier based on the target coordinates through a transparent overlay placed on the target application. Claim 2 An overlay-based software usage guidance method according to claim 1, wherein the lecture-related data includes lecture content including a plurality of steps, and each of the plurality of steps includes subtitle data and visual guidance data for guiding a task to be performed using the target application, the subtitle data includes the text token including the abstract identifier, and the visual guidance data includes the abstract identifier for specifying the UI element on which the visual guidance is to be displayed. Claim 3 In paragraph 2, the locale data comprises data in which a plurality of language-specific strings are mapped and stored corresponding to each of a plurality of abstract identifiers, an overlay-based software usage guide method. Claim 4 A method for providing guidance on the use of overlay-based software, wherein the operation of acquiring the lecture-related data comprises: the operation of the processor confirming a lecture identifier corresponding to a target lecture selected by the user; the operation of the processor querying data corresponding to a lecture, chapter, and step stored in a remote database using the lecture identifier; and the operation of the processor arranging the steps based on the order of the queried chapters and the order of the steps included in each chapter, and configuring and acquiring the lecture-related data to include the arranged steps. Claim 5 An overlay-based software usage guide method according to claim 1, wherein the visual guide includes an arrow pointing to the UI element, and the operation of displaying the visual guide includes the operation of the processor identifying a point where the direction of the arrow pointing toward the UI element intersects at the boundary of the UI element based on the target coordinates and the size of the UI element, and determining a position spaced apart from the intersecting point by a set interval as the endpoint of the arrow.

Images