Dual-track automation system and method with real-time task automation and background learning using vision language model

Abstract

The present invention relates to a dual-track automation system and method using a vision-language agent. The present invention comprises: a behavior monitoring unit (100) that collects behavior data by monitoring a user's screen operation and input behavior in real time; a vectorization unit (200) that vectorizes the collected behavior data using VIT and LLM; a vector database (300) that stores the vectorized behavior data; a training data repository (350) that stores VLM training data; an automation track unit (400) that searches the current user's behavior in the vector database (300) to find similar patterns and immediately generates and executes an automation script; and a background learning track unit (500) that improves long-term inference performance by fine-tuning a vision-language model (VLM) in the background using data accumulated in the vector database (300). The automation track satisfies the user's immediate automation needs, while the background learning track improves long-term performance by operating in parallel without exposing the user to learning delays. The present invention provides a dual-track automation system that solves the time delay problem of VLM learning through VDB-based execution, while continuously training the VLM in the background to progressively improve the quality of automation.

Description

Technology Field

[0001] The present invention relates to artificial intelligence-based business automation technology, and more specifically, to a dual-track automation system and method that performs immediate automation execution using a Vector Database (VDB) and VLM fine-tuning in the background in parallel to solve the problem of learning time delay of a Vision Language Model (VLM). Background Technology

[0002] Conventional business automation technology has evolved in two main approaches.

[0004] Traditional Robotic Process Automation (RPA) technology has been utilized to automate repetitive and standardized tasks. However, existing RPA technology has limitations, such as requiring script writing by experts, making it difficult for general users to utilize directly, and necessitating continuous script modifications due to the difficulty of adapting to new environments, including UI changes or system updates.

[0006] Recently, intelligent automation technologies utilizing Vision-Language Models (VLM) have emerged. VLM combines Vision Transformer (VIT) technology with Large Language Models (LLM) to possess the ability to understand the screen and perform tasks according to natural language instructions. For example, some prior art discloses automation systems using Vision-Language agents. Systems in these prior art receive natural language requests from users and formulate and execute task plans using a local VLM that combines the VIT and LLM.

[0008] However, VLM-based automation systems, including prior art, have the following fundamental limitations:

[0010] There is a time lag issue with VLM training. While general users intend to apply automation immediately after generating new job data, it takes a significant amount of time (from hours to days) for the VLM to learn from that data and achieve inference performance. In particular, VLM fine-tuning requires a sufficient amount of training data (at least several hundred samples) and must undergo processes such as self-instructed data augmentation, combined training of VIT and LLM, and Chain of Vision (CoV)-based validation. This training process requires GPU resources, and there is a problem in that users cannot utilize automation until training is complete.

[0012] There is a trade-off between the demand for immediate automation and the demand for long-term performance improvement. Users want to execute automation immediately upon the generation of work data, but since VLM training takes a long time, existing systems could not satisfy these two demands simultaneously. The aforementioned prior art systems could not provide automation until the VLM was sufficiently trained, or were forced to provide automation with low accuracy when training was insufficient.

[0014] There is a problem regarding user exposure to learning waiting times. When a user requests automation, if VLM learning is in progress, a message such as "Learning in progress, please wait a moment" is displayed, which significantly degrades the user experience. In particular, in work environments where immediate responses are expected, this waiting time acts as a critical disadvantage.

[0016] There is a problem with insufficient initial data. While a sufficient amount of training data is required to effectively train VLM, there is almost no data available when users first use the system, making VLM training impossible or inevitably resulting in very low performance. This means that automation cannot be utilized at all during the initial stages of system implementation.

[0018] There is a problem with wasting training resources. VLM fine-tuning requires expensive computing resources such as GPUs; performing training immediately whenever a small amount of data is added is inefficient, whereas waiting until sufficient data has accumulated prevents users from utilizing automation for an extended period.

[0020] Due to these limitations, VLM-based automation systems have limited utility in actual work environments, and there is a need for a new approach that can simultaneously satisfy users' immediate automation needs and long-term performance improvement needs. Another prior art is Korean registered patent No. 10-2737656. The problem to be solved

[0022] The problem that the present invention aims to solve is as follows.

[0024] It resolves the time lag issue of VLM learning, providing an immediate automation mechanism that enables users to execute automation without delay immediately after generating work data.

[0026] We provide a Dual-Track automation system that pursues immediate automation and long-term performance improvement in parallel. The Immediate Automation Track provides real-time automation to users, while the Background Learning Track progressively improves VLM inference performance without impacting the user experience.

[0028] It provides an immediate automation mechanism based on similar pattern search utilizing a Vector Database (VDB). The VDB enables fast vector search even without a GPU and allows for immediate automation with a small amount of data. It vectorizes behavioral data using Visual Intuition (VIT) and Language Models (LLM), identifies past similar behavioral patterns through searches based on cosine similarity or Euclidean distance, and instantly generates and executes automation scripts.

[0030] A mechanism is provided to continuously train the VLM in the background. Behavioral data accumulated in the vector database (300) is transferred to the training data repository (350), data is augmented using a self-instruct method, the VLM is fine-tuned by combining VIT and LLM, and performance is evaluated through Chain of Vision (CoV)-based verification. All of these processes are performed in parallel in the background without exposing the user to training latency.

[0032] A feedback-based performance improvement mechanism is provided. The execution results, success status, and user feedback of the automation script executed in the immediate automation track (400) are collected and reflected in the vector database (300) to update the weights of each behavior pattern. The weights of patterns with a high success rate are increased, and the weights of patterns with a high failure rate are decreased, and the updated weight information is also reflected in the VLM fine-tuning of the background learning track (500).

[0034] A model integration mechanism is provided. If the inference performance of the VLM fine-tuned by the background learning track (500) exceeds the vector search-based automation performance of the immediate automation track (400), the fine-tuned VLM is integrated into the immediate automation track (400) and utilized for generating automation scripts. Through this, the system naturally evolves from a VDB-based system to a VLM-based system over time.

[0036] A hybrid automation system is provided. It combines an immediate execution module (1040) that receives a natural language-based task request from a user, searches for similar past tasks in a vector database (1020), and executes them immediately, with a background learning module (1050) that fine-tunes the VLM in the background. The immediate execution module (1040) performs context-aware search by comprehensively considering the current screen context, running application, time zone, and user profile, and establishes an optimal automation plan by applying the execution success rate, recency, and user preference of the searched similar tasks as weights.

[0038] It provides a high-speed vector search mechanism utilizing the k-Nearest Neighbors (k-NN) or Approximately Nearest Neighbors (ANN) algorithms. It performs millisecond-scale fast searches even in large-scale behavioral pattern databases, enabling immediate response to real-time automation requirements.

[0040] An exception handling mechanism is provided. When an unexpected screen state is detected during the execution of an automation workflow, an alternative behavior pattern performed in a situation similar to the current screen state is searched in the vector database (300), and the searched alternative behavior pattern is applied to continue the operation.

[0042] It resolves the problem of insufficient initial data. VDB-based immediate automation can operate immediately with only a small amount of data (just a single job record), allowing automation to be utilized from the early stages of system implementation. Once sufficient data is accumulated, VLM learning begins in the background, improving performance over the long term. means of solving the problem

[0043] To solve the above problem, a dual-track automation system using a vision-language agent according to one aspect of the present invention comprises: a behavior monitoring unit (100) that collects behavior data by monitoring a user's screen operation and input behavior in real time; a vectorization unit (200) that vectorizes the collected behavior data using a Vision Transformer (VIT) and a language model (LLM); a vector database (VDB) (300) that stores the vectorized behavior data; a training data storage (350) that generates and stores training data for training a Vision Language Model (VLM); and an immediate automation track unit (400) that vectorizes the current user's behavior, searches for similar past behavior patterns in the vector database (300), and generates and immediately executes an automation script based on the behavior pattern if the similarity is above a preset threshold. The system includes a background learning track unit (500) that transmits behavior data accumulated in the vector database (300) to the learning data storage (350) and fine-tunes the vision-language model (VLM) in the background using the data stored in the learning data storage (350) to gradually improve the inference performance of the vision-language model (VLM); wherein the immediate automation track unit (400) executes automation without delay using the vector database (300) immediately after the user generates behavior data, and the background learning track unit (500) operates in parallel with the execution of the immediate automation track unit (400), and may include continuously training the vision-language model (VLM) in the background without exposing the user to a learning delay time to improve long-term automation performance.

[0045] In another embodiment of the present invention, the immediate automation track unit (400) searches for a similar pattern by calculating the cosine similarity or Euclidean distance between a vector of current user behavior and a past behavior vector stored in the vector database (300), and if the similarity of the searched similar pattern is greater than or equal to a preset threshold, it can generate and immediately execute an automation script based on a task sequence included in the similar pattern.

[0047] In another embodiment of the present invention, the background learning track unit (500) may include: a data transmission module (510) that transmits the behavior data to the learning data storage (350) when the number of behavior data accumulated in the vector database (300) exceeds a preset threshold; a data augmentation module (520) that expands the learning dataset by augmenting the behavior data stored in the learning data storage (350) in a self-instruct manner; and a model learning module (530) that fine-tunes the vision-language model (VLM) in the background using the expanded learning dataset.

[0049] In another embodiment of the present invention, the model learning module (530) can construct the vision-language model (VLM) by combining a Vision Transformer (VIT) and a Large Language Model (LLM), fine-tune the vision-language model (VLM) using the expanded learning dataset, and then verify the performance of the fine-tuned vision-language model (VLM) through a Chain of Vision (CoV) based verification method.

[0051] In another embodiment of the present invention, the system may further include a feedback management unit (600) that collects the execution results, success status, and user feedback of the automation script executed by the immediate automation track unit (400), and updates the weights of each behavior pattern by reflecting the collected information in the vector database (300). The feedback management unit (600) increases the weights of behavior patterns with a high execution success rate and decreases the weights of behavior patterns with a high execution failure rate, and the background learning track unit (500) can fine-tune the vision-language model (VLM) by reflecting the updated weight information.

[0053] In another embodiment of the present invention, the system may further include a model integration unit (700) that integrates the fine-tuned vision-language model (VLM) into the immediate automation track unit (400) and utilizes it for generating automation scripts when the inference performance of the vision-language model (VLM) fine-tuned by the background learning track unit (500) exceeds the vector search-based automation performance of the immediate automation track unit (400).

[0055] In another aspect of the present invention, a dual-track automation method using a vision-language agent comprises the steps of: collecting behavioral data by monitoring a user's screen operation and input behavior in real time; vectorizing the collected behavioral data using a screen recognition technology (VIT) and a language model (LLM); storing the vectorized behavioral data in a vector database (VDB) (300); vectorizing the current user's behavior and searching for similar past behavioral patterns in the vector database (300); and, if the similarity of the searched similar pattern is greater than or equal to a preset threshold, generating and immediately executing an automation script based on the corresponding behavioral pattern. The method includes a background learning step in which behavior data accumulated in the vector database (300) is transmitted to a learning data storage (350), and the vision-language model (VLM) is fine-tuned in the background using the data stored in the learning data storage (350) to gradually improve the inference performance of the vision-language model (VLM); wherein the immediate automation step executes automation without delay using the vector database (300) immediately after the user generates behavior data, and the background learning step is performed in parallel with the immediate automation step, and the vision-language model (VLM) is continuously trained in the background without exposing the user to a learning delay time, thereby improving long-term automation performance.

[0057] In another aspect of the present invention, a user behavior learning and automation system using a vision-language agent comprises: a monitoring module (810) that collects behavior data by monitoring user interactions including user screen manipulation, keyboard input, and mouse movements; an encoding module (820) that converts the collected behavior data into a vector representation using a screen recognition model (VIT) and a language model (LLM); a storage module (830) that stores the converted vector representation in a vector database (VDB) (300) and manages it in association with work metadata; a learning data management module (840) that converts the behavior data accumulated in the vector database (300) into learning data and stores it in a learning data repository (350); and a real-time automation engine (850) that, when a current user task is input, searches the vector database (300) to extract similar task patterns, creates an automation workflow based on the extracted similar task patterns, and executes it in real time. and a background model learning engine (860) that fine-tunes a vision-language model (VLM) in the background using data stored in the learning data repository (350), evaluates the performance of the fine-tuned model, and integrates it into the real-time automation engine (850); wherein the real-time automation engine (850) executes automation without delay based on vector search in response to the user's immediate automation request, and the background model learning engine (860) operates in parallel independently of the execution of the real-time automation engine (850), thereby improving long-term automation performance without exposing the user to learning delay.

[0059] In another embodiment of the present invention, the real-time automation engine (850) searches for similar task patterns in the vector database (300) using a k-nearest neighbor (k-NN) algorithm or an approximate nearest neighbor search (ANN) algorithm, and if the similarity score of the searched similar task patterns is greater than or equal to a preset threshold, it can create and immediately execute an automation workflow.

[0061] In another embodiment of the present invention, the background model learning engine (860) may include: a data augmentation submodule (861) that generates similar data by augmenting behavior data stored in the learning data repository (350) in a self-instruct manner; a model fine-tuning submodule (862) that forms a vision-language model (VLM) by combining a screen recognition device (VIT) and a language model (LLM) and fine-tunes the model using the augmented data; and a model evaluation and distribution submodule (863) that evaluates the performance of the fine-tuned model through Chain of Vision (CoV)-based verification and distributes it to the real-time automation engine (850) if the performance exceeds a threshold value.

[0063] In another embodiment of the present invention, the system may further include: an execution result collection module (870) that collects the execution result, success status, execution time, and user feedback of an automation workflow executed in the real-time automation engine (850); and a feedback processing module (880) that analyzes the collected execution result to update the behavioral pattern weights of the vector database (300) and feeds back to the background model learning engine (860) to reflect in model learning.

[0065] In another aspect of the present invention, a hybrid automation system combining a vision-language agent and a vector database comprises: an input interface (1010) for receiving a natural language-based task request from a user; a vector database (1020) in which past user behavior data is stored in a vector form; a training data repository (1030) in which data for learning a vision-language model (VLM) is stored; an immediate execution module (1040) that encodes the natural language task request into a language model, searches for similar past tasks in the vector database (1020), analyzes the execution history of the searched past tasks and the current screen state to establish an automation plan, and executes it immediately; and a background learning module (1050) that transmits data accumulated in the vector database (1020) to the training data repository (1030) and fine-tunes the vision-language model (VLM) in the background using the data in the training data repository (1030). and an integrated management module (1060) that feeds back the execution results of the immediate execution module (1040) to the vector database (1020) and evaluates the performance of the vision-language model (VLM) fine-tuned in the background learning module (1050) and integrates it into the immediate execution module (1040); wherein the immediate execution module (1040) responds in real time to the user's immediate automation requests through vector search, and the background learning module (1050) can continuously improve the model in the background without affecting the user experience.

[0067] In another embodiment of the present invention, the immediate execution module (1040) performs context-aware search by comprehensively considering the semantic similarity between the user's natural language request and past tasks stored in the vector database (1020), as well as the current screen context, running application, time zone, and user profile, and can establish an optimal automation plan by applying the execution success rate, recency, and user preference of the searched similar tasks as weights. Effects of the invention

[0068] The dual-track automation system and method utilizing VLM and applying a background learning technique for real-time task automation according to the present invention provide the following effects.

[0070] It fundamentally solves the problem of VLM learning delay. Unlike existing VLM-based systems that kept users waiting until learning was complete, the present invention allows automation to be executed immediately after the user creates work data through an immediate automation track (400) using a vector database (VDB). This dramatically improves the user experience and significantly enhances the practicality of the automation system.

[0072] It achieves immediate automation and long-term performance improvement simultaneously. The immediate automation track (400) provides immediate automation through VDB vector search, and the background learning track (500) improves inference performance over the long term by fine-tuning VLM in parallel. This dual-track approach resolves the "immediacy vs. performance" dilemma that existing systems could not solve.

[0074] The learning waiting time is not exposed to the user. Since the background learning track (500) operates in the background independently of the user's work, the user does not see messages such as "Learning, please wait" and always experiences immediate automation. This greatly improves user satisfaction.

[0076] Fourth, it resolves the problem of insufficient initial data. Since VDB-based immediate automation can operate immediately with just a single job record, automation can be utilized from the very first day of system implementation. This is a significant advantage compared to existing VLM systems, which could not be used until sufficient training data was secured.

[0078] The learning resources are utilized efficiently. The data transmission module (510) triggers learning only when the number of behavior data accumulated in the vector database (300) exceeds a preset threshold (e.g., 100), so learning is not performed inefficiently whenever a small amount of data is added. Additionally, since learning is performed in the background, GPU resources can be utilized during idle time.

[0080] The training data is expanded through self-instruct data augmentation. The data augmentation module (520) augments a small amount of seed data by more than 10 times to maximize the effect of VLM fine-tuning. This enables high-quality VLM to be trained even with limited data.

[0082] Continuous quality improvement is achieved through feedback-based performance improvement. The feedback management unit (600) collects the execution results of the automation script and increases the weight of patterns with a high success rate and decreases the weight of failed patterns. This feedback improves both VDB search accuracy and VLM learning quality.

[0084] It provides natural system upgrades through gradual model evolution. The model integration unit (700) automatically upgrades the system when the performance of the VLM fine-tuned in the background exceeds the VDB search performance. Users experience increasingly intelligent automation over time without any separate action.

[0086] The screen semantics are accurately understood through multimodal embeddings. The vectorization unit (200) converts the screen image into a vector via VIT (210), converts the text context into a vector via LLM (220), and combines the two (230) to generate an integrated action vector. This captures much richer semantic information than simple coordinate-based RPA.

[0088] High-speed searching is performed using the k-NN / ANN algorithm. The real-time automation engine (850) performs rapid searches in milliseconds even on a large-scale vector database (millions of behavioral patterns), immediately responding to real-time automation requirements. This is a speed impossible with traditional database searches.

[0090] Accuracy is improved through context-aware search. The immediate execution module (1040) selects the most suitable automation pattern by comprehensively considering the current screen state, running application, time zone, user profile, etc. This provides much higher accuracy than considering only simple vector similarity.

[0092] It provides resilience against exceptional situations. When the real-time automation engine (850) detects a screen state different from what was expected during automation execution, it searches for and applies alternative behavior patterns for similar exceptional situations from the vector database (300). This is a significant advantage compared to conventional RPA, which stops in exceptional situations.

[0094] VLM quality is guaranteed through CoV-based verification. The model training module (530) and the model evaluation and deployment submodule (863) strictly evaluate the performance of the fine-tuned VLM through Chain of Vision (CoV)-based verification and deploy only models whose quality has been verified. This prevents automation errors caused by low-quality models.

[0096] It provides optimal automation in a hybrid mode. The integrated management module (1060) configures a hybrid mode that utilizes fast VDB search for simple tasks and intelligent VLM for complex tasks, thereby applying the most suitable automation method for each task.

[0098] Immediate automation is possible even without a GPU. Since VDB vector search operates sufficiently fast using only a CPU, the system can be implemented without expensive GPUs. GPUs are used selectively only for background VLM training, which significantly reduces system implementation costs.

[0100] These effects are achieved by the organic combination and operation of the components of the present invention, such as the dual-track architecture, VDB-based immediate automation, background VLM learning, feedback-based improvement, and model integration. In particular, the present invention represents a groundbreaking advancement in that it resolves the "learning lag problem," a fundamental limitation of existing VLM-based systems, without sacrificing the long-term performance benefits of VLM. Brief explanation of the drawing

[0102] FIG. 1 is an overall configuration diagram of a dual-track automation system according to a first embodiment of the present invention. FIG. 2 is a parallel processing flowchart of an immediate automation track and a background learning track according to a first embodiment of the present invention. FIG. 3 is a detailed flowchart of an immediate automation track based on a vector database according to a first embodiment of the present invention. FIG. 4 is a diagram showing the VLM fine-tuning process of a background learning track according to the first embodiment of the present invention. FIG. 5 is a diagram showing a feedback-based performance improvement mechanism according to a first embodiment of the present invention. FIG. 6 is a diagram showing the structure and search mechanism of a vector database according to a first embodiment of the present invention. FIG. 7 is a diagram showing a model integration and performance comparison mechanism according to a first embodiment of the present invention. FIG. 8 is a diagram showing a data flow and a dual track timeline according to a first embodiment of the present invention. FIG. 9 is a diagram showing a system deployment architecture according to a second embodiment of the present invention. FIG. 10 is a drawing showing a hybrid automation system according to a third embodiment of the present invention. FIG. 11 is an overall flowchart of a dual-track automation method according to an embodiment of the present invention. FIG. 12 is a detailed flowchart of the vectorization step according to an embodiment of the present invention. FIG. 13 is a detailed flowchart of the background learning step according to an embodiment of the present invention. FIG. 14 is a flowchart of feedback collection and weight update according to an embodiment of the present invention. FIG. 15 is a flowchart of model performance evaluation and integration according to an embodiment of the present invention. Specific details for implementing the invention

[0103] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings. However, the technical concept of the present invention is not limited to some of the described embodiments but can be implemented in various different forms, and within the scope of the technical concept of the present invention, one or more of the components among the embodiments may be selectively combined or substituted.

[0105] In addition, terms used in the embodiments of the present invention (including technical and scientific terms) may be interpreted in a sense that is generally understood by those skilled in the art to which the present invention belongs, unless explicitly and specifically defined otherwise. Terms that are commonly used, such as terms defined in advance, may be interpreted in consideration of their meaning in the context of the relevant technology.

[0107] Furthermore, the terms used in the embodiments of the present invention are for describing the embodiments and are not intended to limit the present invention. In this specification, the singular form may include the plural form unless specifically stated otherwise in the text, and when described as "at least one of A and B and C (or more than one)," it may include one or more of all combinations that can be formed from A, B, and C.

[0109] In addition, terms such as first, second, A, B, (a), (b), etc., may be used when describing the components of the embodiments of the present invention. These terms are used merely to distinguish the components from other components and are not intended to limit the essence, order, or sequence of the components.

[0111] 1. Dual-track automation system

[0113] Referring to FIG. 1, the overall configuration of a dual-track automation system according to a first embodiment of the present invention is illustrated. The system includes a behavior monitoring unit (100), a vectorization unit (200), a vector database (300), a learning data repository (350), an immediate automation track unit (400), a background learning track unit (500), a feedback management unit (600), and a model integration unit (700).

[0115] (1) Behavior monitoring unit (100)

[0117] The behavior monitoring unit (100) collects behavior data by monitoring the user's screen operation and input behavior in real time. The behavior monitoring unit (100) may include a real-time monitoring module (110) and a behavior data collection module (120).

[0119] The real-time monitoring module (110) detects events such as a user's mouse click, keyboard input, screen transition, and application execution in real time using the event hooking mechanism of the operating system. In Windows operating systems, the SetWindowsHookEx API can be used, in macOS the CGEventTap API can be used, and in Linux the XInput extension can be used.

[0121] The behavior data collection module (120) converts events detected by the real-time monitoring module (110) into structured behavior data. The behavior data includes one or more of click coordinates (x, y), input text, name of the executed application, sequence of operations, time taken for operations, and screen capture images. For example, if a user clicks cell A1 in an Excel application and enters "Hello", the behavior data may be recorded as follows:

[0122] - Timestamp: 2024-01-15 10:30:15.123

[0123] - Application: Microsoft Excel

[0124] - Action: Click

[0125] - Coordinates: (250, 180)

[0126] - Subsequent action: Text input

[0127] - Input text: "Hello"

[0128] - Screen image: [Base64 encoded screenshot data]

[0130] (2) Vectorization unit (200)

[0132] The vectorization unit (200) converts behavior data collected by the behavior monitoring unit (100) into a vector representation in a high-dimensional vector space. The vectorization unit (200) includes a Vision Transformer (VIT) (210), a Large Language Model (LM) (220), and a vector combining module (230).

[0134] VIT (210) converts a screen capture image into an image embedding vector. VIT (210) divides the image into patches and processes each patch through a Transformer encoder to generate a high-dimensional vector representation. For example, if a screen image with a resolution of 1920x1080 is divided into 16x16 patches, a total of 8,100 patches are generated, and each patch can be encoded into a 768-dimensional vector. Finally, the output of the CLS (classification) token is used as a 768-dimensional image embedding vector representing the entire image.

[0136] The LLM (220) converts the user's text input and task context into text embedding vectors. The task context consists of text including the application name, window title, previous task history, etc. For example, if context text such as "Enter 'Hello' in cell A1 in Microsoft Excel - Sheet1.xlsx" is entered into the LLM (220), a 768-dimensional text embedding vector is output. The LLM (220) can use a pre-trained language model of the BERT, RoBERTa, or GPT family.

[0138] The vector combining module (230) combines the image embedding vector generated by the VIT (210) and the text embedding vector generated by the LLM (220) to generate an integrated action vector. Vector combining can be performed through concatenation, weighted average, or attention mechanisms. In one embodiment, a 1536-dimensional integrated action vector can be generated by concatenating a 768-dimensional image vector and a 768-dimensional text vector. In another embodiment, cross-attention can be applied to model the interaction between the image and the text, and a 768-dimensional integrated vector can be generated.

[0140] (3) Vector database (300)

[0142] The vector database (300) stores the integrated behavior vectors generated by the vectorization unit (200) and supports fast similarity search. The vector database (300) efficiently performs Approximate Nearest Neighbor search (ANN) in a high-dimensional vector space by utilizing indexing algorithms such as HNSW (Hierarchical Navigable Small World), IVF (Inverted File Index), and PQ (Product Quantization).

[0144] The vector database (300) stores metadata along with each action vector. The metadata includes the action ID, timestamp, application name, user ID, number of successful executions, number of failed executions, average execution time, weight value, etc. The weight value is initially set to 1.0 and is continuously updated by the feedback management unit (600).

[0146] The vector database (300) can be implemented using vector database engines such as Faiss, Milvus, Pinecone, Weaviate, and Qdrant. For example, if the Faiss library is used, search speeds in milliseconds can be achieved even with more than one million vectors by creating IndexHNSWFlat or IndexIVFPQ indexes.

[0148] (4) Training data repository (350)

[0150] The training data store (350) stores training data for fine-tuning a vision-language model (VLM). The training data consists of image-text pairs and action labels. For example, one training sample can be composed as follows:

[0151] - Input Image: [Screen Capture Image]

[0152] - Input text: "Microsoft Excel - Sheet1.xlsx, Select Cell A1"

[0153] - Output action: {"action_type": "click", "coordinates": [250, 180], "text_input": "Hello"}

[0155] The training data storage (350) can be implemented using a relational database (PostgreSQL, MySQL), a NoSQL database (MongoDB), or object storage (Amazon S3, MinIO). To efficiently store large-scale image data, images may be compressed (JPEG, WebP) or stored in object storage, and only image paths may be stored in the database.

[0157] (5) Immediate automated track unit (400)

[0159] Referring to FIGS. 1 and 3, the immediate automation track unit (400) searches the vector database (300) in response to the user's immediate automation request, finds similar past behavior patterns, generates an automation script, and executes it immediately. The immediate automation track unit (400) includes a current behavior vectorization module (410), a similar pattern search module (420), a threshold determination module (430), an automation script generation module (440), and an immediate execution module (450).

[0161] Referring to Fig. 3, the input processing step is a process of collecting and vectorizing user behavior data.

[0163] In the behavioral data collection step (S310), the user's screen operation and input behavior are monitored in real time to collect the following data:

[0164] - Screen Capture: Periodically captures a screenshot of the current screen (e.g., every second)

[0165] - Input Events: Event information such as mouse clicks, keyboard input, and drags

[0166] - Task Context: Name of running application, window title, file path, previous task history, etc.

[0168] In the VIT image embedding transformation step (S320), the collected screen capture image is converted into a 768-dimensional image embedding vector using a Vision Transformer (VIT). The image is preprocessed into 224×224 pixels, divided into patches, and input into the VIT.

[0170] In the LLM text embedding transformation step (S330), the collected text input and work context are transformed into 768-dimensional text embedding vectors using a Large Language Model (LLM). The text is tokenized and embedded through a pre-trained language model such as BERT or RoBERTa.

[0172] In the integrated action vector generation step (S340), the image embedding vector and the text embedding vector are combined to generate the final action vector. The combination method may use concatenation, a weighted average, or an attention mechanism.

[0174] In the vector database query step (S350), similar past behavior patterns are searched in the vector database (300) using the generated integrated behavior vector as a query. The query is executed in real time and guarantees a response time in milliseconds.

[0176] In the cosine similarity calculation step (S360), the similarity between the current action vector and the past action vector stored in the database is calculated. The cosine similarity is calculated as follows:

[0177] similarity(A, B) = (A · B) / (||A|| × ||B||)

[0179] Here, A is the current action vector, B is the past action vector, A · B is the dot product, and ||A|| and ||B|| are the vector magnitudes (norms). Cosine similarity ranges from -1 to 1, and values ​​closer to 1 indicate that the two vectors are more similar.

[0181] In the similar pattern extraction step (S370), the k-nearest neighbor (k-NN) algorithm is used to extract the k most similar past behavior patterns (e.g., k=5) through Top-K search. The search results are returned along with similarity scores, and metadata (operation ID, success rate, weight, etc.) is also provided.

[0183] In the similarity threshold determination step (S380), it is determined whether the highest similarity of the searched similar patterns is greater than or equal to a preset threshold (e.g., 0.85). The threshold is an important parameter that controls the accuracy of the system and the ratio of tasks that can be automated.

[0185] In the automation feasibility determination step (S390), if the similarity is above a threshold, the task is determined to be an automatable task and proceeds to the script generation step. At this time, the possibility of automation can be displayed to the user along with the confidence score.

[0187] In the automation non-determination step (S395), if the similarity is below the threshold, the task is determined to be a task that cannot be automated, and manual execution is waited for or the user is asked to confirm. In this case, a message saying "No similar pattern found" may be displayed to the user.

[0189] In the similar pattern analysis step (S400), the retrieved similar patterns are analyzed to evaluate the characteristics and performance of each pattern. The analysis elements include usage frequency, success rate, recency, weight, etc.

[0191] In the step of determining whether multiple patterns exist (S410), it is determined whether there are two or more similar patterns found. If multiple patterns exist, script generation through weighted average is required, and if there is a single pattern, the pattern is used directly.

[0193] In the weighted average script generation step (S420), when multiple similar patterns exist, a weighted average script is generated by considering the usage frequency, success rate, recency, and weight of each pattern. For example, if 5 similar patterns are found, and their respective similarities are [0.92, 0.91, 0.88, 0.87, 0.85] and their weights are [0.95, 0.90, 0.80, 0.75, 0.70], the final weight is calculated as the product of the similarity and the weight.

[0195] In the single pattern script generation step (S430), if there is only one similar pattern found, the automation script of that pattern is used directly. In this case, the script of that pattern is applied as is without additional weight calculation.

[0197] Automation scripts can be written in Python, JavaScript, or a Domain Specific Language (DSL). For example, a script in JSON format can be generated.

[0199] In the automation script execution step (S440), the generated automation script is passed to the execution engine to start execution. Before execution, the script is validated and necessary resources are prepared.

[0201] In the RPA action execution step (S450), the following actions are performed through an actual RPA engine (e.g., Selenium, Playwright, Puppeteer, or OS-level API):

[0202] - Click: Mouse coordinate click, element selection click

[0203] - Text input: Keyboard input, filling text fields

[0204] - Drag: Mouse drag, move elements

[0206] In the execution result monitoring step (S460), the success or failure of each action is monitored in real time, and if an error occurs, it is retried or an alternative pattern is applied. The execution result is transmitted to the feedback management unit (600) and used to improve system performance.

[0208] (6) Background learning track unit (500)

[0210] Referring to FIGS. 1 and 4, the background learning track unit (500) operates in parallel with the execution of the immediate automation track unit (400) and fine-tunes the vision-language model (VLM) in the background using behavioral data accumulated in the vector database (300). The background learning track unit (500) includes a data transmission module (510), a data augmentation module (520), and a model learning module (530).

[0212] The data transmission module (510) transmits the behavior data accumulated in the vector database (300) to the training data storage (350) when the number of behavior data exceeds a preset threshold (e.g., 100, 500, or 1000). Data transmission is performed in a batch manner and can be scheduled during periods of low system resource usage (e.g., night).

[0214] The data augmentation module (520) expands the training dataset by augmenting the behavior data stored in the training data repository (350) in a self-instruct manner. Self-instruct is a technique that automatically generates similar samples by modifying existing samples. For example, based on the sample "Enter 'Hello' in cell A1," similar samples such as "Enter 'World' in cell B2" and "Enter 'Test' in cell C3" can be generated. The data augmentation module (520) expands the original data by more than 10 times, enabling effective VLM fine-tuning even with limited data.

[0216] Data augmentation techniques may include the following:

[0217] - Parameter transformation: Change parameters such as coordinates, text, and color.

[0218] - Reorder tasks: Change the order of independent tasks

[0219] - Synonym substitution: Use synonyms in text input

[0220] - Context transformation: Performing the same task in a different application or situation

[0222] The model training module (530) fine-tunes the vision-language model (VLM) in the background using an extended training dataset. The model training module (530) constructs the VLM by combining a screen recognition device (VIT) and a language model (LLM). The VLM architecture can be configured as follows:

[0223] - Image Encoder: VIT (e.g., ViT-B / 16, ViT-L / 14)

[0224] - Text Encoder: LLM (e.g., BERT, RoBERTa, GPT-2)

[0225] - Multimodal Fusion Layer: Cross-attention or Gate Mechanism

[0226] - Action Decoder: A multi-head decoder that outputs action type, coordinates, and text.

[0228] Fine-tuning is performed in the following steps:

[0229] 1) Load pre-trained VIT and LLM

[0230] 2) Initialize the Multimodal Fusion Layer and Action Decoder

[0231] 3) Fine-tune the entire model end-to-end with the training dataset

[0232] 4) Loss Function: Action Type Classification Loss + Coordinate Regression Loss + Text Generation Loss

[0233] 5) Optimizer: AdamW, Learning Rate: 1e-5, Batch Size: 16, Epoch: 10

[0235] After fine-tuning is completed, the model training module (530) verifies the performance of the fine-tuned VLM through a Chain of Vision (CoV)-based verification method. CoV verification is a method of actually executing the action sequence generated by the VLM on a test dataset and checking whether the final result matches the target state. Verification metrics include action accuracy, task success rate, average execution time, etc.

[0237] Referring to FIG. 1, the CoV-based verification module (540) is a dedicated module for verifying the performance of the VLM fine-tuned in the model training module (530). The CoV-based verification module (540) checks whether the fine-tuned VLM can operate stably in an actual work environment, and if the performance does not meet the criteria, it feeds back to the model training module (530) to induce retraining.

[0239] The CoV-based verification module (540) performs the following verification process:

[0241] 1) As part of the test dataset preparation, 20% of the recently collected behavioral data is separated into a validation test dataset. The test dataset includes various task types (click, text input, drag, scroll, etc.) and applications (Excel, Word, browser, etc.).

[0243] 2) As a VLM inference execution, the VLM is made to generate an action sequence for each sample in the test dataset. The VLM receives a screen image and a task context as input and predicts the next action.

[0245] 3) As for actual execution and result verification, the action sequence generated by VLM is executed in the actual environment, and it is verified that the final result matches the target state. For example, when there is a goal of "Enter 'Hello' into cell A1 in Excel," verify whether the action sequence generated by VLM actually enters 'Hello' into cell A1.

[0247] 4) To calculate performance metrics, calculate the following metrics:

[0248] - Action Accuracy: The accuracy of the generated action (e.g., clicking at correct coordinates, entering accurate text)

[0249] - Task Success Rate: The percentage of total tasks successfully completed

[0250] - Average Execution Time: The average time taken to complete a task

[0251] - Error Rate: Frequency of errors occurring during execution

[0253] 5) As a verification criterion judgment, determine whether the calculated metric satisfies a preset threshold:

[0254] - Action accuracy ≥ 85%

[0255] - Job success rate ≥ 80%

[0256] - Average execution time ≤ 150% of existing VDB search time

[0257] - Error rate ≤ 5%

[0259] 6) As feedback processing, perform one of the following based on the verification result:

[0260] - Verification passed: Deliver the fine-tuned VLM to the performance evaluation module (710).

[0261] - Verification failure: Feedback to the model training module (530) to induce retraining parameter adjustment and additional fine-tuning.

[0263] The CoV-based verification module (540) evaluates the performance of the VLM from various angles to ensure the stability and effectiveness of the model before actual deployment. This maintains the quality of the automation service provided to the user and enables continuous performance improvement.

[0265] Referring to FIG. 6, the internal structure and search mechanism of the vector database (300) are configured as follows.

[0267] 1) Vector Repository and Metadata Management (S810-S880)

[0269] The vector storage is a core storage area that stores user behavior patterns as multidimensional vectors. Behavior vector 1 (810) is a vector representation of the first stored behavior pattern, which is an action such as "clicking cell A1 in Excel" encoded as a 768-dimensional vector. The vector values ​​are stored in the form v1 = (0.12, 0.34, ..., 0.89), and the weight 0.95 reflects the confidence and success rate of the corresponding pattern. Behavior vector 2 (820), behavior vector 3 (830), and behavior vector N (840) represent the second, third, and Nth behavior patterns, respectively, and all behavior patterns learned by the system are stored in vector form. Each vector has a unique weight value, which reflects the success rate and usage frequency of the corresponding pattern.

[0271] The task ID and basic metadata (850) store basic information associated with each behavior vector. This includes the task ID, timestamp, application name (e.g., Excel, Word, Chrome), user ID, etc. The execution history metadata (860) stores execution performance information for each behavior pattern and includes the execution history, number of successes, number of failures, average execution time, etc.

[0273] The HNSW index structure (870) is a high-performance vector search index using the Hierarchical Navigable Small World algorithm. HNSW is a graph-based approximate nearest neighbor search algorithm that guarantees millisecond search speeds even in large-scale vector data. The IVF index structure (880) is a cluster-based search index using the Inverted File Index, which reduces computational load by grouping vectors into clusters.

[0275] 2) Search Process (S890-S980)

[0277] The query vector (S890) is a vectorized representation of the current user's behavior, and is an integrated behavior vector generated through VIT and LLM. In the similarity calculation (S900), the cosine similarity between the query vector and all stored vectors is calculated. The similarity results per vector (S910-S940) represent the similarity with each stored vector, and in the Top-K search results (S950), the top K vectors (e.g., K=5) are selected in order of highest similarity.

[0279] In the similarity threshold determination (S960), it is determined whether the highest similarity is greater than or equal to a preset threshold (e.g., 0.85). In the automation execution determination (S970), if the similarity is greater than or equal to the threshold, the task is determined to be a task that can be automated, and an automation script is generated using the searched pattern. In the manual execution determination (S980), if the similarity is less than the threshold, the task is determined to be a task that cannot be automated, and manual execution is waited for or confirmation is requested from the user.

[0281] Through this vector database structure and search mechanism, the system can efficiently store and retrieve user behavior patterns, and guarantees fast responses for real-time automation.

[0283] Referring to Fig. 4, the background learning track shows the process of fine-tuning the VLM based on data accumulated in the vector database.

[0285] 1) Data processing and augmentation (S1000-S1070)

[0287] In the cumulative data threshold determination (S1000), it is checked whether the behavior data stored in the vector database is greater than or equal to a preset threshold (e.g., 10,000). If the threshold is reached, background learning is started, otherwise, a waiting state is maintained. In the cumulative behavior data extraction (S1010), behavior data required for learning is extracted from the vector database, and the extracted data includes image-text pairs and action type information. In the training data format conversion (S1020), the extracted data is converted into a format suitable for VLM learning, and in the transmission to training data storage (S1030), the converted data is transmitted to the training data storage (350).

[0289] In seed data analysis (S1040), the characteristics and distribution of the transmitted training data are analyzed to establish a data augmentation strategy. In self-instruct data augmentation (S1050), a self-instruct algorithm is used to generate new training data based on existing data, and action sequences corresponding to natural language instructions are automatically generated. In similar data generation (S1060), expanded data equivalent to 10 times the original data is generated through self-instruct, and in training dataset configuration (S1070), the original data and the augmented data are combined to form the final training dataset.

[0291] 2) Model training and validation (S1080-S1120)

[0293] In the VIT + LLM combined VLM configuration (S1080), a Vision-Language Model is constructed by combining a Vision Transformer and a Large Language Model. In the VLM fine-tuning with an extended dataset (S1090), the constructed VLM is fine-tuned using an extended dataset to learn the mapping relationships between image-text pairs and improve the ability to generate action sequences. In the Chain of Vision CoV-based verification (S1100), the performance of the fine-tuned VLM is verified using the CoV methodology, and the logical consistency and feasibility of the action sequences generated by the VLM are evaluated. In the determination of verification pass / fail status (S1110), it is determined whether the performance of the VLM satisfies the criteria based on the CoV verification results, and in the retraining parameter adjustment (S1120), if verification fails, retraining is performed by adjusting the learning rate, batch size, normalization parameters, etc.

[0295] 3) Feedback Integration and Model Deployment (S1130-S1190)

[0297] In the reflection of weight information (S1130), weight information transmitted from the feedback management unit is reflected in the learning process. In the reflection of execution success rate (S1140), the quality of the training data is evaluated by analyzing the execution success rate of each behavior pattern, and in the adjustment of learning priority (S1150), the learning priority is adjusted based on the weight and success rate information. In the completion of fine-tuning VLM (S1160), the VLM that has passed verification is completed, and in the performance comparison judgment (S1170), the performance of the fine-tuned VLM is compared with the existing VDB-based search performance. In the immediate integration of the model into the automation track (S1180), the VLM with superior performance is immediately integrated into the automation track, and in the waiting for the next training cycle (S1190), the next training cycle is waited for after the integration is completed.

[0299] Referring to Fig. 5, the feedback-based performance improvement mechanism shows a process of continuously improving system performance by collecting and analyzing the results of automated execution.

[0301] 1) Execution and Feedback Collection (S1200-S1240)

[0303] In the automation script execution (S1200), the generated automation script is executed to automate the user's tasks, and the success or failure of each action is monitored during the execution process. In the execution result generation (S1210), the results of the automation execution are generated, and the results include success / failure status, execution time, error information, etc. In the execution result collection (S1220), the automation execution results are collected and utilized for feedback analysis, and in the user feedback collection (S1230), explicit feedback (satisfaction, modification requests) and implicit feedback (changes in behavioral patterns) are collected from the user. In the success rate analysis (S1240), the collected execution results and feedback are analyzed to calculate the success rate of each behavioral pattern.

[0305] 2) Weight update and database update (S1250-S1310)

[0307] In the execution success determination (S1250), the success of each automation execution is determined. In the success pattern weight increase (S1260), the weight of successful patterns is increased, and the amount of weight increase is adjusted according to the success rate and recency. In the failure pattern weight decrease (S1270), the weight of failed patterns is decreased, and the amount of weight decrease is adjusted according to the failure rate and error severity. In the VDB pattern weight update (S1280), the calculated weights are reflected in the patterns of the vector database. The vector database (S1290) stores the updated weight information, and in the pattern-specific weight reflection (S1300), the weight of each pattern is reflected in the actual search process. In the search ranking adjustment (S1310), the search ranking is adjusted based on the weights so that patterns with superior performance are searched preferentially.

[0309] 3) Background learning reflection and performance improvement effects (S1320-S1370)

[0311] In Weight Information Transmission (S1320), updated weight information is transmitted to the background learning track. The background learning track unit (S1330) receives the transmitted weight information and reflects it in the learning process, and in Weight Application to VLM Fine-tuning (S1340), the weight information is applied to the VLM fine-tuning process to strengthen high-weight patterns and weaken low-weight patterns. In Immediate Automation Accuracy Improvement (S1350), the accuracy of immediate automation is improved through weight updates, and in VLM Learning Quality Improvement (S1360), the quality of the model is improved through VLM learning that incorporates feedback information. In User Satisfaction Increase (S1370), user satisfaction increases due to the improved performance.

[0313] Referring to Fig. 7, the model integration and performance comparison mechanism shows the process of selecting the optimal model by comparing the performance of a fine-tuned VLM with that of an existing VDB-based system.

[0315] 1) System Evaluation and Performance Comparison (S1400-S1470)

[0317] The Immediate Automation Track VDB Vector Search Base (S1400) indicates that the current system is using VDB-based vector search and is based on the current performance P_vdb. The Background Learning Track Completion (S1410) indicates that all processes of the background learning track have been completed, and the Fine-Tuned VLM Generation Completion (S1420) indicates that a fine-tuned VLM has been generated through background learning. In Test Dataset Preparation (S1430), a test dataset is prepared for comparing VLM and VDB performance, and in VLM Inference Performance Measurement (S1440), the inference performance P_vlm of the fine-tuned VLM is measured. In VDB Search Performance Measurement (S1450), the performance P_vdb of the existing VDB-based search is measured, and in Performance Comparison Analysis (S1460), the performance of VLM and VDB is compared and analyzed using various indicators such as accuracy, response time, and success rate. In the VLM and VDB performance comparison (S1470), it is determined whether the performance of VLM exceeds that of VDB based on the measured performance indicators.

[0319] 2) Model Integration and Deployment (S1480-S1570)

[0321] In VLM Deployment Preparation (S1480), the deployment of high-performance VLM is prepared, and in VLM Integration into Immediate Automation Track (S1490), VLM is integrated into the immediate automation track. In Hybrid Mode Configuration (S1500), a hybrid mode combining VDB and VLM is configured, where VDB handles fast responses and VLM handles complex decisions. In Gradual Transition (S1510), VLM is introduced gradually through a canary deployment method. In VLM Additional Training Plan Establishment (S1520), an additional training plan is established for VLM with insufficient performance, and in Additional Data Collection (S1530), additional data required for training is collected. In Waiting for Next Training Cycle (S1540), the next training cycle is waited for, and in VLM-based Immediate Automation (S1550), an immediate automation system integrated with VLM is established. In performance enhancement and accuracy improvement (S1560), performance enhancement and accuracy improvement are achieved through VLM integration, and in user experience improvement (S1570), user experience improvement is achieved through improved performance.

[0323] Referring to Fig. 9, the system deployment architecture shows the actual deployment environment of the dual-track automation system.

[0325] 1) Client and local system (S1600-S1750)

[0327] The user PC (S1600) is a personal computer where the user performs tasks, and the monitoring agent (S1610) is an agent that monitors the user's behavior and collects data. The execution agent (S1620) is an agent that executes automation scripts.

[0329] The vectorization module (S1630) converts user behavior into a vector, and the VDB search module (S1640) searches for similar patterns in the vector database. The script generation module (S1650) generates an automation script based on the searched patterns, and the execution module (S1660) executes the generated script. The vector database (S1670) stores behavior pattern vectors, and the training data repository (S1680) stores data required for VLM training.

[0331] The data augmentation submodule (S1690) performs data augmentation through Self-Instruct, and the model fine-tuning submodule (S1700) performs VLM fine-tuning. The model evaluation and deployment submodule (S1710) is responsible for model performance evaluation and deployment, and the execution result collection module (S1720) collects automated execution results. The feedback processing module (S1730) processes the collected feedback.

[0333] VLM fine-tuning acceleration (S1740) provides VLM fine-tuning acceleration using a GPU, and large-scale data augmentation processing (S1750) provides large-scale data augmentation processing using a GPU.

[0335] Referring to Fig. 10, the hybrid automation system shows a system that receives natural language input and performs automation combining VDB and VLM.

[0337] 1) Input Interface and Data Storage (S1760-S1800)

[0339] In the natural language task request (S1760), the user requests a task in natural language, e.g., in the form of "Open an Excel file and organize the data." In the current screen state (S1770), the screen state at the time of task execution is identified, and in the user profile context information (S1780), the user's task patterns and preferences are identified. In the vector database past behavior patterns (S1790), past behavior patterns are stored as vectors, and in the training data repository VLM training data (S1800), data necessary for VLM training is stored.

[0341] 2) Immediate execution and background learning (S1810-S1900)

[0343] In Natural Language Encoding LLM (S1810), natural language requests are encoded into LLM, and in Context-Aware Search (S1820), a search is performed considering semantic similarity, screen context, application, time zone, and user profile. In Similar Task Extraction Top-K Search (S1830), similar tasks are extracted using the Top-K method, and in Weight Application (S1840), weights are applied considering execution success rate, recency, and user preference. In Automation Plan Establishment (S1850), an automation plan is established based on the searched patterns, and in Immediate Execution (S1860), the established plan is executed immediately.

[0345] In the data accumulation monitoring (S1870), the data accumulation status is monitored, and in the training data transmission (S1880), the accumulated data is transmitted for training. In the VLM fine-tuning background (S1890), VLM fine-tuning is performed in the background, and in the performance verification (S1900), the performance of the fine-tuned VLM is verified.

[0347] 3) Integrated Management and Result Output (S1910-S1990)

[0349] In Collection of Execution Result Feedback (S1910), automation execution results and feedback are collected, and in VDB Update (S1920), the VDB is updated based on the collected feedback. In VLM Performance Evaluation (S1930), the performance of VLM is evaluated, and in VLM Performance vs. VDB Performance Comparison (S1940), the performance of VLM and VDB is compared. In VLM Deployment (S1950), the high-performance VLM is deployed, and in Hybrid Mode (S1960), a hybrid mode is configured where simple tasks are handled by VDB and complex tasks by VLM. In Automation Execution Completion (S1970), the automation task is completed, and in Execution Log and Report (S1980), logs of the execution process and result reports are generated. In User Feedback Collection (S1990), feedback is collected from the user.

[0351] (7) Feedback Management Unit (600)

[0353] Referring to FIGS. 1 and 5, the feedback management unit (600) collects the execution results of the automation script executed in the immediate automation track unit (400) and updates the vector database (300) to continuously improve the performance of the system. The feedback management unit (600) includes an execution result collection module (610) and a weight update module (620).

[0355] The execution result collection module (610) collects the execution results (success / failure), execution time, error information, and user feedback of the automation script. User feedback may include explicit feedback (e.g., the user clicking a "like" or "dislike" button) and implicit feedback (e.g., whether the user modified the automation result).

[0357] The weight update module (620) analyzes the collected execution results and updates the weights of each behavior pattern stored in the vector database (300). The weight update rules are as follows:

[0358] - On successful execution: weight += α (e.g., α = 0.1)

[0359] - On execution failure: weight -= β (e.g., β = 0.2)

[0360] - User positive feedback: weight += γ (e.g., γ = 0.3)

[0361] - User negative feedback: weight -= δ (e.g., δ = 0.5)

[0363] Weights are limited to a minimum value (e.g., 0.1) and a maximum value (e.g., 2.0). Patterns with very low weights (e.g., less than 0.2) may be automatically disabled or deleted.

[0365] The updated weight information is reflected in the vector database (300) and immediately affects the search and script generation of the automation track unit (400). Additionally, the weight information is also transmitted to the background learning track unit (500) to assign more learning weights to high-weight patterns and reduce the influence of low-weight patterns during VLM fine-tuning. This improves the quality of the training data and helps the VLM learn successful patterns better.

[0367] (8) Model Integration Unit (700)

[0369] Referring to FIGS. 1 and 7, the model integration unit (700) evaluates the inference performance of the VLM fine-tuned by the background learning track unit (500), and if the performance is sufficiently high, it immediately integrates it into the automation track unit (400). The model integration unit (700) includes a performance evaluation module (710) and a model deployment module (720).

[0371] The performance evaluation module (710) compares the inference performance (P_vlm) of the fine-tuned VLM with the VDB search-based automation performance (P_vdb) of the current immediate automation track. Performance metrics may include task success rate, action accuracy, average execution time, and user satisfaction.

[0373] The evaluation is performed on a test dataset (e.g., the last 100 jobs). The VLM and VDB search systems each perform the same test jobs, and the results are compared. For example, if the VLM job success rate is 92% and the VDB search system job success rate is 85%, the VLM is judged to have higher performance.

[0375] The model distribution module (720) immediately distributes the fine-tuned VLM to the automation track unit (400) when P_vlm > P_vdb. The distribution can be performed in the following manner:

[0377] 1) Full Replacement: VLM completely replaces VDB search and takes charge of generating all automation scripts.

[0378] 2) Hybrid Mode: Simple tasks (similarity > 0.9) are processed by fast VDB search, and complex tasks (similarity < 0.9) are processed by VLM.

[0379] 3) Canary Deployment: Initially, only 10% of the work is processed by VLM, and once performance is stable, the ratio is gradually increased to eventually reach 100%.

[0381] Even after deployment, the VDB search system is maintained as a backup and can be used as an alternative if VLM inference fails or takes too long.

[0383] 2. Parallel Processing and Timeline

[0385] Referring to FIGS. 2 and FIGS. 8, a second embodiment of the present invention describes the parallel processing and temporal flow of an immediate automation track and a background learning track.

[0387] Figure 2 is a sequence diagram showing the process in which immediate automation is executed from the moment a user performs screen manipulation, while VLM learning proceeds in the background simultaneously. The key point is that the two tracks operate independently; the immediate automation track responds in milliseconds without affecting the user experience, while the background learning track proceeds over several hours but is not exposed to the user at all.

[0389] Figure 8 shows the timeline of a dual-track system as a Gantt chart. For example:

[0390] - Day 1-3: Users generate work data, and the immediate automation track runs in real-time every day. The background learning track waits for data accumulation.

[0391] - Day 7-8: As accumulated data exceeds the threshold (100), the background learning track transmits the data and starts augmentation.

[0392] - Day 9-13: VLM fine-tuning is performed in the background. During this time, the immediate automation track continues to operate in real-time.

[0393] - Day 14-15: VLM performance verification and deployment take place.

[0394] - After Day 16: Immediately automated tracks integrated with VLM operate at higher performance.

[0396] This timeline shows that users can utilize automation from the first day of system implementation, and after two weeks, it is naturally upgraded to high-performance VLM-based automation.

[0398] 3. System Deployment Architecture

[0400] Referring to FIG. 9, a third embodiment of the present invention describes a deployment architecture of a dual-track automation system. The system consists of a client environment, a local system (on-premises), and a GPU server (optional).

[0402] A user PC, a monitoring agent, and an execution agent are installed in the client environment. The monitoring agent implements a behavior monitoring unit (100), and the execution agent implements an immediate execution module (450).

[0404] A real-time automation engine (850), a vector database (300), a training data repository (350), a background model training engine (860), and a management module are deployed in the local system. The real-time automation engine (850) implements an immediate automation track unit (400), and the background model training engine (860) implements a background training track unit (500).

[0406] GPU servers are deployed optionally and are used to accelerate large-scale VLM fine-tuning or massive data augmentation. Without GPU servers, VLM training is possible using only CPUs, but it takes longer. Alternatively, costs can be reduced by using cloud GPUs (e.g., AWS EC2 P3, Google Cloud GPU) only when needed.

[0408] 4. Hybrid Automation System

[0410] Referring to FIG. 10, a fourth embodiment of the present invention describes a hybrid automation system that supports natural language request processing. The system includes an input interface (1010), a vector database (1020), a learning data repository (1030), an immediate execution module (1040), a background learning module (1050), and an integrated management module (1060).

[0412] The user inputs a natural language-based task request through the input interface (1010). For example, a request such as "Open an Excel file and organize the data in column A."

[0414] The immediate execution module (1040) encodes the natural language request into LLM and searches for similar past operations in the vector database (1020). When searching, it does not simply consider text similarity, but comprehensively considers the current screen context (which application is running, which file is open), the running application, the time of day (business hours vs. night), and the user profile (user preferences, past behavior patterns). This is called Context-Aware Search.

[0416] Execution success rate, recency (higher weight for recently performed tasks), and user preference (patterns frequently used by users) are applied as weights to the retrieved similar tasks to establish an optimal automation plan. For example, even for the same "data cleaning" task, if User A prefers sorting and User B prefers filtering, an automation plan customized for each user is provided.

[0418] The background learning module (1050) operates in the same way as the background learning track unit (500) of the first embodiment and fine-tunes the VLM using accumulated data. The integration management module (1060) feeds back the execution results of the immediate execution module (1040) to the vector database (1020), evaluates the performance of the fine-tuned VLM, and integrates it into the immediate execution module (1040).

[0420] 5. Additional Examples and Variations

[0422] Various modifications and additional embodiments of the present invention are possible:

[0424] 1) Distributed System: By integrating behavioral data from multiple users into a central vector database and leveraging collective knowledge, the quality of automation across the entire organization can be improved.

[0426] 2) Privacy Protection: By applying the Federated Learning technique, sensitive data is not transmitted to a central server, and each client can train a local model and share only the model parameters.

[0428] 3) Domain Specialization: By pre-training VLMs specialized for specific domains (e.g., accounting, medical, legal), the accuracy of domain-specific automation can be improved.

[0430] 4) Multimodal expansion: Additional input modalities such as voice commands, gesture recognition, and eye tracking can be integrated to support a wider variety of automation scenarios.

[0432] 5) Explainable AI: It can improve reliability by explaining the process of generating automation scripts to the user and providing information such as, "This task is 92% similar to pattern #123, which has shown a 95% success rate in the past."

[0434] FIG. 11 visually illustrates the overall operation flow and key processes of the "dual-track automation system" presented in this specification. This dual-track structure is differentiated from the existing single-track method in that real-time automation that responds immediately to user requests and a learning track that continuously improves system performance in the background are efficiently operated in parallel. This figure consists of steps S100 to S600, each step operating in conjunction with other figures (particularly FIG. 12) and the details of the entire specification.

[0436] First, Step S100 concerns the "real-time monitoring and collection of behavioral data," capturing all operation events—such as clicks, drags, text input, and screen switching performed by a user on a PC or other terminal screen—using operating system-level event hooking techniques (e.g., Windows API hooking, Accessibility API, etc.). The data obtained through this process consists of a wide variety of multimodal information, including coordinate data, text content, application information of the active window, the sequence of operations and time taken for each segment, and real-time captured screen images. This rich data structure directly contributes to the quality of vector transformation and automation in subsequent steps.

[0438] Next, Step S200 is responsible for the "multimodal vectorization of behavioral data," and this specification describes in detail the specific operating principles and technical implementation of this process based on FIG. 12. FIG. 12 intuitively illustrates the structure and flow of the multimodal vectorization step, which is a core preprocessing step of the dual-track automation system proposed in this specification and forms the foundation of the entire process.

[0440] First, in S210, images captured in real-time at the moment of a user's actual screen operation are converted into high-dimensional image embedding vectors of 768 dimensions or more using an advanced deep learning-based image recognition model such as Vision Transformer (VIT). At this time, VIT is a large-scale network that has completed pre-training, ensuring robust visual representation even with various UIs and situational variability. These screen image embeddings contain contextual information such as the user's operation location, the type of open software, and the state of UI elements, providing foundational data for precise pattern recognition in subsequent steps. As mentioned in S100 of the specification, since the source data has rich multimodal characteristics such as coordinates, active windows, and real-time screen information, the quality of the image embedding stage has a significant direct and indirect impact on the overall system performance.

[0442] Next, in step S220, the user's input text (e.g., command, chat, search term, etc.) and contextual information of each action (whether it is an explicit / implicit command, target UI elements, meaning of gestures, purpose by situation, etc.) are quantified into text embedding vectors of the same (or specified) dimension using a large language model (LLM, e.g., GPT, BERT family, etc.). Throughout this specification, the importance of linguistic behavioral context and the various potential applications of LLM are mentioned, and in FIG. 12, LLM performs the role of precisely quantifying and abstracting not only text embeddings but also the higher-level purpose and detailed behavioral intentions required by the user. For example, even for the same click action, the result of the text embedding is designed to vary depending on the context of the command, such as "Save," "Close," or "Search," and the location of the screen UI.

[0444] These image vectors and text vectors are combined into a single integrated (multimodal) behavior vector in step S230. The method to be used in this combination process is covered by various technical details in this specification, and depending on the implementation purpose and performance requirements, various techniques such as simple concatenation, weighted average, or attention mechanism-based multimodal neural network structures may be selected. FIG. 12 clearly visualizes the combination structure and processing flow of each vector. In particular, attention-based combination dynamically adjusts the importance of the two inputs, image and text, to more precisely vectorize complex user behavior patterns. This multimodal behavior vector is stored in a vector database in the subsequent steps S300–S400 and determines the performance of core functions such as real-time search, similarity comparison, and automated command mapping (described in detail in Claims 1, 2, 4, etc.).

[0446] Furthermore, throughout this specification, the vectorization step is consistently mentioned as an essential innovative element of the dual-track system, such as the preservation of semantic information of behavioral data, pattern generalization, and the generation of flexible automation commands. The multimodal embedding and combining algorithm, structured in detail in FIG. 12, allows screen images and text inputs to complement each other and contain high-dimensional semantic information, thereby enabling high search accuracy and learning effectiveness in both the real-time automation track (steps S400–S500) and the background learning track (steps S600–S660). Additionally, this step serves as the starting point for a data-driven infrastructure for subsequent feedback management, weight updates, and long-term performance improvement, and is closely linked to all major technical flows within the specification.

[0448] In the S300 stage, the acquired behavior vectors and various metadata (job ID, timestamp, application name, user identification ID, weight, success / failure history, etc.) are stored in a vector database (VDB, 300). It can be seen from the full specification that this database is designed to serve as a foundational infrastructure for real-time search, automation execution, feedback management, and model training, going beyond the role of a simple log.

[0450] Next, step S400 is a vector-based search for similar behavioral patterns. Regarding the current user's behavior, the previously stored vector database is searched by applying representative vector similarity calculation techniques, such as cosine similarity and Euclidean distance. In the implementation, the k-nearest neighbor (k-NN) search method is utilized to rapidly select the k past behavioral patterns most similar to the task the actual user is currently attempting. This serves as the core basis for the recommendation and generation of automated scripts (closely related to claims 2, 4, etc. of this specification).

[0452] Based on these search results, the S500 step immediately generates and executes an automation script based on the corresponding historical pattern if the similarity is above a predefined threshold (e.g., cosine similarity of 0.85 or higher). If the threshold is not met, the system suspends the execution of the automation and waits for manual approval by an administrator or an additional learning step. In other words, it guarantees that operation can be performed safely based on reliability.

[0454] The final S600 step is directly connected to the "background learning track" (see FIG. 13 and the relevant description in the preceding specification). Multiple behavioral data accumulated in the vector database are periodically (e.g., at a certain quantity or at regular intervals) transferred to the learning data repository (350), and the vision-language model (VLM) is fine-tuned in the background using this data. Since this step is executed independently of or in parallel with S500 (immediate automation track), the user does not directly experience the learning delay for improving automation quality. This structure, together with FIG. 13, illustrates the essential features of the two-track automation system of this specification.

[0456] Accordingly, FIGS. 11 and 12 intuitively reveal the multimodal data flow, information processing, real-time / continuous learning structure, and organic step linkage of the entire dual-track automation system, and are closely related to the technical features of each part of the specification.

[0458] FIG. 13 illustrates in detail, step by step, the overall structure of the background learning track, which is the core of the present invention, and how each detailed step is organically connected to enhance system performance over the long term. In the dual-track structure emphasized throughout this specification, the background learning track operates independently of the immediate automation track, while playing an important role responsible for the intelligent evolution of the entire system.

[0460] In step S610, the first step of FIG. 13, it is determined whether the number of behavior data accumulated in the vector database (300) is greater than or equal to a preset threshold (e.g., 100). This threshold is designed to balance the learning efficiency of the system with periodic performance improvement, and is a value that takes into account both the speed of data accumulation in the actual usage environment and the resource constraints of the learning infrastructure. As described in the preceding figures (steps S100 to S300, see FIG. 11 to 12), the vector database aggregates multidimensional behavior vectors and metadata in real time, serving as a core infrastructure that supports both automation and learning beyond simple logs.

[0462] In step S620, the behavior vectors and related metadata accumulated in the vector database under the threshold conditions satisfied in S610 are transferred to the training data repository (350). This process may be accompanied by preprocessing such as quality control, deduplication, and anonymization of the entire data, thereby improving the reliability and generalization ability of the training dataset. Throughout the specification, it is emphasized that the quality of the training data is directly linked to the mid-to-long-term performance of the automation system.

[0464] In the S630 stage, the raw behavioral dataset is expanded by more than tenfold by introducing a self-instruct data augmentation method, which is receiving attention in recent AI and machine learning research. Specifically, data diversity and comprehensiveness are enhanced by generating virtual scenarios similar to the collected behavioral data or data with various context variations. This enables robust VLM learning against scene variations, UI changes, and contextual variability in real-world usage environments. As mentioned in several chapters of this specification (e.g., multimodal vectorization, pattern generalization), it can be reaffirmed that data augmentation is a key means for securing the generalization performance of the model and enhancing long-term learning quality.

[0466] In the S640, an integrated Vision-Language Model (VLM) is constructed through the combination of VIT and LLM. VIT converts the user's actual screen image into a high-dimensional image embedding, and LLM converts the textual context of an action into a text embedding; these two vectors are then fused into an integrated action vector through the multimodal combination technology of the present invention (see FIG. 12). The VLM prepared in this way is designed to exhibit significantly improved reasoning capabilities compared to existing single-modal models in understanding actions, interpreting complex commands, and generating automation scripts.

[0468] In step S650, the VLM is fine-tuned end-to-end using the previously expanded and augmented multimodal learning dataset. This specification describes in detail in several examples that various loss functions, such as action type classification, coordinate regression, and text command generation, are combined, and an optimal learning environment is applied, such as a recent standard optimizer (e.g., AdamW), a reasonable learning rate (1e-5), and appropriate batch size and epoch settings.

[0470] Finally, the S660 thoroughly evaluates the fine-tuned VLM performance through a Chain of Vision (CoV)-based systematic verification method. CoV verification goes beyond simple static evaluation (e.g., accuracy calculation) to include all practical automation quality indicators, such as whether automation commands are executed correctly in actual test environments, whether the user's target state is achieved, task success rates, and average execution times. The introduction of such rigorous verification demonstrates that this invention is not merely a model development, but a commercial-grade automation platform development that encompasses stability and reliability in real-world usage environments.

[0472] In summary, each step (S610–S660) of FIG. 13 systematically links all elements of the behavioral data collection and vectorization, database management, learning data quality management, multimodal combination technology, and automation track and feedback management of the preceding specification to complete the long-term learning and performance improvement cycle of the dual-track automation system.

[0474] FIG. 14 is a step-by-step diagram illustrating the entire process of "feedback collection and weight update" in the dual-track automation system proposed by the present invention, which plays a key role in the performance maintenance and improvement cycle of the entire system. This figure is closely linked to the real-time behavior data collection, multimodal vectorization, real-time automation, and background learning tracks described in the previous chapter (Figs. 11, 12, 13, etc.), clearly visualizing how the behavior and feedback of actual users are reflected and circulated within the system.

[0476] First, Step S710 is a process of collecting in real-time the actual execution results, success or failure status, and additional feedback provided by the user (e.g., positive / negative evaluations, comments, etc.) regarding scripts automatically generated and executed according to the user's work requirements in the "Immediate Automation Track." These feedback and execution logs are not merely for the purpose of deriving statistics, but are directly linked to data-driven performance improvement, which forms the basis of the present invention. For example, if a user indicates "success" immediately after executing a script and provides additional positive feedback, that behavioral pattern is systematically accumulated as a success story. Conversely, if user dissatisfaction or error reports are collected, the system recognizes them as failure or low-quality patterns. One of the differentiating features of the dual-track structure is that real-time user interaction (operational real-world feedback) is immediately reflected in the quality of the automation service in this step (see Claims 2, 5 and the general specification).

[0478] In step S720, the execution results and feedback data aggregated in S710 are analyzed algorithmically to calculate the weight change of each behavior pattern stored in the vector database (300). Specifically, when the execution of a behavior pattern is successful, the weight is increased by a set ratio (α), and when it fails or negative feedback is collected from the user, the weight is decreased by a larger amount (β, δ) (see "Weight Update Rules" above the specification). In addition, by correcting positive feedback (γ) as a separate item, not only simple action success / failure but also user trust and satisfaction are reflected in a complex manner. This process is distinguished from the simple failure count of existing RPA systems and represents a technical advancement of the present invention in that it enables detailed tracking and learning of various contextual successes / failures in the actual usage environment.

[0480] In step S730, the newly calculated weight information from the previous analysis results is immediately reflected in the vector database (300) in real time. At this time, the latest weights for each pattern provide priority to the search and recommendation engine, so that patterns with a higher probability of success are suggested first when matching similar patterns with subsequent user behavior, or patterns with a high failure rate are excluded or deactivated from automated recommendations. This step serves as the technical basis for the "real-time reflectivity" and "user-centric quality control" repeatedly emphasized in the overall specification (claims 6, 7, etc.). Additionally, patterns whose weights have dropped below a threshold (e.g., less than 0.2) are deactivated or removed from the vector DB according to the data cleansing / automatic deletion policy, thereby maintaining the quality, efficiency, and security of behavior patterns within the system.

[0482] Step S740 is directly linked to maintaining the system's long-term and evolutionary automation quality. In this step, updated weights from the vector database (i.e., values ​​reflecting actual user performance and reliability-based quality information) are automatically transferred to the "background learning track." This weight information is designed so that during subsequent Vision Language Model (VLM) fine-tuning, more learning weight (weighted sampling, loss function weight, etc.) is assigned to high-weight patterns (behaviors with actual success and high user satisfaction). As a result, the VLM learns practically successful and reliable behavioral patterns more intensively, thereby improving the reliability and success rate of inference / automation script generation in the long term (refer to Fig. 13 and the overall background learning structure description). Furthermore, the cyclic structure of Fig. 14, which extends from feedback to weights to learning, demonstrates that it intensively embodies the essential technical effect of the present invention, which is "real-time user experience-performance-oriented automation enhancement."

[0484] FIG. 15 is a key diagram structurally illustrating the model performance evaluation and system integration decision-making processes in the overall operation cycle of the dual-track automation system of the present invention. This diagram is closely linked to FIG. 11 (overall system structure), FIG. 12 (multimodal vectorization and database structure), FIG. 13 (background learning track), and FIG. 14 (feedback and weight update), and comprehensively demonstrates in the final stage how real-time automation and learning-based performance improvement organically circulate in the field.

[0486] In particular, in step S810 of Fig. 15, a test dataset that fully reflects the actual usage environment is constructed to systematically evaluate highly reliable performance, and key metrics to be used for evaluation (e.g., accuracy, F1 score, average response time, success rate, user satisfaction, etc.) are systematically set in advance. This is linked with realistic automation quality indicators used in the previous steps S660 (CoV-based practical evaluation) and feedback collection (see Fig. 14), so that the evaluation is designed to comprehensively reflect actual business automation success rates and user-perceived efficiency, rather than being limited to simple experimental comparisons.

[0488] In the S820 stage, the inference performance of the newly fine-tuned Vision Language Model (VLM), which has undergone the preceding background learning track (Fig. 13, S650 fine-tuning, etc.), is precisely measured. The evaluation criteria include representative automation quality indicators such as the integrated understanding of multimodal input (screen image + behavioral context text) and pattern generalization performance, the success rate of generating actual automation commands, and adaptability to various UI / business contexts. This multi-faceted evaluation is closely linked to the technical effects of "high-dimensional semantic-preserving multimodal vectorization," "real-time / continuous learning," and "real-world suitability," which are emphasized throughout this specification.

[0490] In the S830 phase, the real-world inference performance of the new VLM is cross-analyzed with the search and matching performance of the Vector Database (VDB) used in the existing Immediate Automation track. For example, the strengths and weaknesses of the real-time nature and rapid recommendation quality of VDB’s fast nearest neighbor search, and the high-dimensional semantic generalization and adaptability of VLM, are analyzed from various angles using specific examples (e.g., introduction of new UI, processing of complex commands, incorporation of user feedback, etc.). This phase goes beyond simple numerical comparisons and is directly linked to the goal of improving the actual performance and enhancing the reliability of the entire automation system.

[0492] Step S840 is a decision-making stage that determines whether to integrate or transition models from a system operation perspective, based on the comparative analysis results derived from S830. If VLM demonstrates superior results in areas such as generalization, complex context awareness, and user satisfaction—where existing VDB-based search alone has limitations—it is decided to gradually integrate and transition the system structure to a VLM-centric automation flow. Conversely, if the existing search-based approach remains valid or is more efficient in specialized domains, the current method may be maintained or a hybrid structure may be continued.

[0494] In the S850, if evaluation results indicate that the VLM clearly exceeds the existing VDB search performance (based on key indicators such as automation success rate, response speed, and complex behavior processing capability), the newly fine-tuned VLM is immediately integrated into the operational automation track (immediate automation phase). At this time, the system configuration is updated so that the VLM-based automation engine generates and executes automation commands more accurately and flexibly in response to the user's latest behaviors. This is the step that serves as the core basis for the realization of the "continuous self-evolving platform" presented throughout this specification.

[0496] On the other hand, in S860, if the performance of the latest VLM falls short of expectations or fails to secure a clear advantage over existing methods, immediate integration is withheld, and subsequent improvement and verification processes, such as additional learning, data augmentation, and hyperparameter adjustment, are planned. This process is organically connected to the automated quality feedback-weighting-relearning loop highlighted in Figures 13 and 14, meaning that data from the actual user environment is continuously utilized to improve system performance.

[0498] Finally, in the S870 stage, following the practical application of the integrated VLM-based automation system, various performance indicators (automation success rate, average processing time, usage frequency, qualitative / quantitative user feedback, etc.) are continuously monitored to quantitatively evaluate the actual effects of the model integration. User satisfaction measurement results and field performance data are fed back into the vector database and background learning track to be utilized for maintaining the system's long-term quality and evolving its performance.

[0500] As such, the entire process of FIG. 15 is a final stage closely connected to all technical flows of the present specification, and naturally leads to clinical / quantitative verification of the integrated model in a real-world usage environment and the establishment of an optimized operational structure through real-time collection of multimodal behavioral data (S100~S200), advanced vectorization and combination (S210~S230, FIG. 12), performance evaluation / feedback-based quality control (Fig. 14), and background iterative learning / fine-tuning (Fig. 13).

[0502] Consequently, the dual-track automation methodology of the present invention allows users to experience immediate automation services while continuously enhancing the quality of automation over the long term through advanced AI models, such as VLM, which are systematically learned and evaluated in the background. The immediate automation track is responsible for real-time, fast similarity search and immediate automation execution based on a vector database, while the background learning track is responsible for iterative learning and model enhancement based on actual environment data, feedback, and weight changes. Thanks to the evaluation and integration structure of FIG. 15, the system maintains a high degree of flexibility and self-evolution capabilities, and can be rapidly adapted to changes in the field environment and user requirements, as can be confirmed in conjunction with the entire specification.

[0504] The embodiments according to the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded on a computer-readable medium. In this case, the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Furthermore, it may include hardware devices for accelerating the computation of artificial intelligence models, such as a graphics processing unit (GPU), a tensor processing unit (TPU), and a neural network processing unit (NPU).

[0506] Meanwhile, the above-mentioned computer program may be one specifically designed and configured for the present invention, or one known and available to those skilled in the art of computer software. Examples of computer programs may include not only machine code, such as that generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc. Furthermore, all or part of the functions of the present invention may include, or be implemented in conjunction with, an artificial intelligence model, machine learning model, deep learning model, foundation model, or large language model (LLM) that is pre-trained or fine-tuned for a specific purpose. Such models may be called through a service in the form of an Application Programming Interface (API) and may be utilized for the generation, analysis, transformation, or control of data to perform the functions of the present invention.

[0508] In the specification of the present invention (particularly in the claims), the use of the term "above" and similar descriptive terms may be in both singular and plural. Furthermore, where a range is described in the present invention, it is to include an invention to which individual values ​​belonging to said range are applied (unless otherwise stated), and this is equivalent to describing each individual value constituting said range in the detailed description of the invention.

[0510] Unless explicitly stated or contrary to the order of the steps constituting the method according to the present invention, said steps may be performed in a suitable order. The present invention is not necessarily limited by the order in which said steps are described. The use of all examples or exemplary terms (e.g., etc.) in the present invention is merely for the purpose of describing the present invention in detail, and the scope of the present invention is not limited by said examples or exemplary terms unless limited by the claims. Furthermore, those skilled in the art will understand that various modifications, combinations, and changes, including the application of newly emerging technical means, may be configured according to design conditions and factors within the scope of the claims or equivalents to which they are added.

[0512] Accordingly, the scope of the present invention should not be limited to the embodiments described above, and all scopes equivalent to or equivalently modified from the claims set forth below, as well as the claims set forth below, shall be considered to fall within the scope of the concept of the present invention. Explanation of the symbols

[0514] 100: Behavior Monitoring Unit 110: Real-time monitoring module 120: Behavioral data collection module 200: Vectorization Unit 210: VIT (Vision Transformer) 220: LLM (Large Language Model) 230: Vector Combination Module 300: Vector Database (VDB) 350: Training data repository 400: Immediate Automation Track Unit 410: Current Behavior Vectorization Module 420: Similar Pattern Search Module 430: Threshold determination module 440: Automation Script Generation Module 450: Immediately Executed Module 500: Background learning track unit 510: Data transmission module 520: Data Augmentation Module 530: Model Training Module 600: Feedback Management Unit 610: Execution Result Collection Module 620: Weight Update Module 700: Model Integration Unit 710: Performance Evaluation Module 720: Model Deployment Module 810: Monitoring Module 820: Encoding module 830: Storage module 840: Training Data Management Module 850: Real-time Automation Engine 860: Background model training engine 861: Data Augmentation Submodule 862: Model Fine-Tuning Submodule 863: Model Evaluation and Deployment Submodule 870: Execution result collection module 880: Feedback processing module 1010: Input Interface 1020: Vector Database 1030: Training Data Repository 1040: Immediate Execution Module 1050: Background learning module 1060: Integrated Management Module

Claims

Claim 1 A dual-track automation system using a vision-language agent, comprising: a behavior monitoring unit (100) that collects behavior data by monitoring a user's screen operation and input behavior in real time; a vectorization unit (200) that vectorizes the collected behavior data using a Vision Transformer (VIT) and a language model (LLM); a vector database (VDB) (300) that stores the vectorized behavior data; a training data storage (350) that generates and stores training data for training a Vision Language Model (VLM); and an automation track unit (400) that vectorizes the current user's behavior, searches for similar past behavior patterns in the vector database (300), and generates and executes an automation script based on the behavior pattern when the similarity is above a preset threshold. and a background learning track unit (500) that transmits behavior data accumulated in the vector database (300) to the learning data storage (350) and fine-tunes the vision-language model (VLM) in the background using the data stored in the learning data storage (350) to improve the inference performance of the vision-language model (VLM); wherein the automation track unit (400) executes automation using the vector database (300) immediately after the user generates behavior data, and the background learning track unit (500) operates in parallel with the execution of the automation track unit (400) and is configured to continuously train the vision-language model (VLM) in the background without exposing the user to a learning delay time to improve long-term automation performance, and the background learning track unit (500) includes a data transmission module (510) that transmits the behavior data to the learning data storage (350) when the behavior data accumulated in the vector database (300) exceeds a preset threshold number.A dual-track automation system characterized by comprising: a data augmentation module (520) that expands the learning dataset by augmenting behavior data stored in the learning data repository (350) in a self-instruct manner; and a model learning module (530) that fine-tunes the vision-language model (VLM) in the background using the expanded learning dataset. Claim 2 A dual-track automation system according to claim 1, wherein the automation track unit (400) calculates a cosine similarity or Euclidean distance between a vector of current user behavior and a past behavior vector stored in the vector database (300) to search for a similar pattern, and if the similarity of the searched similar pattern is greater than or equal to a preset threshold, generates and executes an automation script based on a task sequence included in the similar pattern. Claim 3 A dual-track automation system according to claim 1, wherein the model learning module (530) combines a Vision Transformer (VIT) and a Large Language Model (LLM) to form the Vision-Language Model (VLM), fine-tunes the Vision-Language Model (VLM) using the expanded learning dataset, and then verifies the performance of the fine-tuned Vision-Language Model (VLM) through a Chain of Vision (CoV) based verification method. Claim 4 A dual-track automation system using a vision-language agent comprises: a processor; and a memory storing instructions that cause the processor to perform the following when executed by the processor, wherein the instructions cause the processor to: collect behavior data by monitoring the user’s screen operation and input behavior in real time; vectorize the collected behavior data using screen recognition technology and a language model (LLM); store the vectorized behavior data in a vector database (300); generate training data for training the vision-language model (VLM) and store it in a training data repository (300); vectorize the current user’s behavior, search for similar past behavior patterns in the vector database (300), and if the similarity is above a preset threshold, generate an automation script based on the corresponding behavior pattern and execute the automation. and a background learning step for improving the inference performance of a vision-language model (VLM) by fine-tuning the vision-language model (VLM) in the background using data stored in the learning data storage (350), wherein the step of executing the automation executes the automation using the vector database (300) immediately after the user generates the behavior data, and the background learning step operates in parallel with the step of executing the automation and is configured to continuously train the vision-language model (VLM) in the background without exposing the user to a learning delay time to improve long-term automation performance, and the background learning step includes the step of transmitting the behavior data to the learning data storage (350) when the number of behavior data accumulated in the vector database (300) exceeds a preset threshold.A dual-track automation system characterized by comprising: a step of expanding a training dataset by augmenting behavior data stored in the training data repository (350) in a self-instruct manner; a step of configuring a vision-language model (VLM) by combining a screen recognition device (VIT) and a language model (LLM); and a step of fine-tuning the vision-language model (VLM) using the expanded training dataset. Claim 5 A dual-track automation system using a vision-language agent, comprising: a behavior monitoring unit (100) that collects behavior data by monitoring a user's screen operation and input behavior in real time; a vectorization unit (200) that vectorizes the collected behavior data using a Vision Transformer (VIT) and a language model (LLM); a vector database (VDB) (300) that stores the vectorized behavior data; a training data storage (350) that generates and stores training data for training a Vision Language Model (VLM); and an automation track unit (400) that vectorizes the current user's behavior, searches for similar past behavior patterns in the vector database (300), and generates and executes an automation script based on the behavior pattern when the similarity is above a preset threshold. The system includes a background learning track unit (500) that transmits behavior data accumulated in the vector database (300) to the learning data storage (350) and fine-tunes the vision-language model (VLM) in the background using the data stored in the learning data storage (350) to improve the inference performance of the vision-language model (VLM); wherein the automation track unit (400) executes automation using the vector database (300) immediately after the user generates behavior data, and the background learning track unit (500) operates in parallel with the execution of the automation track unit (400) and is configured to continuously train the vision-language model (VLM) in the background without exposing the user to a learning delay time to improve long-term automation performance, and the system includes a feedback management unit (600) that collects the execution results, success status, and user feedback of the automation script executed by the automation track unit (400), and updates the weights of each behavior pattern by reflecting the collected information in the vector database (300).A dual-track automation system including additionally.; Claim 6 A dual-track automation system according to claim 5, wherein the feedback management unit (600) increases the weight of behavior patterns with an execution success rate higher than a preset threshold and decreases the weight of behavior patterns with an execution failure rate higher than a preset threshold, and the background learning track unit (500) fine-tunes the vision-language model (VLM) by reflecting the updated weight information. Claim 7 A dual-track automation system using a vision-language agent, comprising: a behavior monitoring unit (100) that collects behavior data by monitoring a user's screen operation and input behavior in real time; a vectorization unit (200) that vectorizes the collected behavior data using a Vision Transformer (VIT) and a language model (LLM); a vector database (VDB) (300) that stores the vectorized behavior data; a training data storage (350) that generates and stores training data for training a Vision Language Model (VLM); and an automation track unit (400) that vectorizes the current user's behavior, searches for similar past behavior patterns in the vector database (300), and generates and executes an automation script based on the behavior pattern when the similarity is above a preset threshold. A background learning track unit (500) that transmits behavior data accumulated in the vector database (300) to the learning data storage (350) and fine-tunes the vision-language model (VLM) in the background using the data stored in the learning data storage (350) to improve the inference performance of the vision-language model (VLM);A dual-track automation system comprising: an automation track unit (400) that executes automation using the vector database (300) immediately after a user generates behavior data; a background learning track unit (500) that operates in parallel with the execution of the automation track unit (400) and is configured to continuously train the vision-language model (VLM) in the background without exposing the user to a learning delay time to improve long-term automation performance; and a model integration unit (700) that integrates the fine-tuned vision-language model (VLM) into the automation track unit (400) and utilizes it for generating automation scripts when the inference performance of the vision-language model (VLM) fine-tuned by the background learning track unit (500) exceeds the vector search-based automation performance of the automation track unit (400). Claim 8 A dual-track automation method using a vision-language agent, comprising: a step of collecting behavioral data by monitoring a user's screen operation and input behavior in real time; a step of vectorizing the collected behavioral data using a screen recognition technology (VIT) and a language model (LLM); a step of storing the vectorized behavioral data in a vector database (VDB) (300); a step of vectorizing the current user's behavior and searching for similar past behavioral patterns in the vector database (300); and an automation step of generating an automation script based on the behavioral pattern and executing automation when the similarity of the searched similar pattern is greater than or equal to a preset threshold. The method includes a background learning step for improving the inference performance of a vision-language model (VLM) by fine-tuning the vision-language model (VLM) in the background using the data stored in the learning data storage (350), wherein the automation step executes automation without delay using the vector database (300) immediately after the user generates the behavior data, and the background learning step is performed in parallel with the automation step, and continuously trains the vision-language model (VLM) in the background without exposing the user to a learning delay time to improve long-term automation performance, and the background learning step includes the step of transmitting the behavior data to the learning data storage (350) when the number of behavior data accumulated in the vector database (300) exceeds a preset threshold; the step of expanding the learning dataset by augmenting the behavior data stored in the learning data storage (350) in a self-instruct manner; and the step of configuring the vision-language model (VLM) by combining a screen recognition device (VIT) and a language model (LLM). A dual-track automation method characterized by including the step of fine-tuning the vision-language model (VLM) using the expanded training dataset. Claim 9 A dual-track automation method according to claim 8, wherein the vectorizing step comprises: converting a screen capture image into an image embedding vector through a Vision Transformer (VIT); converting a user's text input and task context into a text embedding vector through a Large Language Model (LLM); and combining the image embedding vector and the text embedding vector to generate an integrated action vector. Claim 10 A dual-track automation method characterized in that, in claim 8, the background learning step further includes a step of verifying the performance of the fine-tuned vision-language model (VLM) through a Chain of Vision (CoV)-based verification method. Claim 11 A dual-track automation method using a vision-language agent, comprising: a step of collecting behavioral data by monitoring a user's screen operation and input behavior in real time; a step of vectorizing the collected behavioral data using a screen recognition technology (VIT) and a language model (LLM); a step of storing the vectorized behavioral data in a vector database (VDB) (300); a step of vectorizing the current user's behavior and searching for similar past behavioral patterns in the vector database (300); and an automation step of generating and executing an automation script based on the behavioral pattern when the similarity of the searched similar pattern is greater than or equal to a preset threshold. A dual-track automation method comprising: a background learning step of transmitting behavioral data accumulated in the vector database (300) to a learning data repository (350) and fine-tuning the vision-language model (VLM) in the background using the data stored in the learning data repository (350) to improve the inference performance of the vision-language model (VLM); wherein the automation step executes automation without delay using the vector database (300) immediately after the user generates behavioral data, and the background learning step is performed in parallel with the automation step, and continuously trains the vision-language model (VLM) in the background without exposing the user to a learning delay time to improve long-term automation performance, and further comprising: a step of collecting the execution results, success status, and user feedback of the automation script executed in the automation step; and a step of updating the weights of each behavioral pattern by reflecting the collected information in the vector database (300) and reflecting the updated weight information in the fine-tuning of the vision-language model (VLM) in the background learning step. Claim 12 A dual-track automation method using a vision-language agent, comprising: a step of collecting behavioral data by monitoring a user's screen operation and input behavior in real time; a step of vectorizing the collected behavioral data using a screen recognition technology (VIT) and a language model (LLM); a step of storing the vectorized behavioral data in a vector database (VDB) (300); a step of vectorizing the current user's behavior and searching for similar past behavioral patterns in the vector database (300); and an automation step of generating and executing an automation script based on the behavioral pattern when the similarity of the searched similar pattern is greater than or equal to a preset threshold. A dual-track automation method comprising: a background learning step of transmitting behavioral data accumulated in the vector database (300) to a learning data repository (350) and fine-tuning the vision-language model (VLM) in the background using the data stored in the learning data repository (350) to improve the inference performance of the vision-language model (VLM); wherein the automation step executes automation without delay using the vector database (300) immediately after the user generates behavioral data, and the background learning step is performed in parallel with the automation step, and continuously trains the vision-language model (VLM) in the background without exposing the user to a learning delay time to improve long-term automation performance, and further comprising: a step of evaluating the inference performance of the vision-language model (VLM) fine-tuned by the background learning step; and, if the inference performance of the fine-tuned vision-language model (VLM) exceeds the vector search-based automation performance of the automation step, a step of integrating the fine-tuned vision-language model (VLM) into the automation step and utilizing it for generating an automation script. Claim 13 A user behavior learning and automation system using a vision-language agent, comprising: a monitoring module (810) that collects behavior data by monitoring user interactions including user screen manipulation, keyboard input, and mouse movements; an encoding module (820) that converts the collected behavior data into a vector representation using a screen recognition model (VIT) and a language model (LLM); a storage module (830) that stores the converted vector representation in a vector database (VDB) (300) and manages it in association with work metadata; a learning data management module (840) that converts the behavior data accumulated in the vector database (300) into learning data and stores it in a learning data repository (350); and a real-time automation engine (850) that, when a current user task is input, searches the vector database (300) to extract similar task patterns, creates an automation workflow based on the extracted similar task patterns, and executes it in real time. and a background model learning engine (860) that fine-tunes a vision-language model (VLM) in the background using data stored in the learning data repository (350), evaluates the performance of the fine-tuned model, and integrates it into the real-time automation engine (850); wherein the real-time automation engine (850) executes automation without delay based on vector search in response to the user's immediate automation request, and the background model learning engine (860) operates in parallel independently of the execution of the real-time automation engine (850) and is configured to improve long-term automation performance without exposing learning delays to the user, and the background model learning engine (860) comprises: a data augmentation submodule (861) that generates similar data by augmenting behavior data stored in the learning data repository (350) in a self-instruct manner; and a model fine-tuning submodule (862) that configures a vision-language model (VLM) by combining a screen recognition device (VIT) and a language model (LLM) and fine-tunes using the augmented data.A user behavior learning and automation system characterized by including Claim 14 A user behavior learning and automation system according to claim 13, wherein the real-time automation engine (850) searches for similar task patterns in the vector database (300) using a k-nearest neighbor (k-NN) algorithm or an approximate nearest neighbor search (ANN) algorithm, and if the similarity score of the searched similar task patterns is greater than or equal to a preset threshold, it creates and immediately executes an automation workflow. Claim 15 A user behavior learning and automation system, characterized in that, in claim 13, the background model learning engine (860) further comprises a model evaluation and distribution submodule (863) that evaluates the performance of a fine-tuned model through Chain of Vision (CoV)-based verification and distributes it to the real-time automation engine (850) if the performance exceeds a threshold. Claim 16 A user behavior learning and automation system using a vision-language agent, comprising: a monitoring module (810) that collects behavior data by monitoring user interactions including user screen manipulation, keyboard input, and mouse movements; an encoding module (820) that converts the collected behavior data into a vector representation using a screen recognition model (VIT) and a language model (LLM); a storage module (830) that stores the converted vector representation in a vector database (VDB) (300) and manages it in association with work metadata; a learning data management module (840) that converts the behavior data accumulated in the vector database (300) into learning data and stores it in a learning data repository (350); and a real-time automation engine (850) that, when a current user task is input, searches the vector database (300) to extract similar task patterns, creates an automation workflow based on the extracted similar task patterns, and executes it in real time. The system includes a background model learning engine (860) that fine-tunes a vision-language model (VLM) in the background using data stored in the learning data repository (350), evaluates the performance of the fine-tuned model, and integrates it into the real-time automation engine (850); wherein the real-time automation engine (850) is configured to execute automation without delay based on vector search in response to the user's immediate automation request, and the background model learning engine (860) operates in parallel independently of the execution of the real-time automation engine (850) and is configured to improve long-term automation performance without exposing learning delays to the user; and the system includes an execution result collection module (870) that collects execution results, success status, execution time, and user feedback of the automation workflow executed in the real-time automation engine (850); and a feedback processing module (880) that analyzes the collected execution results to update the behavior pattern weights of the vector database (300) and feeds back to the background model learning engine (860) to reflect in model learning.A user behavior learning and automation system that further includes Claim 17 A user behavior learning and automation system using a vision-language agent, comprising: a monitoring module (810) that collects behavior data by monitoring user interactions including user screen manipulation, keyboard input, and mouse movements; an encoding module (820) that converts the collected behavior data into a vector representation using a screen recognition model (VIT) and a language model (LLM); a storage module (830) that stores the converted vector representation in a vector database (VDB) (300) and manages it in association with work metadata; a learning data management module (840) that converts the behavior data accumulated in the vector database (300) into learning data and stores it in a learning data repository (350); and a real-time automation engine (850) that, when a current user task is input, searches the vector database (300) to extract similar task patterns, creates an automation workflow based on the extracted similar task patterns, and executes it in real time. A user behavior learning and automation system comprising: a background model learning engine (860) that fine-tunes a vision-language model (VLM) in the background using data stored in the learning data repository (350), evaluates the performance of the fine-tuned model, and integrates it into the real-time automation engine (850); wherein the real-time automation engine (850) is configured to execute automation without delay based on vector search in response to the user's immediate automation request, and the background model learning engine (860) operates in parallel independently of the execution of the real-time automation engine (850) and is configured to improve long-term automation performance without exposing learning delays to the user, and the real-time automation engine (850) includes an exception handling function that, when a screen state different from the expectation is detected during the execution of the automation workflow, searches for alternative behavior patterns performed in a situation similar to the current screen state in the vector database (300) and continues to perform the work by applying the searched alternative behavior patterns. Claim 18 A non-transient computer-readable recording medium storing a computer program for implementing a vision-language agent-based dual-track automation system, wherein the computer program comprises: a step of collecting behavioral data by monitoring a user's screen operation and input behavior in real time; a step of vectorizing the collected behavioral data using screen recognition technology and a language model; a step of storing the vectorized behavioral data in a vector database (VDB) (300); and a step of performing an automation track that vectorizes the current user's behavior, searches for similar past behavioral patterns in the vector database (300), and immediately generates and executes an automation script when the similarity is greater than or equal to a preset threshold. A computer-readable recording medium comprising: a step of performing a background learning track in parallel to convert behavioral data accumulated in the vector database (300) into learning data and fine-tune a vision-language model (VLM) in the background to improve long-term inference performance; wherein the step of performing the automation track provides automation to the user without delay, and the step of performing the background learning track in parallel continuously improves model performance in the background without exposing learning delay time to the user, and wherein the step of performing the background learning track in parallel includes: a step of transmitting the behavioral data to the learning data storage (350) when the behavioral data accumulated in the vector database (300) exceeds a preset threshold number; a step of expanding the learning dataset by augmenting the behavioral data stored in the learning data storage (350) in a self-instruct manner; a step of configuring a vision-language model (VLM) by combining a screen recognition device (VIT) and a language model (LLM); and a step of fine-tuning the vision-language model (VLM) using the expanded learning dataset. Claim 19 A non-transient computer-readable recording medium storing a computer program for implementing a dual-track automation system based on a vision-language agent, wherein the computer program comprises: a step of collecting behavioral data by monitoring a user's screen operation and input behavior in real time; a step of vectorizing the collected behavioral data using screen recognition technology and a language model; a step of storing the vectorized behavioral data in a vector database (VDB) (300); a step of performing an automation track that vectorizes the current user's behavior, searches for similar past behavioral patterns in the vector database (300), and immediately generates and executes an automation script if the similarity is above a preset threshold; a step of performing a background learning track in parallel that converts the behavioral data accumulated in the vector database (300) into training data and fine-tunes a vision-language model (VLM) in the background to improve long-term inference performance; and a step of collecting the execution results, success status, and user feedback of the automation script executed in the step of performing the automation track. A computer-readable recording medium comprising a command to execute the step of updating the weights of each behavior pattern by reflecting the collected information in the vector database (300) and reflecting the updated weight information in the fine-tuning of the vision-language model (VLM) in the step of performing the background learning track in parallel, wherein the step of performing the automation track provides automation to the user without delay, and the step of performing the background learning track in parallel continuously improves model performance in the background without exposing the user to learning delay time. Claim 20 A hybrid automation system combining a vision-language agent and a vector database, comprising: an input interface (1010) for receiving natural language-based task requests from a user; a vector database (1020) in which past user behavior data is stored in vector form; a training data repository (1030) in which data for learning a vision-language model (VLM) is stored; an execution module (1040) that encodes the natural language task request into a language model, searches for similar past tasks in the vector database (1020), analyzes the execution history of the searched past tasks and the current screen state to establish an automation plan, and executes it immediately; and a background learning module (1050) that transmits data accumulated in the vector database (1020) to the training data repository (1030) and fine-tunes the vision-language model (VLM) in the background using the data in the training data repository (1030). A hybrid automation system comprising: an integrated management module (1060) that feeds back the execution results of the execution module (1040) to the vector database (1020) and evaluates the performance of the vision-language model (VLM) fine-tuned in the background learning module (1050) and integrates it into the execution module (1040); wherein the execution module (1040) is configured to respond in real-time to the user's immediate automation requests through vector search, the background learning module (1050) is configured to continuously improve the model in the background without affecting the user experience, and the execution module (1040) is configured to perform context-aware search by comprehensively considering the semantic similarity between the user's natural language request and past tasks stored in the vector database (1020), as well as the current screen context, running application, time zone, and user profile, and to establish an optimal automation plan by applying the execution success rate, recency, and user preference of the searched similar tasks as weights.

Images