Vlm-based document / web automatic inspection system and method

Abstract

The present invention relates to a VLM-based document / web automatic inspection system and method that automatically inspects documents and web pages without relying on external APIs by utilizing a Vision-Language Model (Local VLM) operating in a local environment. The system includes a Local VLM combining a Visual Indicator (VIT) and a Large Language Model (LLM), an AI Operational Task Management System (ITOMS), a Real-time Object Recognition RPA Component (RPACA), an AI object detection and text recognition module, a Chain of Vision (CoV) 4-stage verification module, and a Self-Instruct data augmentation module. The CoV verification module verifies structure / formatting, content completeness, compliance, and logical consistency step-by-step to minimize illusion issues in VLM and guarantee inspection accuracy. The Self-Instruct data augmentation module expands limited inspection history data by more than tenfold, enabling high-quality VLM learning. The present invention provides automatic inspection and real-time monitoring of documents and web content while maintaining data security, thereby significantly improving the efficiency and accuracy of inspection tasks.

Description

Technology Field

[0001] The present invention relates to a technology for automated document and web inspection utilizing a Vision Language Model (VLM), and more specifically, to a system and method that automatically verifies the format, content, and regulatory compliance of documents by combining a VLM operating in a local environment with real-time object recognition RPA, and guarantees the accuracy of the inspection results through Chain of Vision (CoV) verification. Background Technology

[0002] The traditional document review process has the following limitations.

[0003] Registered Patent Publication No. 10-1515256 (hereinafter referred to as 'Prior Art 1') discloses a document verification technology using a dynamic document identification framework. Prior Art 1 is a technology for computer-assisted identification and validation of security documents (passports, driver's licenses, birth certificates, etc.), and identifies the document type and verifies its authenticity by sequentially applying a classifier, a verifier, and a validator through a document identification framework with a hierarchical tree structure.

[0005] However, prior art 1 is specialized for verifying the authenticity of security documents and is not suitable for inspection purposes that verify the format, content, and regulatory compliance of general documents. Furthermore, as it is a static process that sequentially applies predefined algorithms for each document type, it is difficult to respond flexibly to new inspection standards or regulations. There is no function to interpret inspection standards entered by the user in natural language form, and inspection standards must be defined in a pre-structured form. In addition, while there is a validator for identifying document types, there is no verification mechanism dedicated to the inspection process to guarantee inspection accuracy.

[0007] Published Patent Application No. 10-2023-0121637 (hereinafter referred to as 'Prior Art 2') discloses a form data extractor technology for document processing. Prior Art 2 is a technology that identifies document types using an RPA workflow and extracts data based on a template, and extracts data from a document using a template in which the user predefines the fields to be extracted and the locations of the fields.

[0009] However, Prior Art 2 focuses on extracting data from documents and is therefore not suitable for inspection purposes that verify the format, content, and regulatory compliance of documents. Since it requires a template in which the user pre-defines field locations, it is difficult to respond immediately to new document formats or inspection criteria. As a template-based method, it lacks the functionality to interpret inspection criteria entered by the user in natural language. It includes a verification module in which the user manually verifies classification and extraction results, so automated inspection verification is not performed. Furthermore, it uses traditional OCR and template matching technologies and does not utilize VLM-based technology that combines Vision Transformer and Large Language Model.

[0011] The problems with existing technologies can be summarized as follows.

[0012] Document inspection requires manual verification by specialized personnel, which is time-consuming and costly. Inconsistencies in inspection results are lacking as standards vary among inspectors. It is difficult to process large volumes of documents and cope with increased workloads.

[0014] It can only check predefined rules, making it difficult to respond to new regulations or exceptions. If the document format changes, the entire system must be modified. Responding only to simple pattern matching, it fails to understand the context or meaning of the documents.

[0016] It is difficult to verify dynamic content in real time. Integrated inspection is difficult because each website has a different UI structure. It is difficult to automatically detect and inspect real-time changes to web content.

[0018] Using cloud-based AI services raises concerns regarding data security. Costs are incurred for every API call, resulting in a significant financial burden when processing large volumes of documents. An internet connection is required, and the service cannot be used in the event of a network failure. The lack of mechanisms to verify the AI's decision results can lead to hallucination issues. The problem to be solved

[0020] The problem that the present invention aims to solve is as follows.

[0022] Utilizing a VLM operating in a local environment, this invention provides a system specialized in the automated inspection of documents and web content without relying on external APIs. Unlike general business automation, this is a dedicated inspection system that interprets inspection criteria in natural language and verifies regulatory compliance. Unlike existing document processing technologies (document classification, data extraction), this invention is a system specialized for document and web inspection that comprehensively verifies the formatting, content, and regulatory compliance of documents.

[0024] We provide a system that combines real-time object recognition RPA with VLM to comprehensively verify document formatting, content, and regulatory compliance. This goes beyond simple screen automation to perform systematic verification based on inspection standards.

[0026] A Chain of Vision (CoV) verification module dedicated to the document / web inspection process is provided to verify the inspection results of VLM in four stages. This is not a verification of the quality of VLM training data, but a verification mechanism designed to minimize hallucination issues during the actual inspection process. By storing the verification results of each stage in a verification database and comparing their consistency, inspection accuracy is guaranteed. The CoV verification module of the present invention has a completely different purpose and configuration from existing document type identification verifiers.

[0028] It provides self-instruct data augmentation techniques specialized for document inspection. Through inspection-specific augmentation strategies such as format parameter modification, content reorganization, rule item rearrangement, and form modification, high-quality VLM can be trained even with limited inspection data.

[0030] It provides a system that automatically plans and manages inspection tasks through ITOMS (AI Operational Task Management System) and automatically handles exceptional situations. This system divides inspection criteria into detailed tasks such as "form verification," "required item verification," "numerical verification," and "regulatory compliance verification," and converts each into specific actions.

[0032] It provides an integrated inspection system capable of inspecting not only documents but also dynamic content on web pages in real time, automatically detecting changes through periodic monitoring, and providing notifications in case of regulation violations. means of solving the problem

[0034] A VLM-based automatic document / web inspection system according to an embodiment of the present invention for solving a problem comprises: an input interface that receives information on a document or web page to be inspected and inspection criteria in natural language form from a user; a local VLM that is a vision-language model (Local VLM) operating in a local environment, which captures the screen of a document or web page and analyzes visual information through a screen recognition device, interprets inspection criteria in natural language form through a large-scale language model to convert them into structured inspection requirements and analyzes text content, and generates an inspection plan that determines the format compliance, content completeness, and regulatory compliance of the document / web by combining information processed by the screen recognition device and the large-scale language model, wherein the inspection plan includes structured inspection requirements generated by interpreting inspection criteria in natural language form, and includes an inspection plan and script generator composed of format verification items, content verification items, and regulatory compliance items; an AI operational task management system that processes the inspection plan generated in the local VLM and manages its execution; and a real-time object recognition RPA component that recognizes specific UI elements on a document or web screen, extracts text, and verifies each item of a regulatory checklist according to instructions from the AI ​​operational task management system.As a Chain of Vision (CoV) verification module that verifies inspection data collected by a real-time object recognition RPA component step-by-step in the document / web inspection process, the Chain of Vision verification module is a verification mechanism dedicated to the inspection process designed to minimize the illusion of VLM occurring during the actual inspection process, rather than verifying the quality of VLM training data; the Chain of Vision verification module performs a first verification step that verifies whether the basic format and structure of the document / web conform to predefined template standards and checks margins, fonts, line spacing, and section order; a second verification step that verifies whether required items are written, the accuracy of spelling / grammar / technical terminology of the text, and the consistency of sums and ratios of numerical data; a third verification step that verifies compliance with legal requirements, industry standards, and internal regulations based on a checklist; and a fourth verification step that verifies whether there are no contradictions between the results of the first to third verification steps and re-evaluates the logical validity of the overall inspection results. The system includes a result output module that generates a report of the completed inspection results and provides it to the user; wherein the system automates the entire inspection process for the automatic inspection of documents and web content, so that when the user provides inspection criteria in natural language form, it automatically performs document / web capture, analysis, verification, and result generation; the Chain of Vision inspection verification module stores the verification results for each verification stage in a verification database, compares consistency with the verification results of the previous stage, and performs a re-verification process or provides a request for manual user verification if a discrepancy is found; the verification database stores the verification results of the first through fourth verification stages step by step and automatically detects whether there are logical contradictions between the verification results of each stage, thereby minimizing the illusion problem of VLM during the document / web inspection process and guaranteeing the accuracy of inspection regarding format, content, and compliance; and the local VLM maintains the security of sensitive document information by performing all processing in a local environment without using external cloud APIs.

[0036] According to one embodiment of the present invention, a chain of vision inspection verification module checks in a first verification step whether the layout of a document / web matches a predefined template and verifies one or more formatting parameters among margins, fonts, and line spacing; in a second verification step, checks whether required items have been written, verifies the accuracy of one or more of the spelling, grammar, and technical terms of the text, and verifies the consistency of sums and ratios of numerical data; in a third verification step, verifies compliance with one or more of legal requirements, industry standards, and internal regulations based on a checklist; and in a fourth verification step, checks whether there are no contradictions between the results of the first to third verification steps and re-evaluates the logical validity of the overall inspection results.

[0038] According to one embodiment of the present invention, an AI operational task management system performs a task planning step that analyzes inspection criteria and divides the entire inspection task into detailed units, and an action planning step that converts each detailed task into a specific action for a specific location or element on a screen, wherein the task planning step analyzes the inspection goal and divides it into one or more detailed tasks among form verification, required item verification, numerical verification, and compliance verification, and the action planning step generates one or more specific action sequences among click, scroll, text extraction, and screenshot capture for each detailed task, and includes the location coordinates of the target UI element for each action.

[0040] According to one embodiment of the present invention, an AI object detection and text recognition module that supports a real-time object recognition RPA component is additionally included, wherein the AI ​​object detection and text recognition module uses a YOLO-based object detection model to identify one or more UI elements among tables, graphs, images, text blocks, buttons, and input fields on a document / web screen and provides location coordinates, and uses a Transformer-based OCR model to extract text on the screen and provide content information.

[0042] According to one embodiment of the present invention, a self-instructed data augmentation module for training a local VLM is further included, wherein the self-instructed data augmentation module includes a seed data collection unit that collects seed inspection data from an inspection work history; an augmented data generation unit that generates training data of at least 10 times the original data by applying one or more augmentation strategies to the seed inspection data, such as a format parameter transformation strategy that changes one or more parameters among margin, font size, and line spacing; a content reconstruction strategy that replaces text with synonyms or changes sentence order; a regulation item rearrangement strategy that changes the order of checklists or includes additional regulation items; and a form transformation strategy that applies the same content to different document forms; and a VLM fine-tuning unit that fine-tunes the local VLM with the generated training data.

[0044] According to one embodiment of the present invention, the system separates the test environment and the actual environment, first executes an inspection plan in the test environment to verify accuracy, and performs a final inspection in the actual environment after verification through the chain of vision inspection verification module is completed.

[0046] According to one embodiment of the present invention, an input interface includes a server / web automatic detection module, wherein the server / web automatic detection module includes a change detection engine that automatically captures a designated web page periodically to detect changes, an automatic inspection trigger that automatically transmits an inspection request to an AI operation task management system when changes are detected, and a result processing unit that provides real-time notifications to a user when violations of regulations are found.

[0048] According to one embodiment of the present invention, a local VLM processes a screen recognition system and a large-scale language model in parallel to simultaneously analyze visual information and text information, and combines the visual information and text information through a multimodal fusion layer, wherein the multimodal fusion layer learns the interaction between the two pieces of information using a cross-attention mechanism or a gate mechanism, and improves the accuracy of inspection judgment using the fused representation.

[0050] According to one embodiment of the present invention, a result output module comprises: an inspection report generation unit that generates an inspection report including summary information including inspection subject, date and time, and overall results; detailed items including results and scores for each inspection item; findings listing problems found, omitted items, and violations of regulations; recommendations suggesting directions for improvement; and supporting materials including attached screenshots and extracted data; a visualization unit that visualizes the inspection report as graphs and charts; and a notification function unit that transmits to a user via one or more channels among email, messenger, and SMS.

[0052] According to one embodiment of the present invention, a self-instruct data augmentation module includes a chain of vision verification evaluation unit that evaluates one or more of inspection accuracy, completeness, and consistency using a chain of vision inspection verification module on a test dataset to evaluate the quality of a fine-tuned local VLM, and a local VLM distribution unit that distributes the local VLM to a system only when the inspection accuracy is 90% or higher, the completeness is 95% or higher, and the consistency is 95% or higher.

[0054] A VLM-based automatic document / web inspection method according to another embodiment of the present invention for solving the problem comprises: (a) receiving information on a document or web page to be inspected and inspection criteria in natural language form from a user; (b) capturing and analyzing the current screen through a vision-language model (Local VLM) operating in a local environment, wherein one or more visual information among the format, structure, and UI elements of the document / web are processed through a screen recognition device, and the inspection criteria in natural language form are interpreted through a large-scale language model and converted into structured inspection requirements; (c) inputting the inspection criteria and screen analysis results into the local VLM to automatically generate an inspection work plan and a script through two stages of work planning and action planning for document / web inspection; (d) processing the generated script in an AI operational work management system; and (e) recognizing UI elements on the document or web screen, extracting text, and verifying each item of a regulation checklist through a real-time object recognition RPA component. (f) A step of verifying the inspection results through a four-step verification process using a Chain of Vision (CoV) inspection verification module in the document / web inspection process, wherein the first verification verifies whether the basic format and structure of the document / web conform to predefined template standards and checks one or more of margins, fonts, line spacing, and section order; the second verification verifies whether required items are completed, the accuracy of spelling / grammar / technical terms of the text, and the consistency of sums and ratios of numerical data; the third verification verifies compliance with one or more of legal requirements, industry standards, and internal regulations based on a checklist; and the fourth verification verifies whether there are no contradictions between the results of the first to third verifications and re-evaluates the logical validity of the entire inspection result; and (g) A step of generating the verified inspection result into a report including one or more of findings, recommendations, and supporting materials and providing it to the user;Including, however, the work plan in step (c) analyzes the inspection criteria and divides them into one or more detailed inspection tasks among form verification, required item verification, numerical verification, and compliance verification; the action plan in step (c) translates each detailed task into one or more specific actions among clicking, scrolling, text extraction, and screenshot capture on specific locations or elements on the screen; and the chain of vision inspection verification in step (f) stores the verification results in the verification database for each verification step and compares them with the consistency of the verification results from the previous step, performing a re-verification process or providing a request for manual user verification if a discrepancy is found, thereby minimizing the illusion problem of VLM during the document / web inspection process and guaranteeing the accuracy of inspections for format, content, and compliance, and the local VLM maintains the security of sensitive document information by performing all processing in a local environment without using external cloud APIs.

[0056] According to another embodiment of the present invention, step (e) further includes a step of supporting the operation of a real-time object recognition RPA component through an AI object detection and text recognition module, wherein the AI ​​object detection and text recognition module identifies UI elements of a document / web screen and provides location coordinates using a YOLO-based object detection model and extracts text content using a Transformer-based OCR model.

[0058] According to another embodiment of the present invention, step (e) comprises: a sub-step of first executing an inspection plan in a test environment to verify accuracy; and a sub-step of performing a final inspection in a real environment only if the chain of vision verification is passed.

[0060] According to another embodiment of the present invention, the method further comprises: a step of periodically automatically capturing a specified web page through a server / web auto-detection module; a change detection algorithm step of detecting changes by comparing the currently captured web page with the previous capture result; an automatic inspection trigger step of automatically executing steps (a) through (g) to perform an inspection when a change is detected; and a result processing unit step of providing real-time notification to a user when a violation of regulations is found.

[0062] According to another embodiment of the present invention, (h) further includes a step of continuously improving the local VLM by augmenting inspection history data in a self-instruct manner, wherein (h) includes: a seed data collection unit sub-step for collecting seed inspection data from inspection work history; an augmented data generation unit sub-step for expanding the original data by more than 10 times by applying one or more augmentation strategies among format parameter transformation strategy, content reconstruction strategy, regulation item rearrangement strategy, and form transformation strategy to the seed inspection data; a VLM fine-tuning unit sub-step for fine-tuning the local VLM using the expanded training data; and a chain of vision verification evaluation unit sub-step for evaluating the inspection accuracy of the fine-tuned VLM through a chain of vision verification method and a local VLM distribution unit sub-step for distributing to the system upon passing the evaluation.

[0064] According to another embodiment of the present invention, the first verification of step (f) includes verifying whether the layout of the document / web matches a predefined template, verifying formatting parameters such as margins, font size, and line spacing, and verifying the section structure and order; the second verification of step (f) includes verifying whether all required items have been written, verifying the accuracy of the spelling, grammar, and technical terms of the text, and verifying the consistency of sums and ratios of numerical data; the third verification of step (f) includes verifying compliance with one or more of legal requirements, industry standards, and internal regulations based on a checklist; and the fourth verification of step (f) includes verifying whether there are no contradictions between the results of the first to third verifications and re-evaluating the logical validity of the overall inspection results.

[0066] According to another embodiment of the present invention, a formatting parameter transformation strategy includes changing the margin to one of 1.5 cm, 2 cm, 2.5 cm, or 3 cm, changing the font size to one of 10 pt, 11 pt, 12 pt, or 14 pt, and changing the line spacing to one of 1.0, 1.15, 1.5, or 2.0; a content reorganization strategy includes replacing text with synonyms, changing the sentence order while maintaining a logical flow, and changing the expression method; a regulation item rearrangement strategy includes changing the checklist order and including additional regulation items; and a form transformation strategy includes changing the table format to a list format or changing the vertical layout to a horizontal layout.

[0068] A local VLM learning method for automatic document / web inspection according to another embodiment of the present invention for solving the problem comprises: (a) collecting seed data including an image of a document / web screen to be inspected, inspection criteria text, an inspection plan and action sequence, an inspection result, and a Chain of Vision (CoV) 4-stage verification result from an inspection work history; (b) augmenting the seed data in a self-instruct manner, wherein one or more of the following are performed to expand the original data by more than 10 times: augmentation through format parameter modification that changes one or more of margins, fonts, and line spacing; augmentation through content reconstruction that includes synonym substitution and sentence order change; augmentation through modification of the order of regulation items that includes checklist rearrangement; and augmentation through the application of various document forms that includes layout modification; (c) configuring a local VLM architecture that combines a screen recognizer and a large-scale language model, wherein the architecture comprises a visual encoder using a ViT-family model, a language encoder using an LLM lightweight model, a multimodal fusion layer using a cross-attention or gate mechanism, and an output decoder using multiple work heads. (d) a step of fine-tuning the local VLM using augmented training data, wherein the loss function is optimized by combining the task classification loss, the action sequence generation loss, and the validation score loss; (e) a step of evaluating the inspection accuracy of the fine-tuned VLM through a chain of vision validation method, wherein the four-stage validation is performed on a test dataset and the inspection accuracy, completeness, and consistency metrics are measured; and (f) a step of deploying the local VLM to the validation system only when the inspection accuracy is 90% or higher, completeness is 95% or higher, and consistency is 95% or higher in the evaluation; wherein the chain of vision validation minimizes the illusion problem by verifying the inspection results generated by the VLM in four stages, and the local VLM operates in a local environment to maintain data security without relying on external APIs.

[0070] According to another embodiment of the present invention, step (a) collects only those cases that have passed all four stages of Chain of Vision verification among successful inspection cases as seed data, sets the modified data as the correct label when the user has modified it, and stores failed inspection cases separately to use for error pattern analysis.

[0072] According to another embodiment of the present invention, the visual encoder of step (c) uses ViT-B / 16 or ViT-L / 14, the language encoder uses a 4-bit or 8-bit quantized LLM and is optimized to be executable on a local workstation, and the multimodal fusion layer includes 12 or more cross-attention layers to learn the interaction between visual information and text information. Effects of the invention

[0074] The VLM-based document / web automatic inspection system and method of the present invention provide the following effects.

[0076] Data security is enhanced through the use of a local VLM. Since all verification is performed within a local environment without using external cloud APIs, there is no risk of sensitive document information being leaked externally. It can be safely utilized, particularly in security-critical fields such as finance, healthcare, and law. Unlike general business automation systems, this security structure is optimized for handling sensitive information, specifically for document verification.

[0078] It significantly improves inspection accuracy through Chain of Vision (CoV) verification dedicated to the document / web inspection process. This is not merely a quality verification of VLM training data, but a four-step verification process designed to minimize hallucination issues that occur during actual inspections. By storing the results of each inspection step (structure / formatting, content completeness, compliance, and logical consistency) in a verification database and comparing consistency, it provides reliable inspection results.

[0080] It solves the problem of insufficient training data through Self-Instruct data augmentation specialized for document inspection. By utilizing inspection-specific augmentation strategies—such as modifying format parameters, reorganizing content, rearranging regulation items, and modifying forms—it is possible to generate more than ten times the amount of training data using only limited inspection history data. Unlike augmentation for general business automation, this is an augmentation method that reflects the characteristics of document and web inspection.

[0082] Maximize inspection efficiency through automation using ITOMS. Simply providing inspection criteria automatically executes the entire inspection process via Task Planning and Action Planning, and automatically responds to any exceptional situations.

[0084] Accuracy is improved by combining real-time object recognition RPA and AI models. All elements of a document are accurately identified and verified by utilizing YOLOv8-based object detection and TrOCR-based text recognition.

[0086] It enables integrated inspection of documents and web content. Since it can inspect not only document files but also the dynamic content of web pages using the same system, it can be applied to various business scenarios.

[0088] Safety is ensured by separating test and actual environments. The possibility of errors is minimized by first performing inspections in the test environment to verify accuracy before applying it to the actual environment.

[0090] Real-time compliance is ensured through web monitoring features. By automatically inspecting web pages periodically, violations can be detected and responded to early.

[0092] The cost-saving benefits are significant. Since there are no cloud API costs and manual inspection personnel can be drastically reduced, substantial long-term cost savings can be achieved.

[0094] Continuous quality improvement is possible. Inspection history data is automatically converted into training data and continuously improves VLM through Self-Instruct augmentation, so inspection accuracy improves over time. Brief explanation of the drawing

[0096] Figure 1 is an overall configuration diagram of a VLM-based document / web automatic inspection system according to one embodiment of the present invention. FIG. 2 is a structure and operation flowchart of a local VLM according to one embodiment of the present invention. FIG. 3 is a diagram showing the Task Planning and Action Planning processes of an inspection operation according to one embodiment of the present invention. FIG. 4 is a diagram illustrating a Chain of Vision (CoV) 4-step verification process according to one embodiment of the present invention. FIG. 5 is a diagram illustrating a Self-Instruct data augmentation process according to one embodiment of the present invention. FIG. 6 is a detailed flowchart of a document inspection process according to one embodiment of the present invention. FIG. 7 is a diagram showing an automatic web page inspection and monitoring system according to an embodiment of the present invention. Specific details for implementing the invention

[0097] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

[0099] <First Embodiment: VLM-based Document / Web Automated Inspection System>

[0101] Referring to FIG. 1, the overall configuration of a VLM-based document / web automatic inspection system according to a first embodiment of the present invention is illustrated.

[0103] The system includes an input interface (100), a local VLM (200), an ITOMS (300), an RPACA (400), an AI model (500), a CoV verification module (600), a result output module (700), and a Self-Instruct data augmentation module (800).

[0105] (1) Input interface (100)

[0106] The input interface (100) receives information to be inspected and inspection criteria from the user. The information to be inspected includes a document file path, a web page URL, or information about an application window that is already open.

[0108] Inspection criteria can be entered in natural language form, for example, in the format of "verify that this contract conforms to the company's standard form," "verify that the website's privacy policy complies with GDPR regulations," and "check that all required items in the financial statements have been completed."

[0110] The input interface (100) may also include a Server / Web auto-detection module (110), which performs the function of automatically detecting changes in the document / web and transmitting a request for inspection to the ITOMS (300).

[0112] (2) Local VLM(200)

[0113] Referring to FIG. 2, the detailed structure of a local VLM (200) is illustrated. The local VLM (200) consists of a screen recognition unit (VIT, 210), a large-scale language model (LLM, 220), and a verification plan and script generator (230).

[0115] VIT (210) is a Vision Transformer-based screen processing processor that performs real-time screen capture and image preprocessing, UI element identification and location identification, text OCR and image analysis, object recognition and classification, and converts all visual information of a document / web screen into a vector form.

[0117] LLM (220) is a large-scale language model that runs in a local environment and performs user inspection criteria understanding and interpretation, task decomposition and planning, multi-stage reasoning and decision-making, result interpretation and report generation. In this embodiment, a lightweight version of Mistral, an open-source Korean language support model, is used to optimize it for execution on a local workstation.

[0119] The inspection plan and script generator (230) combines the outputs of VIT and LLM to perform work unit subdivision, script generation and command configuration, resource allocation and priority determination, and exception response scenario planning, and establishes a step-by-step plan using the Chain of Action Thought (CoAT) method.

[0121] Looking at the detailed operation of the Vision Branch (210), in the image preprocessing (211) step, the screen capture image is resized to 224×224 pixels and divided into patches (16×16). In the ViT-B / 16 encoder (212) step, each patch is encoded into a 768-dimensional vector using 86M parameters and 12 Transformer layers. In the visual embedding (213) step, a 768-dimensional vector representing the entire image is generated.

[0123] Looking at the detailed operation of the Language Branch (220), in the text tokenization (221) step, the verification standard text is tokenized using a BPE tokenizer. In the LLM encoder (222) step, a 4-bit quantized version of the Mistral-7B model is used, which uses about 4GB of memory and is processed through 32 Transformer layers. In the language embedding (223) step, the meaning of the text is encoded into a 768-dimensional vector.

[0125] The multimodal fusion layer (231) includes 12 cross-attention layers (2311, 2312, 2313), and fuses visual information and linguistic information by cross-referencing the Query as a language embedding (223) and the Key / Value as a visual embedding (213). For example, it combines the text meaning of "total assets of financial statements" with the location of total assets on the screen. In the fusion embedding (2314) step, a 768-dimensional integrated multimodal representation is generated.

[0127] In the work plan decoder (232), the work classification head (2321) classifies the inspection type (form verification, content verification, compliance, etc.) and outputs the probability of each type to Softmax. The work sequence generator (2322) generates a work sequence in an autoregressive manner (e.g., [Form Verification] → [Required Item Verification] → [Numerical Verification] → [Compliance Check]). The work unit subdivision (2323) breaks down high-level inspection goals into specific detailed tasks.

[0129] In the action plan decoder (233), the action type classifier (2331) selects Click, Scroll, Extract, Type, and Screenshot to determine the action type required for each task. The coordinate predictor (2332) predicts screen coordinates (x, y) using Regression Head, outputs normalized coordinates (range 0 to 1), and converts them into actual pixel coordinates. The parameter generator (2333) generates additional parameters required for the action. The action sequence (2334) is output in the form of an executable script.

[0131] Multimodal fusion improves inspection accuracy by 15-20% compared to single modal by simultaneously considering visual information of the screen and text information of the inspection criteria. Local VLM (200) has data security as a key advantage, and since all processing is done locally, sensitive document information is not transmitted externally.

[0133] (3) ITOMS - AI Operations Task Management System (300)

[0134] ITOMS (300) is a central control system that converts and manages inspection plans generated in the local VLM (200) into an executable form. Its main functions include job scheduling, resource management, exception handling, and parallel processing. ITOMS (300) receives an inspection plan (S2000) from the local VLM (200) and converts it (S3000) into specific commands that can be executed by the RPACA (400).

[0136] Referring to Fig. 3, the Task Planning and Action Planning processes of the inspection work are illustrated in detail.

[0138] The characteristics of the Task Planning stage (232) are as follows. First, high-level abstract tasks are defined. The user's natural language verification criteria are analyzed, the overall verification goal is broken down into independent detailed tasks, and the purpose and scope of each task are clearly defined. Second, the task order is optimized. Dependencies are analyzed (e.g., numerical verification is possible only after checking essential items), parallelizable tasks are identified, and efficiency is considered. Third, consistency between tasks is ensured.

[0140] The characteristics of the Action Planning stage (233) are as follows. First, specific actions are created. Each task of Task Planning is converted into an executable low-level action, a specific location on the screen (x, y coordinates) is specified, and an action type is selected. Second, coordinate-based accuracy is ensured. The location of UI elements is identified using an AI object detection module, and the click location is determined with pixel-level accuracy. Third, the action sequence is optimized. Unnecessary actions are removed, waiting time is minimized, and error recovery actions are included.

[0142] (4) RPACA - Real-time object recognition RPA component (400)

[0143] RPACA (400) is an execution engine that performs actual inspection work according to the instructions of ITOMS (300). It performs functions such as screen element recognition, text extraction, coordinate-based actions, and screenshot collection. RPACA (400) operates in a test environment (410) and a real environment (420), respectively, and verifies in the test environment (410) first and then applies to the real environment (420).

[0145] (5) AI model(500)

[0146] The AI ​​model (500) consists of an object detection and text recognition module (510) and a state determination and error detection module (520). The object detection and text recognition module (510) uses YOLOv8 to identify tables, graphs, images, and text blocks, performs text recognition using TrOCR, and analyzes layouts. The state determination and error detection module (520) performs violation detection, logical error detection, and missing item verification.

[0148] (6) CoV verification module (600)

[0149] Referring to FIG. 4, the Chain of Vision (CoV) 4-step verification process is illustrated in detail. The CoV verification module (600) verifies the inspection results of the VLM step by step to minimize the problem of hallucination.

[0151] The first verification step (610) is a structure and format verification, which identifies the document structure through layout analysis, compares it with a standard form through template comparison, checks format parameters such as margins, fonts, and line spacing, and verifies the section order.

[0153] The second verification step (620) is a content completeness verification, which checks essential items based on a checklist, verifies text accuracy, verifies numerical consistency, and detects missing items. It verifies consistency between the first verification and the second verification.

[0155] The third verification step (630) is a compliance verification that checks for compliance with legal requirements, industry standards, and internal regulations, and checks for exceptions. It also checks for consistency between the second verification and the third verification.

[0157] The fourth verification step (640) is a logical consistency verification, which integrates the results of the first to third verifications to detect contradictions, evaluates logical validity, and calculates a reliability score.

[0159] Each verification step is performed independently, and the results are stored in the verification database (650). If a discrepancy or error is found, a re-verification process (660) is performed or a user manual verification request (670) is made.

[0161] (7) Result output module (700)

[0162] The result output module (700) generates an inspection report (710), visualizes (720), and provides a notification function (730). The inspection report includes summary information, details, findings, recommendations, and supporting materials.

[0164] (8) Self-Instruct data augmentation module (800)

[0165] Referring to FIG. 5, the Self-Instruct data augmentation process is illustrated in detail. This module automatically generates training data to continuously improve the local VLM (200).

[0167] In the seed data collection (810) stage, only the history of successful inspection cases that passed the CoV 4-stage verification is collected.

[0169] Augmentation strategies include format parameter modification (820), content reorganization (830), regulation item rearrangement (840), and form modification (850).

[0171] Format parameter transformation (820) changes margins, font size, line spacing, layout, etc. Content reorganization (830) performs synonym substitution, changes sentence order, changes expression style, etc. Regulation item rearrangement (840) performs changes checklist order, inclusion of additional regulation items, etc. Form transformation (850) converts a table format to a list format or changes the layout.

[0173] In the augmented data generation (860) step, training data is obtained by generating augmented samples for each seed data. In the VLM fine-tuning (870) step, the local VLM (200) is retrained with the augmented data. After training, the model quality is evaluated (872) by CoV validation, and if the criteria are met, it is deployed (874).

[0175] <Second Embodiment: Document Inspection Process>

[0177] Referring to FIG. 6, a document inspection process according to a second embodiment of the present invention is illustrated.

[0179] In step S100, the user enters an inspection request, and in step S1000, the task is transferred to the local VLM.

[0180] In step S1100, the local VLM captures and analyzes the screen, and in steps S1200 and S1300, an inspection plan is established through Task Planning and Action Planning.

[0181] In step S2000, the inspection plan is transmitted to ITOMS, and in steps S2100 and S3000, ITOMS processes the plan and issues commands to RPACA.

[0182] In step S4000, RPACA performs an inspection in a test environment, executes actions with the support of an AI model (S4100, S4300) (S4200), and collects inspection data (S4400).

[0183] In steps S5000 to S5300, the CoV verification module performs a 4-step verification, and if it passes (S5400), it notifies RPACA.

[0184] In step S6000, the RPACA performs a final inspection in the actual environment, and in step S6200, the final data is transferred to the CoV verification module to receive final verification (S6300).

[0185] In step S8000, the result is transferred to the result output module, where report generation (S8100), visualization (S8200), provision (S8300), and notification sending (S8400) take place.

[0186] Finally, in steps S8500 to S8700, the inspection history is saved, and Self-Instruct augmentation and VLM fine-tuning are performed.

[0188] <Third Embodiment: Automated Web Page Inspection and Monitoring>

[0190] Referring to FIG. 7, a web page automatic inspection and monitoring system according to a third embodiment of the present invention is illustrated.

[0192] When the user enters the URL and criteria for inspection (S100), the local VLM launches a web browser to access and analyze the page (S1100). After establishing an inspection plan (S1200, S1300), the RPACA performs the inspection in a test environment via ITOMS (S4000). It captures the dynamic content of the web page while scrolling and recognizes UI elements and text through an AI model. Subsequently, it provides results after undergoing CoV verification (S5000–S5300) and inspection in a real environment (S6000).

[0194] The web monitoring function utilizes a Server / Web automatic detection module (110). A scheduler (111) periodically initiates an inspection, a web browser (113) loads the page in the background, and a screenshot collector (114) captures it. A change detection engine (115) detects changes by comparing images, text, and structures, and when a violation or change is detected, an automatic inspection trigger (117) is activated to execute the entire inspection process. In the result processing (118) stage, an immediate notification is sent in the event of a violation, and the data is stored in a history management (119) database to provide time-series analysis.

[0196] <Fourth Embodiment: Local VLM Learning and Improvement>

[0198] This embodiment describes a method for continuous quality improvement of a local VLM through self-instruct data augmentation.

[0200] Successful cases (CoV verification passed) among the automatically collected inspection history are used as seed data (810). The seed data consists of screen images, inspection criteria, plans and actions, and results.

[0202] The Self-Instruct augmentation process expands the original data by more than 10 times (860) using four strategies: format parameter transformation (820), content reorganization (830), rule item rearrangement (840), and form transformation (850).

[0204] The local VLM is fine-tuned with augmented data (870). After training, the accuracy, completeness, false positive rate, consistency, etc. of the inspection are evaluated through CoV verification (872), and if the criteria (e.g., accuracy of 90% or higher) are met, it is deployed (874).

[0206] Meanwhile, the components of the VLM-based document / web automatic inspection system according to an embodiment of the present invention, such as an input interface (100), local VLM (200), ITOMS (300), RPACA (400), AI model (500), CoV verification module (600), result output module (700), and Self-Instruct data augmentation module (800), may each be implemented as a separate hardware device or as a software module executed on a computing device including one or more processors and memory. In this case, each component performs its corresponding function by utilizing hardware resources such as a processor, memory, storage device, and network interface, and the database may be implemented in a data storage device such as a hard disk drive, SSD, or cloud storage.

[0208] The embodiments according to the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded on a computer-readable medium. In this case, the medium may include a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a hardware device specifically configured to store and execute program instructions, such as a ROM, RAM, and flash memory. Furthermore, it may include a central processing unit (CPU) for executing instructions, and a hardware device for accelerating the computation of an artificial intelligence model, such as a graphics processing unit (GPU), a tensor processing unit (TPU), and a neural network processing unit (NPU).

[0210] Meanwhile, the above-mentioned computer program may be one specifically designed and configured for the present invention, or one known and available to those skilled in the art of computer software. Examples of computer programs may include not only machine code, such as that generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc. Furthermore, all or part of the functions of the present invention may include, or be implemented in conjunction with, an artificial intelligence model, machine learning model, deep learning model, foundation model, or large language model (LLM) that is pre-trained or fine-tuned for a specific purpose. Such models may be called through a service in the form of an Application Programming Interface (API) and may be utilized for the generation, analysis, transformation, or control of data to perform the functions of the present invention.

[0212] In the specification of the present invention (particularly in the claims), the use of the term "above" and similar descriptive terms may be in both singular and plural. Furthermore, where a range is described in the present invention, it is to include an invention to which individual values ​​belonging to said range are applied (unless otherwise stated), and this is equivalent to describing each individual value constituting said range in the detailed description of the invention.

[0214] Unless explicitly stated or contrary to the order of the steps constituting the method according to the present invention, said steps may be performed in a suitable order. The present invention is not necessarily limited by the order in which said steps are described. The use of all examples or exemplary terms (e.g., etc.) in the present invention is merely for the purpose of describing the present invention in detail, and the scope of the present invention is not limited by said examples or exemplary terms unless limited by the claims. Furthermore, those skilled in the art will understand that various modifications, combinations, and changes, including the application of newly emerging technical means, may be configured according to design conditions and factors within the scope of the claims or equivalents to which they are added.

[0216] Accordingly, the scope of the present invention should not be limited to the embodiments described above, and all scopes equivalent to or equivalently modified from the claims set forth below, as well as the claims set forth below, shall be considered to fall within the scope of the concept of the present invention. Explanation of the symbols

[0218] 100: Input Interface 110: Server / Web Auto-detection Module 111: Scheduler 112: URL Manager 113: Web browser 114: Screenshot Collector 115: Change detection engine 116: Change Detection Algorithm 1161: Image Comparison 1162: Text Comparison 1163: Structure Comparison 1164: Calculate Change Rate 1165: Change Rate Determination 117: Automatic Inspection Trigger 1171: Create inspection request 1172: Automatic selection of inspection criteria 1173: Determining Priority 118: Result processing 1181: Change Summary 1182: Rule violation detection 1183: Determining whether there is a violation of regulations 1184: Immediate notification 1185: Daily Report 119: History Management 1191: Change history database 1192: Time Series Analysis 1193: Compliance Rate 200: Local VLM 210: Screen Recognizer (VIT) 211: Image Preprocessing 212: ViT-B / 16 encoder 213: Visual Embeddings 220: Large-scale Language Models (LLM) 221: Text Tokenization 222: LLM encoder 223: Language Embeddings 230: Inspection Plan and Script Generator 231: Multimodal Fusion Layer 2311: Intersection Warning Layer 1-4 2312: Cross-Caution Layer 5-8 2313: Intersection Warning Layer 9-12 2314: Fusion Embedding 232: Job Plan Decoder 2321: Job Classification Head 2322: Task Sequence Generator 2323: Work Unit Subdivision 233: Action Plan Decoder 2331: Action Type Classifier 2332: Coordinate Prediction System 2333: Parameter generator 2334: Action Sequence 300: ITOMS (AI Operational Task Management System) 400: RPACA (Real-time Object Recognition RPA Component) 410: Test Environment 420: Real environment 500: AI Models 510: Object Detection and Text Recognition Module 520: Status determination and error detection module 600: CoV Verification Module 610: First Verification (Structure / Format) 611: Layout Analysis 612: Template Comparison 613: Check format parameters 614: Check section order 615: Save 1st verification result 620: Second Verification (Content Completeness) 621: Check required items 622: Check text accuracy 623: Numerical Consistency Verification 624: Missing Item Detected 625: Save 2nd verification result 626: 1st↔2nd Consistency Check 630: Third Verification (Compliance) 631: Checking Legal Requirements 632: Check Industry Standards 633: Check internal regulations 634: Check exception clauses 635: Save third verification result 636: Second↔Third consistency check 640: Verification 4 (Logical Consistency) 641: Integration of Verification Results 1–3 642: Contradiction Detection 643: Evaluation of Logical Validity 644: Calculate Confidence Score 645: Save 4th verification result 650: Validation database 660: Re-verification process 670: User manual verification request 700: Result output module 710: Generate Inspection Report 720: Visualization 730: Notification function 800: Self-Instruct Data Augmentation Module 810: Collect Seed Data 811: Seed Data Configuration 8111: Screen image subject to inspection 8112: Inspection Criteria Text 8113: Inspection Plan and Action Sequence 8114: Inspection Results and CoV Verification Results 812: Select Augmentation Strategy 820: Augmentation Strategy 1 (Format Variation) 821: Change margin 822: Change font size 823: Change line spacing 824: Change layout 825: Augmented Data Generation 830: Augmentation Strategy 2 (Content Reorganized) 831: Synonym substitution 832: Change sentence order 833: Change display method 834: Change value 835: Augmented Data Generation 840: Augmentation Strategy 3 (Reorganization of Regulations) 841: Change Checklist Order 842: Includes additional regulation items 843: Change in regulation level 844: Augmented Data Generation 850: Augmentation Strategy 4 (Form Variations) 851: Converting Table to List Format 852: Vertical layout → Horizontal layout 853: Single Page → Multiple Pages 854: Augmented Data Generation 860: Augmented Data Integration 861: Duplicate removal and quality filtering 870: VLM Fine-tuning 871: Fine-tuning settings 8711: Training data 8712: Validation data 8713: Test data 8714: Epochs 8715: Batch Size 8716: Learning Rate 872: Model Evaluation with CoV Validation 873: Determine whether evaluation passed 874: Local VLM Deployment 875: Relearning

Claims

Claim 1 In a VLM-based document / web automatic inspection system, an input interface (100) receives information on a document or web page to be inspected and inspection criteria in natural language form from a user; a vision-language model (200, Local VLM) operating in a local environment, which captures the screen of the document or web page and analyzes visual information through a screen recognition device (210, VIT), interprets the inspection criteria in natural language form through a large-scale language model (220, LLM) to convert them into structured inspection requirements and analyzes text content, and generates an inspection plan that determines one or more of the format compliance, content completeness, and regulation compliance of the document / web by combining information processed by the screen recognition device (210) and the large-scale language model (220), wherein the inspection plan includes structured inspection requirements generated by interpreting the inspection criteria in natural language form, and comprises an inspection plan and script generator (230) composed of one or more items among format verification items, content verification items, and regulation compliance items; and an inspection plan generated in the local VLM (200). An AI operational task management system (300, ITOMS) that manages processing and execution; a real-time object recognition RPA component (400, RPACA) that recognizes specific UI elements on a document or web screen, extracts text, and checks each item of a regulation checklist according to the instructions of the AI ​​operational task management system (300);A Chain of Vision (CoV) inspection verification module (600) that verifies inspection data collected by the above-mentioned real-time object recognition RPA component (400) step-by-step in a document / web inspection process, wherein the Chain of Vision (CoV) inspection verification module (600) is a verification mechanism dedicated to the inspection process to minimize the problem of VLM hallucination occurring during the actual inspection process, rather than verifying the quality of VLM training data, comprising: a first verification step (610) that verifies whether the basic format and structure of the document / web conform to predefined template standards and checks margins, fonts, line spacing, and section order; a second verification step (620) that verifies whether required items are written, the accuracy of spelling / grammar / technical terms of the text, and the consistency of sums and ratios of numerical data; a third verification step (630) that verifies compliance with legal requirements, industry standards, and internal regulations based on a checklist; and verifies whether there are no contradictions between the results of the first to third verification steps and verifies the logical validity of the overall inspection result A chain of vision (CoV) inspection verification module (600) that performs a fourth verification step (640) for re-evaluation; and a result output module (700) that generates a report of the inspection completed results and provides it to the user;The system includes, but is characterized by automating the entire inspection process for the automatic inspection of documents and web content, so that when a user provides inspection criteria in natural language form, it automatically performs document / web capture, analysis, verification, and result generation; the Chain of Vision (CoV) inspection verification module (600) stores the verification results for each verification step in a verification database (650), compares consistency with the verification results of the previous step, and performs a re-verification process (660) or provides a user manual verification request (670) when a discrepancy is found; the verification database (650) stores the verification results of the first to fourth verification steps (610, 620, 630, 640) step by step and automatically detects whether there is a logical contradiction between the verification results of each step, thereby minimizing the hallucination problem of VLM during the document / web inspection process and guaranteeing the accuracy of one or more inspections of format, content, and compliance; and the local VLM (200) maintains the security of sensitive document information by performing all processing in a local environment without using an external cloud API. A VLM-based automated document / web inspection system. Claim 2 A VLM-based document / web automatic inspection system according to claim 1, wherein the Chain of Vision (CoV) inspection verification module (600) checks whether the layout of the document / web matches a predefined template in the first verification step (610) and verifies one or more formatting parameters among margins, fonts, and line spacing; checks whether required items are written in the second verification step (620) and verifies the accuracy of one or more of the spelling, grammar, and technical terms of the text, and verifies the consistency of sums and ratios of numerical data; verifies compliance with one or more of legal requirements, industry standards, and internal regulations based on a checklist in the third verification step (630); and checks whether there are no contradictions between the results of the first to third verification steps (610, 620, 630) in the fourth verification step (640) and re-evaluates the logical validity of the overall inspection results. Claim 3 In claim 1, the AI ​​operation task management system (300) performs a task planning step (232) that analyzes the inspection criteria and divides the entire inspection task into detailed units, and an action planning step (233) that converts each detailed task into a specific action for a specific location or element on the screen, wherein the task planning step (232) analyzes the inspection goal and divides it into one or more detailed tasks among form verification, required item verification, numerical verification, and compliance verification, and the action planning step (233) generates one or more specific action sequences among click, scroll, text extraction, and screenshot capture for each detailed task, and includes the location coordinates of the target UI element for each action, characterized by a VLM-based document / web automatic inspection system. Claim 4 A VLM-based document / web automatic inspection system according to claim 1, further comprising an AI object detection and text recognition module (510) that supports the real-time object recognition RPA component (400), wherein the AI ​​object detection and text recognition module (510) uses a YOLO-based object detection model to identify one or more UI elements among tables, graphs, images, text blocks, buttons, and input fields on a document / web screen and provides location coordinates, and uses a Transformer-based OCR model to extract text on the screen and provide content information. Claim 5 In claim 1, the self-instruct data augmentation module (800) for training the local VLM (200) is further included, wherein the self-instruct data augmentation module (800) comprises: a seed data collection unit (810) for collecting seed inspection data from an inspection work history; an augmentation data generation unit (860) for generating training data at least 10 times the original data by applying one or more augmentation strategies to the seed inspection data, such as a format parameter transformation strategy (820) for changing one or more parameters among margin, font size, and line spacing; a content reconstruction strategy (830) for replacing text with synonyms or changing sentence order; a regulation item rearrangement strategy (840) for changing the checklist order or including additional regulation items; and a format transformation strategy (850) for applying the same content to different document forms; and a VLM fine-tuning unit (870) for fine-tuning the local VLM (200) with the generated training data. Document / Web Automated Review System. Claim 6 A VLM-based document / web automatic inspection system according to claim 1, wherein the system separates the test environment (410) and the actual environment (420), first executes the inspection plan in the test environment (410) to verify accuracy, and performs a final inspection in the actual environment (420) after the verification through the Chain of Vision (CoV) inspection verification module (600) is completed. Claim 7 A VLM-based document / web automatic inspection system, wherein the input interface (100) includes a server / web automatic detection module (110), and the server / web automatic detection module (110) includes a change detection engine (115) that periodically automatically captures a designated web page to detect changes, an automatic inspection trigger (117) that automatically transmits an inspection request to the AI ​​operation task management system (300) when changes are detected, and a result processing unit (118) that provides real-time notifications to the user when violations of regulations are found. Claim 8 A VLM-based document / web automatic inspection system according to claim 1, wherein the local VLM (200) processes the screen recognition unit (210) and the large-scale language model (220) in parallel to simultaneously analyze visual information and text information, and combines the visual information and text information through a multimodal fusion layer (231), wherein the multimodal fusion layer (231) learns the interaction between the two pieces of information using a cross-attention mechanism or a gate mechanism, and improves the accuracy of inspection judgment using the fused expression. Claim 9 A VLM-based document / web automatic inspection system, wherein the self-instruct data augmentation module (800) comprises a chain of vision (CoV) verification evaluation unit (872) that evaluates one or more of inspection accuracy, completeness, and consistency using the chain of vision (CoV) inspection verification module (600) in a test dataset to evaluate the quality of the fine-tuned local VLM (200), and a local VLM distribution unit (874) that distributes the local VLM (200) to the system only when the inspection accuracy is 90% or higher, the completeness is 95% or higher, and the consistency is 95% or higher. Claim 10 A VLM-based automated document / web inspection method comprises: (a) receiving information on the document or web page to be inspected and inspection criteria in natural language form from a user (S100); (b) capturing and analyzing the current screen through a vision-language model (200, Local VLM) operating in a local environment (S1100), wherein one or more visual information among the format, structure, and UI elements of the document / web are processed through a screen recognizer (210, VIT), and the inspection criteria in natural language form are interpreted through a large-scale language model (220, LLM) and converted into structured inspection requirements; (c) inputting the inspection criteria and screen analysis results into the local VLM (200) to automatically generate an inspection work plan and script through two stages of task planning (232) and action planning (233) for document / web inspection (S1200, S1300); and (d) processing the generated script in an AI operational task management system (300, ITOMS). Step (S2100); (e) A step of recognizing UI elements on a document or web screen through a real-time object recognition RPA component (400, RPACA), extracting text, and verifying each item of a regulation checklist (S4000, S4100, S4200, S4300, S4400);(f) A step (S5000, S5100, S5200, S5300) of verifying the inspection results through a four-step verification process via a Chain of Vision (CoV) inspection verification module (600) in the document / web inspection process, wherein the first verification (610) verifies whether the basic format and structure of the document / web conform to predefined template standards and checks one or more of margins, fonts, line spacing, and section order; the second verification (620) verifies whether required items are written, the accuracy of spelling / grammar / technical terms of the text, and the consistency of sums and ratios of numerical data; the third verification (630) verifies compliance with one or more of legal requirements, industry standards, and internal regulations based on a checklist; and the fourth verification (640) verifies whether there are no contradictions between the results of the first to third verifications (610, 620, 630) and re-evaluates the logical validity of the entire inspection result; and (g) a step of generating a report containing one or more of findings, recommendations, and supporting materials from the verified inspection results and providing it to the user (S8100, S8200, S8300, S8400);VLM-based, wherein the task planning (232) of step (c) analyzes inspection criteria and divides them into one or more detailed inspection tasks among form verification, required item verification, numerical verification, and compliance verification; the action planning (233) of step (c) converts each detailed task into one or more specific actions among clicking, scrolling, text extraction, and screenshot capture on a specific location or element on the screen; the chain of vision (CoV) inspection verification of step (f) stores the inspection results in the verification database (650) for each inspection step and compares consistency with the inspection results of the previous step, and performs a re-verification process (660) or provides a user manual verification request (670) when a discrepancy is found, thereby minimizing the hallucination problem of VLM during the document / web inspection process and ensuring the accuracy of one or more inspections of format, content, and compliance; and the local VLM (200) maintains the security of sensitive document information by performing all processing in a local environment without using an external cloud API. Automated Document / Web Verification Method.; Claim 11 In claim 10, the above step (e) further includes steps (S4100, S4300) of supporting the operation of the real-time object recognition RPA component (400) through an AI object detection and text recognition module (510), wherein the AI ​​object detection and text recognition module (510) is characterized by using a YOLO-based object detection model to identify UI elements of a document / web screen and provide location coordinates, and using a Transformer-based OCR model to extract text content, VLM-based document / web automatic inspection method. Claim 12 A VLM-based document / web automatic inspection method, characterized in that, in claim 10, the above step (e) includes: a sub-step (S4000) of first executing the inspection work plan in a test environment (410) to verify accuracy; and a sub-step (S6000, S6100) of performing a final inspection in a real environment (420) only if the chain of vision (CoV) verification (600) is passed. Claim 13 A VLM-based document / web automatic inspection method, further comprising: a step of periodically automatically capturing a specified web page through a server / web automatic detection module (110); a change detection algorithm (116) step of detecting changes by comparing the currently captured web page with the previous capture result; an automatic inspection trigger (117) step of automatically executing steps (a) through (g) to perform an inspection when a change is detected; and a result processing unit (118) step of providing real-time notification to the user when a violation of regulations is found. Claim 14 In claim 10, (h) additionally includes a step (S8500, S8600, S8700) of continuously improving the local VLM (200) by augmenting inspection history data in a self-instruct manner, wherein the step (h) comprises: a sub-step of a seed data collection unit (810) that collects seed inspection data from inspection work history; a sub-step of an augmented data generation unit (860) that expands the original data by more than 10 times by applying one or more augmentation strategies among a format parameter transformation strategy (820), a content reconstruction strategy (830), a regulation item rearrangement strategy (840), and a form transformation strategy (850) to the seed inspection data; and a sub-step of a VLM fine-tuning unit (870) that fine-tunes the local VLM (200) using the expanded training data. A VLM-based document / web automatic inspection method characterized by including: a chain of vision (CoV) verification evaluation unit (872) sub-step that evaluates the inspection accuracy of the fine-tuned VLM through the chain of vision (CoV) verification method (600) above, and a local VLM distribution unit (874) sub-step that distributes to the system when the evaluation passes. Claim 15 A VLM-based automatic document / web inspection method according to claim 10, wherein the first verification (610) of step (f) includes verifying whether the layout of the document / web matches a predefined template, verifying formatting parameters such as margins, font size, and line spacing, and verifying the section structure and order; the second verification (620) of step (f) includes verifying whether all required items have been written, verifying the accuracy of the spelling, grammar, and technical terms of the text, and verifying the consistency of the sum and ratio of numerical data; the third verification (630) of step (f) includes verifying compliance with one or more of legal requirements, industry standards, and internal regulations based on a checklist; and the fourth verification (640) of step (f) includes verifying whether there are no contradictions between the results of the first to third verifications (610, 620, 630) and re-evaluating the logical validity of the overall inspection results. Claim 16 A local VLM learning method for automated document / web inspection comprises: (a) collecting seed data (810) including an image of the document / web screen to be inspected (8111), inspection criteria text (8112), inspection plan and action sequence (8113), inspection result, and Chain of Vision (CoV) 4-stage verification result (8114) from an inspection work history; (b) augmenting the seed data in a self-instruct manner (860), wherein one or more of the following are performed to expand the original data by more than 10 times: augmentation through formatting parameter modification (820) which changes one or more of margins, fonts, and line spacing; augmentation through content reconstruction including synonym substitution and sentence order change (830); augmentation through modification of the order of regulation items including checklist rearrangement (840); and augmentation through the application of various document forms including layout modification (850); (c) combining a screen recognition device (210, VIT) and a large-scale language model (220, LLM). (d) a step of configuring a local VLM (200) architecture, comprising a visual encoder (212) using a ViT family model, a language encoder (222) using an LLM lightweight model, a multimodal fusion layer (231) using a cross-attention or gate mechanism, and an output decoder (232, 233) using a multi-task head (classification + generation); (d) a step (870) of fine-tuning the local VLM (200) using the augmented training data, comprising optimizing a loss function that combines a task classification loss, an action sequence generation loss, and a verification score loss;(e) a step (872) of evaluating the inspection accuracy of a fine-tuned VLM through a Chain of Vision (CoV) verification method (600), wherein a four-stage verification (610, 620, 630, 640) is performed on a test dataset and the inspection accuracy, completeness, and consistency metrics are measured; and (f) a step (874) of deploying the local VLM (200) to an inspection system only when the inspection accuracy is 90% or higher, completeness is 95% or higher, and consistency is 95% or higher in the evaluation; wherein the Chain of Vision (CoV) verification minimizes hallucination problems by verifying the inspection results generated by the VLM in four stages, and the local VLM operates in a local environment to maintain data security without relying on external APIs, thereby comprising a local VLM learning method for automatic document / web inspection. Claim 17 A local VLM learning method for automatic document / web inspection according to claim 16, wherein the above step (a) (810) is characterized by collecting only cases where all of the Chain of Vision (CoV) 4-step verification (610, 620, 630, 640) of successful inspection cases have passed as seed data, setting the modified data as the correct label when the user has modified it, and storing inspection failure cases separately to use for error pattern analysis. Claim 18 A local VLM learning method for document / web automatic verification according to claim 16, wherein the visual encoder (212) of step (c) uses ViT-B / 16 or ViT-L / 14, the language encoder (222) uses a 4-bit or 8-bit quantized LLM and is optimized to be executable on a local workstation, and the multimodal fusion layer (231) includes 12 or more cross-attention layers (2311, 2312, 2313) to learn the interaction between visual information and text information. Claim 19 A computer-readable recording medium having a program for executing the method of any one of paragraphs 10 through 18 on a computer.

Images