System and method for automated web navigation and usability evaluation using multimodal large language models

Abstract

The present invention relates to an automated system and method for web navigation and usability evaluation utilizing a multimodal large-scale language model. The automated web navigation and usability evaluation system utilizing a multimodal large-scale language model according to the present invention comprises an input data setting module for collecting data from a web page, an automated web navigation module for performing operations within a web page using a multimodal large-scale language model, a usability evaluation module for performing tests on user scenarios and evaluating the usability of said web page, and an action log storage module for storing log data generated during the web navigation and usability evaluation process.

Description

Technology Field

[0001] The present invention relates to an automated system and method for web navigation and usability evaluation utilizing a multimodal large-scale language model. Background Technology

[0003] Script-based web crawling according to conventional technology operates based on static data, which has the problem of making it difficult to accurately recognize and process dynamic and complex user interfaces.

[0004] Conventional usability evaluation methods are performed manually, which not only consumes a significant amount of time and cost but also suffers from a lack of consistency in collected data and requires additional analysis time, making it difficult to rapidly improve web pages. The problem to be solved

[0006] The present invention is proposed to solve the aforementioned problems and aims to provide a system and method for automating web navigation and usability evaluation by utilizing a large-scale language model. means of solving the problem

[0008] The automated web navigation and usability evaluation system utilizing a multimodal large-scale language model according to the present invention comprises an input data setting module for collecting data from a web page, an automated web navigation module for performing operations within a web page using a multimodal large-scale language model, a usability evaluation module for performing tests on user scenarios and evaluating the usability of said web page, and an action log storage module for storing log data generated during the web navigation and usability evaluation process.

[0009] The above input data setting module captures the image data and HTML source code of the web page and provides them as input values ​​for the multimodal large-scale language model.

[0010] The above input data setting module collects multiple screenshots considering screen size and resolution, and analyzes the above HTML source code to extract elements necessary for navigation.

[0011] The above-mentioned automated web navigation module interacts with the interface of the web page using the above-mentioned multimodal large-scale language model, and when moving to another web page through an action, it retrieves data again from the above-mentioned input data setting module.

[0012] The above-described automated web navigation module analyzes the current state of the web page, generates an action plan for achieving the user's goal, and generates feedback information by analyzing the action plan, action, reason for action selection, and change information within the page.

[0013] The above-mentioned automated web navigation module stores the history of the operation plan, operation, reason for operation selection, and feedback.

[0014] The above usability evaluation module defines criteria for evaluating the usability of the web page and measures the performance of the web page according to the criteria.

[0015] A method for automating web navigation and usability evaluation using a multimodal large-scale language model according to the present invention comprises: (a) a step of collecting and preprocessing data of a web page according to a test scenario; (b) a step of automatically performing operations within a web page; and (c) a step of evaluating the usability of a web page and storing log data.

[0016] Step (a) above collects the data including image data and HTML source code of the web page.

[0017] Step (b) above uses a multimodal large-scale language model to analyze the current state of the web page, generate an action plan, execute the action, and generate feedback information using transformation information within the page.

[0018] Step (c) above measures the performance of the web page according to criteria for evaluating the usability of the web page. Effects of the invention

[0020] According to the present invention, it is possible to automatically perform various operations of a web page by utilizing a multimodal large-scale language model, and by having the AI ​​automatically analyze and perform even ambiguous goals that are not clearly defined by the user, it is possible to significantly improve the efficiency of web navigation.

[0021] According to conventional technology, web crawling and user simulation are based on static data, which limits their ability to handle dynamic and complex user interfaces. In contrast, the present invention solves the aforementioned problems and has the effect of performing more sophisticated automation by comprehensively analyzing the visual and text elements of a web page.

[0022] According to the present invention, by utilizing AI to automatically test various user scenarios and efficiently analyzing the design and interface elements of a web page to quickly identify usability issues and derive improvement measures, there is an effect of increasing the accuracy and efficiency of usability testing and enabling rapid improvement of web pages.

[0023] According to the present invention, it is possible to systematically manage log data generated during web navigation and usability evaluation processes through an automated log storage module. By enabling rapid searching and analysis of collected data through an optimized database-based log storage system, it is possible to quickly identify problems with web pages and provide specific improvement measures. Furthermore, the automated management and analysis of log data contribute to maintaining consistent quality and continuously improving web pages.

[0024] According to the present invention, there is an effect of significantly improving the efficiency and user experience of web pages. By having AI automatically perform various operations of web pages and rapidly identify problems through usability evaluation to derive improvement measures, the design and interface of web pages are optimized, ultimately increasing user satisfaction and improving the performance of web pages.

[0025] According to the present invention, it is possible to utilize it effectively in various application fields such as e-commerce, information retrieval, and online marketing.

[0026] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below. Brief explanation of the drawing

[0028] FIG. 1 illustrates a web navigation and usability evaluation automation system utilizing a multimodal large-scale language model according to an embodiment of the present invention. FIG. 2 illustrates the configuration of an input data setting module according to an embodiment of the present invention. FIG. 3 illustrates the configuration of an automated web navigation module according to an embodiment of the present invention. FIG. 4 illustrates the configuration of a usability evaluation module according to an embodiment of the present invention. FIG. 5 illustrates a method for automating web navigation and usability evaluation using a multimodal large-scale language model according to an embodiment of the present invention. FIG. 6 is a block diagram showing a computer system for implementing a method according to an embodiment of the present invention. Specific details for implementing the invention

[0029] The aforementioned objectives of the present invention, as well as other objectives, advantages, and features, and the methods for achieving them, will become clear from the embodiments described in detail below together with the accompanying drawings.

[0030] However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms, and the following embodiments are provided merely to easily inform those skilled in the art of the purpose, structure, and effects of the invention, and the scope of the rights of the present invention is defined by the description in the claims.

[0031] Meanwhile, the terms used in this specification are for describing the embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. As used in this specification, "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.

[0032] Below, the background of the proposed invention is explained, followed by a description of a preferred embodiment of the invention.

[0033] With the advancement of artificial intelligence and computer vision technologies, research and technological development aimed at automation and efficiency improvement are actively underway across various industrial sectors. As the need for automated web page navigation and usability evaluation increases, conventional technologies related to simple script-based web crawling and user simulation have been proposed.

[0034] According to conventional technology, there are limitations in interacting with the complex visual elements of web pages. Conventional technology, which operates based on static data such as HTML source code, faces difficulties in accurately recognizing and processing dynamic and complex user interfaces. Furthermore, it fails to effectively handle screen images on web pages, resulting in deficiencies in the analysis and interaction of visual elements. Additionally, it is limited to being suitable only for clearly defined user goals, making it unsuitable for processing ambiguous commands or complex user scenarios.

[0035] Conventional usability evaluation methods are primarily conducted manually, which results in significant time and cost consumption. In other words, not only is a large workforce required for usability evaluation, but manually collected data also lacks consistency and requires additional time for analysis, making it difficult to rapidly improve web pages.

[0036] The present invention is proposed to solve the aforementioned problems and proposes a system and method for automating web page navigation and usability evaluation based on a multimodal large-scale language model, which is capable of automatically performing various operations of a web page using visual elements of the web page and HTML source code as input data.

[0037] According to an embodiment of the present invention, by utilizing a multimodal large-scale language model, there is an advantage in that automatic analysis using AI and automatic execution of web operations are possible even for ambiguous goals that are not clearly defined by the user. The AI ​​accurately recognizes and processes complex user interfaces by combining image data and HTML source code of a web page.

[0038] According to an embodiment of the present invention, web usability evaluation is performed automatically by utilizing an automated web navigation function. By testing various user scenarios and efficiently analyzing the design and interface elements of web pages to identify usability issues, the AI ​​makes it possible to reduce testing costs and rapidly improve web pages.

[0039] According to an embodiment of the present invention, by supporting the recording and analysis of the web navigation and usability evaluation process through automated log storage, it is possible to quickly identify problems with web pages, derive improvement measures, and provide results of consistent quality.

[0040] According to an embodiment of the present invention, by utilizing a large-scale vision model and a large-scale language model to automate the navigation, interaction, and usability evaluation of web pages, the overall user experience of web pages is improved, and it can be effectively utilized in various application fields such as e-commerce, information retrieval, and online marketing.

[0041] FIG. 1 illustrates a web navigation and usability evaluation automation system utilizing a multimodal large-scale language model according to an embodiment of the present invention.

[0042] A web navigation and usability evaluation automation system utilizing a multimodal large-scale language model according to an embodiment of the present invention comprises a test scenario generation module (110) for generating various usage scenarios, an input data setting module (120), an automated web navigation module (130), a usability evaluation module (140), and an action log storage module (150).

[0043] The test scenario generation module (110) generates various usage scenarios, for example, as a scenario for evaluating the usability of an e-commerce website, it generates a scenario in which a user searches for, selects, adds to a shopping cart, and makes a payment. The test scenario generation module (110) is created through a rule-based or large-scale language model.

[0044] The input data setting module (120) captures image data and HTML source code of a web page related to a test scenario and provides them as input values ​​for a multimodal large-scale language model. The input data setting module (120) collects and preprocesses the necessary data from the web page and converts it into a format required for analysis.

[0045] The automated web navigation module (130) automatically performs various actions within a web page by utilizing a multimodal large-scale language model. Through algorithms related to AI agents, the automated web navigation module (130) interacts with the complex interface of the web page by having the multimodal large-scale language model recognize visual elements and analyze the HTML structure. The automated web navigation module (130) is capable of automatically analyzing and performing actions even for ambiguous goals that are not clearly defined by the user. The automated web navigation module (130) interacts with the input data setting module (120), and when moving to another web page through actions such as clicking, it receives new data from the input data setting module (120).

[0046] The usability evaluation module (140) utilizes the automated web navigation module (130) to test various user scenarios and evaluate the usability of web pages.

[0047] The action log storage module (150) automatically saves log data generated during the web navigation and usability evaluation process so that it can be analyzed by a person later.

[0048] FIG. 2 illustrates the configuration of an input data setting module (120) according to an embodiment of the present invention.

[0049] The input data setting module (120) collects screenshot image data and HTML data for a given URL, preprocesses them, and transmits them to the automated web navigation module (130).

[0050] The input data setting module (120) includes a website scraping module (121) that collects screenshot data and HTML data from a given URL, and a data parser preprocessing module (122) that selects HTML data related to actual web navigation.

[0051] The website scraping module (121) collects screenshot image data and HTML data of a web page for a given URL. The website scraping module (121) collects multiple screenshots considering various screen sizes and resolutions, and collects the entire HTML source code so that it can be utilized in a subsequent preprocessing step. According to an embodiment of the present invention, visual elements and structural data of a web page can be provided as input values ​​for a multimodal large-scale language model.

[0052] The data parser preprocessing module (122) analyzes HTML data collected from the website scraping module (121), analyzes the HTML structure to extract only the key elements necessary for navigation, and provides them as input values ​​for a multimodal large-scale language model.

[0053] FIG. 3 illustrates the configuration of an automated web navigation module (130) according to an embodiment of the present invention.

[0054] An automated web navigation module according to an embodiment of the present invention includes a planning module (131), an execution module (132), a feedback module (133), and a memory module (134), and utilizes a plurality of multimodal large-scale language models (LLM).

[0055] The planning module (131) is the first step of automated web navigation and uses a multimodal large-scale language model to receive input values ​​from the input data setting module (120) and create a plan to be executed within the web page. The input values ​​consist of screenshot image data and HTML source code, and based on this, the multimodal large-scale language model analyzes the current state of the web page. Through prompt engineering techniques such as Chain-of-Thought (CoT), ambiguous commands are first clearly defined, and the plan is divided into multiple tasks to allow for step-by-step resolution. In this process, the large-scale language model comprehensively grasps the structure and visual elements of the web page to generate an optimal action plan to achieve the user's goal. The generated plan includes specific actions to be performed and the reasons for selecting those actions.

[0056] The execution module (132) performs actions on the actual web page according to the plan generated by the planning module (131). It defines presets for all actions a user can perform on the web (e.g., clicking, scrolling, text input, etc.) and allows a multimodal large-scale language model to select the corresponding action and the HTML element to perform that action. Various actions can be performed by selecting the action with the maximum value of the probability distribution to select the optimal action, or by adjusting a temperature parameter for navigation. After performing each action, the execution module (132) records the results, and these records provide important data for subsequent feedback and evaluation processes. Additionally, the execution module (132) includes basic error handling functions to respond to unexpected situations that occur during the execution of actions.

[0057] The feedback module (133) uses another multimodal large-scale language model to evaluate the plan of the planning module (131) and the execution results of the execution module (132). The feedback module (133) determines whether the plan is proceeding in the right direction by comprehensively analyzing the plan, the action, the reason for selecting the action, and the changed information within the given page. The feedback module (133) checks whether the goal has been reached closer by comparing the before and after states through the linguistic feedback provided by the multimodal large-scale language model. The feedback module (133) monitors changes in the state of the page in real time, detects discrepancies between the plan and the execution results, and provides feedback to correct them. The planning module (131) receives the feedback and readjusts or modifies the plan.

[0058] The memory module (134) serves as a buffer that stores all history generated during the automated web navigation process. The memory module (134) temporarily stores details of plans, actions, and feedback, allowing them to be referenced during subsequent decision-making processes. The memory module (134) uses a time-series database or cache memory to effectively manage history data, thereby enhancing the system's learning and adaptive capabilities. The history stored in the memory module (134) can also be utilized later during analysis and optimization processes.

[0059] FIG. 4 illustrates the configuration of a usability evaluation module (140) according to an embodiment of the present invention.

[0060] The usability evaluation module (140) according to an embodiment of the present invention includes an evaluation criteria setting module (141) and an evaluation element performance measurement module (142).

[0061] The evaluation criteria setting module (141) sets criteria for evaluating web page usability and defines specific criteria for evaluating web page usability based on various user scenarios. The evaluation criteria include elements such as web page accessibility, responsiveness, intuitiveness, and efficiency. The evaluation criteria setting module (141) utilizes a multimodal large-scale language model to comprehensively analyze the visual and structural elements of the web page and sets weights for each evaluation criterion to increase the objectivity and consistency of the evaluation.

[0062] The evaluation element performance measurement module (142) measures the performance of each element of the web page based on the criteria defined in the evaluation criteria setting module (141). The evaluation element performance measurement module (142) uses a multimodal large-scale language model to interact with various user interface (UI) elements of the web page and quantitatively evaluates the performance of each element. For example, it measures performance indicators such as button click response time and user task completion time. The evaluation results are provided as feedback to improve the usability of the web page and, if necessary, are passed to the evaluation criteria setting module (141) to be used to adjust or update the criteria. Through this, it is possible to systematically and precisely evaluate the usability of the web page.

[0063] FIG. 5 illustrates a method for automating web navigation and usability evaluation using a multimodal large-scale language model according to an embodiment of the present invention.

[0064] A method for automating web navigation and usability evaluation using a multimodal large-scale language model according to an embodiment of the present invention includes a step of collecting and preprocessing data of a web page according to a test scenario (S510), a step of automatically performing operations within a web page (S520), and a step of evaluating the usability of a web page and storing log data (S530).

[0065] Step S510 collects data including image data and HTML source code of web pages.

[0066] Step S520 uses a multimodal large-scale language model to analyze the current state of the web page, generate an action plan, execute actions, and generate feedback information using transformation information within the page.

[0067] Step S530 measures the performance of the web page according to criteria for evaluating the usability of the web page.

[0068] FIG. 6 is a block diagram showing a computer system for implementing a method according to an embodiment of the present invention.

[0069] Referring to FIG. 6, a computer system (1300) may include at least one of a processor (1313), memory (1330), an input interface device (1350), an output interface device (1360), and a storage device (1340) that communicate via a bus (1370). The computer system (1300) may also include a communication device (1320) coupled to a network. The processor (1310) may be a central processing unit (CPU) or a semiconductor device that executes instructions stored in memory (1330) or storage device (1340). Memory (1330) and storage device (1340) may include various forms of volatile or non-volatile storage media. For example, memory may include read-only memory (ROM) and random access memory (RAM). In the embodiments of this description, memory may be located inside or outside the processor, and memory may be connected to the processor through various known means. Memory is a volatile or non-volatile storage medium of various forms, and for example, memory may include read-only memory (ROM) or random access memory (RAM).

[0070] Accordingly, embodiments of the present invention may be implemented as a method implemented on a computer or as a non-transient computer-readable medium storing computer-executable instructions. In one embodiment, when executed by a processor, the computer-readable instructions may perform a method according to at least one aspect of the present description.

[0071] The communication device (1320) can transmit or receive wired or wireless signals.

[0072] In addition, the method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and may be recorded on a computer-readable medium.

[0073] The above computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the computer-readable medium may be specially designed and configured for embodiments of the present invention, or they may be known and available to a person skilled in the art of computer software. The computer-readable recording medium may include a hardware device configured to store and execute program instructions. For example, the computer-readable recording medium may be magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; ROM; RAM; flash memory, etc. The program instructions may include not only machine code, such as that generated by a compiler, but also high-level language code that can be executed by a computer through an interpreter, etc.

[0074] Although embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims also fall within the scope of the present invention.

Claims

Claim 1 An input data setting module that collects data by capturing image data and HTML source code of a web page, collects multiple screenshots considering screen size and resolution, analyzes the HTML source code to extract elements necessary for navigation, and provides them as input values ​​for a multimodal large-scale language model; an automated web navigation module that performs operations within the web page using the multimodal large-scale language model; and a usability evaluation module that performs tests on user scenarios and evaluates the usability of the web page.It includes an action log storage module that stores log data generated during the web navigation and usability evaluation process, and the automated web navigation module defines presets for actions performed by a user on the web, including clicks, scrolls, and text input, and selects actions and HTML elements that are targets for the actions using the multimodal large-scale language model, and the usability evaluation module measures performance indicators including button click response time and user task completion time, and provides the evaluation results as feedback to improve the usability of the web page, and the automated web navigation module includes a planning module, an execution module, a feedback module, and a memory module, and utilizes a plurality of the multimodal large-scale language models, and the planning module generates an operation plan by dividing commands into a plurality of tasks so that commands can be solved step-by-step using Chain-of-Thought (CoT) prompt engineering technology using any one of the plurality of multimodal large-scale language models, and the execution module selects an action with a maximum probability distribution or adjusts temperature parameters for exploration according to the operation plan generated by the planning module using any one of the plurality of multimodal large-scale language models. A web navigation and usability evaluation automation system utilizing a multimodal large-scale language model, wherein the above action and the HTML element to which the above action is performed are selected, the feedback module uses a multimodal large-scale language model different from the multimodal large-scale language model used by the planning module and the execution module to compare the state before and after the execution of the above action through linguistic feedback, and if a discrepancy between the operation plan and the result of the execution of the above action is detected, the feedback is provided to the planning module to correct it, the planning module receives the feedback and readjusts the operation plan, and the memory module stores the history of the operation plan, the action, and the feedback. Claim 2 delete Claim 3 delete Claim 4 A web navigation and usability evaluation automation system utilizing a multimodal large-scale language model, wherein, in claim 1, the automated web navigation module interacts with the interface of the web page using the multimodal large-scale language model and receives data again from the input data setting module when moving to another web page through an action. Claim 5 In claim 4, the automated web navigation module analyzes the current state of the web page, generates the operation plan for achieving the user's goal, and generates feedback information by analyzing the operation plan, operation, reason for operation selection, and change information within the page, thereby forming an automated web navigation and usability evaluation system utilizing a multimodal large-scale language model. Claim 6 In claim 5, the above-mentioned automated web navigation module stores the history of the operation plan, operation, reason for operation selection, and feedback, in a web navigation and usability evaluation automation system utilizing a multimodal large-scale language model. Claim 7 delete Claim 8 delete Claim 9 delete Claim 10 delete Claim 11 delete

Images