Programs, information processing devices, methods, and systems

Abstract

The purpose of this disclosure is to enable the evaluation of a realistic operational experience that closely resembles human testing. [Solution] This is a program for operating a computer equipped with a processor and memory. The program causes the processor to perform the following steps: accept the setting of a virtual user persona and test conditions for the virtual user for the target of evaluation; generate an autonomous agent that behaves as a virtual user based on the test conditions; cause the autonomous agent to acquire information about the user interface of the target of evaluation and recognize the operation elements included in the information about the user interface; and cause the autonomous agent to decide and execute an operation to be performed on the recognized operation elements.

Description

Technical Field

[0001] The present disclosure relates to programs, information processing apparatuses, methods, and systems.

Background Art

[0002] Conventionally, in usability testing (operability evaluation) of websites and applications, the mainstream method has been to actually have human subjects operate and observe their actions. While this method can observe the reactions of actual users, it has problems such as requiring a great deal of time for recruiting subjects and scheduling adjustments, and high costs such as compensation.

[0003] Also, attempts have been made to automate testing using AI (Artificial Intelligence) technology. For example, Patent Document 1 discloses a technique in which an AI impersonates a specific persona and conducts a test by interacting with a system.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In a dialogue-type AI technology such as that of Patent Document 1, while it is useful for evaluating dialogue systems such as chatbots, it has not been fully able to realize the evaluation of a realistic operation experience where a user with a specific persona actually visually recognizes the user interface (UI) of a website or application and operates while aiming at a goal and making trial and error.

[0006] An object of the present disclosure is to realize an evaluation of a realistic operation experience close to human testing.

Means for Solving the Problems

[0007] To solve the above problems, a program according to one aspect of the present disclosure is a program for operating a computer comprising a processor and memory, the program causing the processor to perform the following steps: receiving a virtual user persona and test conditions for the virtual user for a target to be evaluated; generating an autonomous agent that behaves as a virtual user based on the test conditions; causing the autonomous agent to acquire information about the user interface of the target to be evaluated and to recognize the operation elements included in the information about the user interface; and causing the autonomous agent to determine and execute an operation to be performed on the recognized operation elements. [Effects of the Invention]

[0008] According to this disclosure, it is possible to evaluate a realistic operational experience that closely resembles human testing. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the overall configuration of the system according to this embodiment. [Figure 2] This is a block diagram showing an example of a terminal device hardware configuration. [Figure 3] This is a block diagram showing an example of the functional configuration of a terminal device. [Figure 4] Block diagram showing an example of a server hardware configuration. [Figure 5] This block shows an example of a server's functional configuration. [Figure 6] This figure shows an example of a prompt configuration. [Figure 7] This figure shows an example of the data structure of a test management table. [Figure 8] This figure shows an example of the data structure of a persona table. [Figure 9] This figure shows an example of the data structure of a log table. [Figure 10]This is a sequence diagram showing an example of the evaluation process according to this embodiment. [Figure 11] This figure shows an example of how the project list screen looks. [Figure 12] This figure shows an example of the test group list screen. [Figure 13] This figure shows an example of the test group details screen. [Figure 14] This figure shows an example of how the session details screen is displayed. [Figure 15] This figure shows an example of the session restart settings screen (intervention function). [Figure 16] This figure shows an example of the status export screen. [Modes for carrying out the invention]

[0010] The embodiments of this disclosure will be described below with reference to the drawings. In all the drawings illustrating the embodiments, common components are denoted by the same reference numerals, and repeated explanations are omitted. The following embodiments are not intended to unduly limit the content of this disclosure as described in the claims. Not all components shown in the embodiments are necessarily essential components of this disclosure. Also, each drawing is a schematic diagram and is not necessarily a strict illustration.

[0011] Furthermore, in the following description, "processor" refers to one or more processors. A processor may be expressed as, for example, processing circuitry. At least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may be other types of processors such as a GPU (Graphics Processing Unit). At least one processor may be single-core or multi-core. At least one processor may be a general-purpose processor or a dedicated-purpose processor. Note that the processor in this invention is not limited to a physical processor, but may also be a logical processor realized by virtualization technology.

[0012] Further, at least one processor may be a processor in a broad sense, such as a hardware circuit (e.g., FPGA (Field-Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) that performs part or all of the processing.

[0013] Also, in the following description, expressions such as "xxx table" may be used to describe information from which an output can be obtained for an input. However, this information may be data of any structure or a learning model such as a neural network that generates an output for an input. Furthermore, the data handled in the present invention is not limited to structured data and may be unstructured data. Therefore, "xxx table" can be referred to as "xxx information".

[0014] Also, in the following description, the configuration of each table is an example. One table may be divided into two or more tables, or all or part of two or more tables may be one table. Furthermore, these tables may be realized as a database, a file, an object storage, or a combination thereof.

[0015] The program may be pre-installed on the information processing device described below, or, for example, the information processing device may be on a readable (e.g., non-temporary) recording medium and the program may be installed on the information processing device. Alternatively, the program may be sent from a program distribution server to the information processing device and installed there. Furthermore, in the following description, two or more programs may be implemented as one program, or one program may be implemented as two or more programs. In this specification, "information processing device" is not limited to a single device but is a concept that includes multiple devices that cooperate via a network. Therefore, the program may be configured to be executed in a distributed manner across client and server devices, and the processing according to the present invention may be realized by these devices cooperating via a network. Furthermore, in this specification, "execution of a program" is not limited to execution on a specific physical device but is a concept that includes execution in a virtualization environment, container environment, serverless execution environment, edge computing environment, cloud computing environment, or a distributed execution environment that combines these. In addition, the program may be executed as a software agent that is autonomously started, cooperates, or stopped without human intervention, and the processing according to the present invention may be realized by multiple execution entities working in cooperation or in a chain reaction.

[0016] Furthermore, while various types of identification information are used in the following description, the identification information only needs to be information that indicates a predetermined object, and the specific data is not limited to the embodiments. The identification information may be an identification number, or an identifier that includes letters or symbols.

[0017] <1. Overview>

[0018] The evaluation system according to this embodiment is a system that automatically evaluates the operability of a wide range of evaluation targets, from software such as websites and applications (regardless of whether they are GUI or CUI) to hardware products such as car navigation systems, home appliances, or robots. This system utilizes information processing technologies such as generative AI (large-scale language models, etc.), image recognition models, rule-based algorithms, or combinations thereof to generate an entity that behaves not merely as an automated robot, but as a "virtual user" (hereinafter also referred to as an "autonomous agent" in this embodiment) with specific attributes (personas) and psychological states (internal states or personality traits, etc.). This disclosure can be applied not only to the evaluation of the usability of websites and applications, but also to the automation of software tests such as functional tests, regression tests, exploratory tests, and performance evaluations. In particular, by simulating human cognition and emotions, it enables the detection of bugs caused by human errors that tend to be overlooked in conventional automated tests, as well as problems involving complex user behavior.

[0019] This autonomous agent not only executes pre-programmed, fixed procedures, but also perceives information about the user interface obtained from the object being evaluated (screen information, information about the structure or state of operating elements, the situation of the operator or the object being operated, sensor values, or sound, etc.), and autonomously decides to act according to that situation.

[0020] Furthermore, in an optional configuration, the autonomous agent possesses a human-like "internal state (knowledge, emotions, thought processes, etc.)" and has the function of learning information obtained through operations as "knowledge" and dynamically changing "emotions (stress levels, satisfaction levels, etc.)" in response to the experience of operations. In this case, the autonomous agent is characterized by updating its internal state based on all "operation-related information" observed in connection with the execution of operations. Here, "thought log" included in "operation-related information" refers to dynamic reasoning data observed accompanying the execution of operations, such as temporary hesitation, the basis for judgment, or the detection of specific errors, which occur in the process (reasoning process) in which the autonomous agent decides on an operation. The autonomous agent processes this thought log as "input information" and updates and stores the results as "thought processes" included in its "internal state". In other words, while the "thought log" is instantaneous observational information at each step, the "thought process" refers to a continuous internal state updated based on the thought log. "Operation-related information" is a concept that includes not only explicit responses to operations, but also the absence of response (no response), the passage of time after an operation is performed (waiting time), state changes that occur independently of user operation (asynchronous updates, interrupt notifications, etc.), information about the structure or state of operation elements, and changes in the situation of the operator or the object being operated on. From this information, the autonomous agent simulates changes in human knowledge and psychology, such as "the image displayed on the right side of the screen is an advertisement, so I don't need to pay attention to it," "the response to my operation was slow and frustrating," "the screen suddenly changed and I was confused," and "I suddenly received a phone call and forgot what I was doing or thinking about just before." It then reflects these changed internal states in its next action decision (interrupting the operation, hesitating, or speeding up through proficiency, etc.). This makes it possible to extract usability issues that are in line with human perception, which were difficult to detect with conventional static testing tools.

[0021] <2. Overall System Configuration> Figure 1 is a block diagram showing an example of the overall configuration of System 1. System 1 is a system for providing an operability evaluation service by an AI virtual user.

[0022] The system 1 shown in Figure 1 includes, for example, a terminal device 10, a server 20, and a generation AI system 40. The terminal device 10, server 20, and generation AI system 40 communicate with each other, for example, via a network 80. Figure 1 also shows a web server 30 that provides a website as an example of the system under evaluation. The system 1 may include the web server 30, or it may be connected to an external web server 30 via the network 80 in a way that allows it to communicate with it.

[0023] Figure 1 shows an example where System 1 includes one terminal device 10, but System 1 may include two or more terminal devices 10. Also, while Server 20 is assumed to consist of one unit, it may also be configured as a collection of multiple devices. The method of distributing the multiple functions required to realize Server 20 to multiple devices can be appropriately determined according to the processing capacity of each device and / or the specifications required for Server 20.

[0024] The terminal device 10 is, for example, an information processing device operated by a user (evaluator). The terminal device 10 can be implemented as, for example, a mobile device such as a smartphone or tablet. In this embodiment, the terminal device 10 is assumed to be a PC (Personal Computer). The terminal device 10 may also be implemented as, for example, a stationary PC or a laptop PC. The terminal device 10 displays a test condition setting screen provided by the server 20 via a web browser or a dedicated application, and functions as an interface that receives user operations (persona settings, scenario input, evaluation execution instructions, etc.) and transmits them to the server 20.

[0025] Server 20 is, for example, an information processing device that performs various processes of the evaluation system 1, and is implemented by a computer connected to the network 80. Server 20 builds a test environment in response to a request from the terminal device 10, accesses the evaluation target (for example, a website provided by the web server 30, or image data of a prototype, etc.) to obtain information about the user interface, generates appropriate prompts to query the generation AI system 40, makes decisions on actions as a virtual user and updates the internal state, and returns the evaluation results to the terminal device 10.

[0026] The web server 30 is an information processing device that holds and provides content such as a website to be evaluated. In addition to its functions as a general HTTP server, the web server 30 may also have bidirectional communication capabilities using WebSocket or other communication protocols. The web server 30 transmits the relevant data via the network 80 in response to a request from the server 20 (specifically, the browser function or communication client function of an autonomous agent operating within the server 20). In this embodiment, the target of evaluation is not limited to a website running on the web server 30, but may also be image data (such as a prototype) uploaded directly to the server 20 without going through the web server 30, or an application program that runs standalone on the server 20.

[0027] The generative AI system 40 provides the use of generative AI. This disclosure describes, as an example, the case in which the generative AI system 40 provides the use of Large Language Models (LLMs), which are a type of generative AI. Large Language Models are natural language models designed to perform multiple tasks of natural language processing. Large Language Models are an example of a trained model, and are models trained using a large number of parameters (e.g., billions to hundreds of billions) and high-level computing resources. Large Language Models are computer programs or algorithms designed to perform tasks of natural language processing. For example, in natural language processing, processes such as morphological analysis, syntactic analysis, information extraction, and text generation are performed, enabling a computer to analyze human language (i.e., natural language) and perform predetermined processing. When a prompt (instruction) is input to a Large Language Model, it generates output based on the text, image, etc., of the prompt. The prompt can be defined in natural language.

[0028] Examples of large-scale language models include the GPT® series (Generative Pre-Trained Transformer) developed by OPENAI, StableLM developed by Stability AI, Llama2 developed by Meta, and PaLm2® and LaMDA2® developed by Google. Note that other language models are also acceptable, not limited to large-scale language models. For example, BERT (Bidirectional Encoder Representations from Transformers) developed by Google may also be used. In this embodiment, the generative AI system 40 is used via an API (Application Programming Interface) provided by the server 20. Note that the generative AI system 40 is not limited to an external service; it may also be operated by the provider of the server 20 itself. Furthermore, multiple AI systems may be used interchangeably or in combination depending on the situation.

[0029] A prompt is primarily an inquiry statement input to the generating AI. The inquiry statement may include, for example, text, strings, images, videos, audio, etc. In this embodiment, it is assumed that the prompt includes image data such as a screenshot of the screen being evaluated. Alternatively, the state of the target of operation may be understood and interpreted without relying on image data such as screenshots, or in combination with image data such as screenshots, by obtaining the DOM (Document Object Model), CSS (Cascading Style Sheets), JavaScript (trademarked) code, etc., that constitute the screen being evaluated. Furthermore, the prompt may also include gestures. By inputting a prompt to the generating AI, the server 20 instructs the generating AI to process information such as thinking as a virtual user, making action decisions, and updating knowledge and emotions. The server 20 may also include information in the prompt to cause the generating AI to output a desired output result.

[0030] The prompt is automatically generated and entered, for example, by the internal processing of the server 20, based on test conditions entered from the terminal device 10. Alternatively, the prompt may be entered by uploading document files, images, videos, audio, etc.

[0031] The prompt may include information to specify the output format. Examples of such information include function definition information in a function call, JSON schema, YAML format, XML format, and other structured data format specifications. This information allows the output from the generating AI to be obtained in a predetermined structured format. For example, function definition information may include the function name, parameter names, parameter types, and parameter descriptions. In this embodiment, this is used to obtain the virtual user's actions and internal state (knowledge and emotion) as structured data. Based on this output format specification information, the generating AI generates output results in the specified format.

[0032] A prompt contains an instruction. A prompt may also contain reference data. Here, an instruction is data that indicates the information processing to be performed by the generating AI, such as a persona definition for a virtual user or the goal of the task. Reference data is data that the generating AI uses as a reference (learns) when performing information processing, such as information about the user interface being evaluated (images, HTML source, audio, the situation of the operator or the object being operated on, or log data, etc.). The instruction and reference data may be included in the prompt in any form. For example, the instruction and reference data may be included in the prompt as text data entered in response to the processing of the server 20. Another example is that the instruction may be included in the prompt as text data entered in response to the processing of the server 20, and the reference data may be included in the prompt as a separate file (image file, etc.) from the text data. Furthermore, the instruction and reference data for the same instruction content do not necessarily have to be entered into the generating AI as a single prompt. For example, a prompt containing the instruction may be entered into the generating AI first, and then a prompt containing the reference data may be entered into the generating AI.

[0033] <3. Configuration of terminal equipment> Figure 2 is a block diagram showing an example of the hardware configuration of the terminal device 10 shown in Figure 1. As shown in Figure 2, the terminal device 10 comprises a control unit 101, a storage unit 102, a communication unit 103, an input unit 104, and an output unit 105. Note that the terminal device 10 may also include components not essential to the invention of this embodiment, such as a camera 106, a position sensor 107, an acceleration sensor 108, and an audio processing unit (not shown). Each block included in the terminal device 10 is electrically connected, for example, by a bus.

[0034] The control unit 101 executes various processes by running various programs stored in the memory unit 102. The control unit 101 is, for example, a processor such as a CPU. A processor is hardware for executing instruction sets written in a program. A processor consists of an arithmetic unit, registers, peripheral circuits, etc. By operating according to the program, the control unit 101 performs the functions of an operation reception unit 131, a transmission / reception unit 132, and a presentation control unit 133.

[0035] The storage unit 102 includes a main memory and an auxiliary memory. The storage unit 102 stores various programs and various information. For example, the storage unit 102 stores an application program 120. The application program 120 includes, for example, a programming language that runs on a web browser application (not shown) stored in the storage unit 102.

[0036] The communication unit 103 performs processing such as modulation and demodulation processing for the terminal device 10 to communicate with an external device (for example, a server 20). The communication unit 103 performs transmission processing on the signal generated by the control unit 101 and transmits it to the external device. The communication unit 103 performs reception processing on the signal received from the external device and outputs it to the control unit 101.

[0037] The input unit 104 receives instructions or information input from the user. The input unit 104 may be implemented by, for example, a touch-sensitive device that inputs instructions, etc. by touching an operating surface. If the terminal device 10 is a PC, the input unit 104 may be implemented by a reader, keyboard, mouse, etc. The input unit 104 converts the instructions, etc. input by the user into electrical signals and outputs them to the control unit 101. The input unit 104 may also include, for example, a receiving port that receives electrical signals input from an external input device. The input unit 104 may also include a microphone that receives audio input.

[0038] The output unit 105 presents information to the user. The output unit 105 is implemented, for example, by a display. The display displays various information according to the control of the control unit 101. The display is implemented, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display. The output unit 105 may also include an output port that outputs electrical signals to an external output device. The output unit 105 may also include a speaker that outputs sound. In other words, presentation includes display on the display unit and output to a speaker or other output device. Information may also be provided to the user via email, SMS (Short Message Service), or push notification. The output unit 105 and the input unit 104 or other blocks do not necessarily have to be implemented in a single device; multiple physical devices (multiple PCs, or a PC and a smartphone, etc.) may work together to implement the functions.

[0039] Camera 106 is an imaging device that captures images using visible light. In other words, camera 106 is a device that receives visible light using a photodetector and outputs image data as an imaging signal. In this embodiment, camera 106 may be used, for example, to photograph paper documents (e.g., handwritten paper prototypes) or whiteboard sketches registered by the user as evaluation targets, and to include the image data in an evaluation request sent to server 20. Furthermore, in applications to hardware products described later, the camera is not limited to an imaging device that captures images using visible light. For example, it may be a depth camera such as a ToF sensor for acquiring a 3D structure, or it may perform 3D reconstruction processing based on a visible light image, or two or more cameras may be used in combination.

[0040] The position sensor 107 is a sensor that detects the position of the terminal device 10, and is generally a GNSS device, such as a GPS module. A GPS module is a receiving device used in a satellite positioning system. In a satellite positioning system, signals are received from at least three or four satellites, and based on the received signals, the current position of the terminal device 10, which is equipped with a GPS module, is detected as coordinate values. The position sensor 107 may also detect the current position of the terminal device 10 from the position of a wireless base station to which the terminal device 10 is connected via the communication unit 103.

[0041] The acceleration sensor 108 is a sensor that detects the acceleration applied to the terminal device 10. Preferably, the acceleration sensor 108 has the function of detecting the tilt around each axis (X axis, Y axis, Z axis) of a three-dimensional coordinate system with the position of the terminal device 10 as the origin. An acceleration sensor 108 having such a function can detect the orientation of the terminal device 10, that is, its direction with respect to the X, Y, and Z axes, by detecting the gravitational acceleration of the Earth's gravity.

[0042] Figure 3 is a block diagram showing the functional units realized by the control unit 101. The control unit 101 comprises an operation reception unit 131, a transmission / reception unit 132, and a presentation control unit 133 as functional units. Specifically, the control unit 101 realizes each functional unit by reading the application program 120 stored in the storage unit 102 and executing the instructions contained in the application program 120.

[0043] The operation reception unit 131 processes instructions or information input from the input unit 104. In this embodiment, it specifically receives input of test conditions (persona, scenario, etc.), instructions to execute evaluations, or input of questions in the "AI interview" function described later. The transmission / reception unit 132 processes data for the terminal device 10 to send and receive data with an external device according to a communication protocol. Specifically, the transmission / reception unit 132 transmits information such as test conditions input by the user to the server 20. The transmission / reception unit 132 receives information such as evaluation results transmitted from the server 20. The presentation control unit 133 controls the output unit 105 to present various information to the user, such as the test condition setting screen and the generated evaluation report (timeline, analysis results, etc.).

[0044] <4. Server Configuration> Figure 4 is a block diagram showing an example configuration of the server 20 shown in Figure 1. As shown in Figure 4, the server 20 comprises a control unit 201, a storage unit 202, a communication unit 203, and an input / output IF 204. The control unit 201 performs various processes by executing various programs stored in the storage unit 202. The control unit 201 is, for example, a processor such as a CPU.

[0045] The storage unit 202 includes a main memory and an auxiliary memory. The storage unit 202 stores various programs and various information. The storage unit 202 stores various databases. As various databases, the storage unit 202 stores, for example, a test management table 2021, a persona table 2022, a log table 2023, and an application program 2024 for executing the evaluation service according to this embodiment.

[0046] The Test Management Table 2021 is a database for managing test projects and their execution status. The Persona Table 2022 is a database for managing the attributes of virtual users. The Log Table 2023 is a database for storing the operation history of autonomous agents and the history of their internal state (knowledge and emotion) at that time.

[0047] Application Program 2024 is application software for executing and managing this evaluation service. Application Program 2024 includes, for example, a programming language such as JavaScript® that runs on a web browser application stored on terminal device 10.

[0048] The communication unit 203 performs processing such as modulation and demodulation processing for the server 20 to communicate with external devices (for example, terminal device 10, web server 30, generation AI system 40).

[0049] Figure 5 is a block diagram showing the functional units implemented by the control unit 201. The control unit 201 comprises a receiving control unit 2011, a transmitting control unit 2012, a presentation control unit 2013, and a service processing unit 2014 as functional units. Specifically, the control unit 201 implements each functional unit by reading a program (including an application program 2024) stored in the storage unit 202 and executing instructions contained in the program.

[0050] The receiving control unit 2011 controls the process by which the server 20 receives signals from external devices (such as the terminal device 10) according to a communication protocol. For example, the receiving control unit 2011 receives a request signal from the terminal device 10 for registration of test conditions or an instruction to execute an evaluation. The transmitting control unit 2012 controls the process by which the server 20 transmits signals to external devices (such as the terminal device 10 or the generation AI system 40) according to a communication protocol. The presentation control unit 2013 controls the process of presenting various information to the user. The presentation control unit 2013 causes the output unit 105 of the terminal device 10 to present the information via the network 80. Specifically, the presentation control unit 2013 transmits evaluation report screen data or information such as a timeline displayed by the output unit 105 to the terminal device 10 via the network 80. When the terminal device 10 receives the information transmitted by the presentation control unit 2013, it outputs it to the output unit 105.

[0051] The service processing unit 2014 executes a series of processes related to operability evaluation based on the request received by the receiving control unit 2011. In this embodiment, the service processing unit 2014 plays the role of generating and executing an autonomous agent (sandbox). Specifically, the service processing unit 2014 works in cooperation with the generation AI system 40 to activate an autonomous agent based on test conditions. This autonomous agent accesses the evaluation target (website, application, or sensor information of hardware equipment, etc.) via a browser operation interface (the operation execution environment, such as browser functions, may be embedded in the generation AI system 40 itself, or it may be an external tool provided by the service processing unit 2014) to acquire (perceive) information about the user interface and recognize operation elements based on that information. The autonomous agent then generates prompts (thought processes) that match the current situation and queries the generation AI system 40, autonomously making decisions on actions as a virtual user and updating its internal state.

[0052] The service processing unit 2014 generates a prompt that includes a predetermined instruction statement that instructs the large-scale language model to behave as a virtual user, and data such as information about the user interface.

[0053] The prescribed instruction text is text that describes the virtual user's persona and, if set, behavioral guidelines (goals, tasks, or abstract behavioral objectives, etc.), and provides instructions to encourage autonomous action (instructions for the AI ​​to decide on the next action). In this disclosure, "behavioral guidelines" is a broad concept that includes not only clear goals or tasks that the virtual user should achieve, but also conditions, constraints, abstract behavioral objectives (e.g., "play freely for 10 minutes," "buy as cheaply as possible"), or combinations thereof, that influence the virtual user's choices or decisions regarding their actions, and is composed of optional items in the test conditions. The prescribed instruction text is generated based on the information stored in the persona table 2022 of the memory unit 202. The prescribed instructions may include a clear goal, such as, "You are Hanako Sato, a 38-year-old housewife. You have intermediate IT literacy and a slightly impatient personality. Based on the following user interface information, please select the next action you should take to achieve the goal of purchasing a water bottle for your child." Alternatively, they may include an abstract action guideline without a specific goal, such as, "Please play this game app freely for 10 minutes," or they may include an action guideline that includes a condition, such as, "Continue operating until an error occurs." Furthermore, by setting a clear goal, such as executing the steps for a specific function correctly to the end, it is possible to aim for functional verification similar to that of conventional test scripts. In addition, the prescribed instructions may be defined in a form that embeds dynamic variables to change the action according to the current internal state, such as, "Currently, you are frustrated with the usability of the site (stress level: high). Please decide on the next action based on this emotional state."

[0054] Furthermore, the prescribed instructions may include "interruption events due to external factors" that may occur during test execution. For example, a scenario such as "a delivery arrives 15 minutes after the start of the operation, interrupting the operation for 5 minutes" may be included. In addition, when this interruption event occurs, the service processing unit 2014 may perform a process to erase (forget) some of the "short-term memory (product information viewed immediately before)" in the virtual user's internal state. This makes it possible to detect usability problems (such as insufficient reminder functions) caused by human cognitive limitations, such as "not knowing where the operation left off" when resuming the operation.

[0055] The service processing unit 2014 generates a predetermined instruction statement based on the test conditions received by the receiving control unit 2011 and the current internal state managed by the log table 2023, etc. Then, the service processing unit 2014 generates a prompt that includes the generated predetermined instruction statement and the acquired information about the user interface to be evaluated. The configuration of the prompt can be arbitrarily set according to the specifications of the generation AI system 40 used.

[0056] Figure 6 shows an example of the configuration of a prompt generated by the service processing unit 2014. As shown in Figure 6, prompt P is text data and image data, and includes data for each part exemplified by instruction P1, reference data P2, output format P4, and internal state information P5. The order in which instruction P1, reference data P2, output format P4, and internal state information P5 are listed is not particularly limited. Prompt P does not necessarily have to contain internal state information P5, and may contain information other than instruction P1, reference data P2, output format P4, and internal state information P5.

[0057] (Instruction P1) Instruction P1 includes a predetermined instruction statement that specifies the role (persona) that the large-scale language model will take when behaving as a virtual user. The instruction includes a sentence that specifies the position of the large-scale language model in determining thoughts and actions, such as, "You are Hanako Sato, a 38-year-old housewife." The service processing unit 2014 generates a prompt by combining this instruction with the following reference data, etc.

[0058] (Reference data P2) The reference data includes information about the user interface being evaluated acquired by the service processing unit 2014 (screenshots, HTML data, sensor data, information about the structure or state of the operating elements, the situation in which the operator or the target of the operation is placed, etc.), and text data of the test scenario (goal, conditions). In addition, the service processing unit 2014 may also acquire attribute information such as the IT literacy or personality traits of the virtual user by referring to the persona table 2022 based on the configured persona, and include it in the reference data. The service processing unit 2014 combines the reference data P2 with instruction P1.

[0059] Furthermore, the virtual user may be given persona information or a knowledge base that includes "knowledge (mental models) about competitor sites and common UI patterns" as well as knowledge about the specific specifications and business logic of the site being evaluated. For example, the user might have knowledge such as "frequently uses major e-commerce site A and is familiar with its UI operation," or specific knowledge about the site being evaluated, such as "the shopping cart button is located in the upper right corner of the screen" and "shipping is free if the total purchase amount is 3,000 yen or more." This allows the autonomous agent to detect and report discomfort and stress based on comparison, such as "it's different from what I'm used to, so it's difficult to use," when the site being evaluated deviates from common practices or the user's expected mental model.

[0060] (Output format P4) The output format includes the output format (schema) of the action decision, or information about the items to be output. The service processing unit 2014 generates prompts containing information about the output format of the response in order to obtain structured data that the system can analyze. For example, the service processing unit 2014 obtains instructions from the application program 2024 such as, "Please output the next operation (e.g., click target), the thought process that led to that operation, and the current sentiment score in JSON format," or "If you have acquired any knowledge, please include the updates to the knowledge base," and combines these instructions as the output format.

[0061] Furthermore, the output format may include not only sentiment values ​​but also cognitive indicators. For example, it may output parameters such as "Confusion," which indicates a state of not knowing what to do; "Efficiency Ratio," which shows the ratio of the actual operation procedure to the shortest procedure procedure calculated after the operation is completed; or "Memory Retention Rate," which shows how well the knowledge acquired during the operation process is retained. This makes it possible to quantitatively evaluate not only whether the task was completed or not, but also the level of cognitive load on the user and the ease of learning.

[0062] (Internal status information P5) Internal state information P5 is an area where the service processing unit 2014 stores contextual information such as past operation history in the session, the current virtual user's knowledge and emotions, or the thought process in past steps, obtained from the log table 2023 or a memory area it manages. This is used, for example, when the "current internal state" updated by the service processing unit 2014 is used for prompt generation, as will be described later. For example, if the virtual user learned the information that "shipping is an additional charge" on the previous screen, the service processing unit 2014 will include information indicating that knowledge (e.g., "I am aware that shipping is an additional charge") and the associated emotions (e.g., "I am worried about exceeding the budget") as internal state information P5 in prompt P. This makes it possible for the generating AI to generate consistent, human-like behavior and emotional changes, taking into account not only the information from instruction P1 and reference data P2, but also the context contained in internal state information P5.

[0063] The service processing unit 2014 functions as the execution environment for the autonomous agent and inputs prompts generated during the agent's thought process to the generation AI system 40. Here, the generation AI system 40 autonomously determines the next action based on the input user interface information, action guidelines, and current internal state (knowledge and emotion), and generates and outputs an operation command (for example, a function call to execute the API of a browser automation tool, or an executable operation script) to realize that action. The service processing unit 2014 immediately executes this generated operation command in an execution environment such as a browser (which may be a headless browser on the server 20, or a sandbox environment inside the generation AI system 40) (or receives the result of the generation AI system 40's direct execution). This enables human-like behavior in which the autonomous agent observes the target, makes judgments, and directly performs operations, rather than the server translating instructions from natural language step by step as in conventional programs. The result of the operation (such as the screen or device state change after the transition) is fed back to the autonomous agent as input information for the next step.

[0064] <5. Data Structure> In this embodiment, the main data structures managed by the storage unit 202 of the server 20 will be described with reference to Figures 7-9. Note that the data structures described are examples and do not exclude data not described.

[0065] Figures 7-9 show an example of the structure of a database stored in the memory unit 202. The memory unit 202 stores, for example, a test management table 2021, a persona table 2022, and a log table 2023. The test management table 2021 is a database for managing test projects and their execution status. The persona table 2022 is a database for managing virtual user definition information. The log table 2023 is a database for managing operation history during test execution, etc.

[0066] The Test Management Table 2021 is a table that has columns such as "Test ID" as the key, "Evaluation Target Information," and "Status." The "Test ID" field stores an identifier to uniquely identify each test project. The "Evaluation Target Information" field stores the URL of the website being evaluated, or the file path of a prototype image, etc. The "Status" field stores information indicating whether the test is "Waiting to Execute," "Executing," "Completed," or "Error." The Test Management Table 2021 is generated by the Service Processing Unit 2014, for example, when an evaluator creates a new project via the terminal device 10. Note that the types of statuses used in the Test Management Table 2021 are not limited to these.

[0067] The persona table 2022 shown in Figure 8 is a table that has columns for attribute information (age, IT literacy, personality traits, etc.) and context information of a virtual user, with the persona ID as the key. The "Persona ID" item stores an identifier to uniquely identify the virtual user used for testing. The "Attribute Information" item stores static characteristics of the virtual user, such as their age group, occupation, or proficiency with digital devices (IT literacy). The "Context Information" item stores the situation settings at the time of test execution, such as "in a hurry" or "first time visiting the site." This information is used by the service processing unit 2014 to create prompts that determine the behavior and decision criteria of the autonomous agent when it generates it.

[0068] The log table 2023 shown in Figure 9 is a table that has columns such as the executed operation, screen information, and internal state (knowledge, emotion) at that time, with the session ID and step ID as keys. During test execution by the autonomous agent, the log table 2023 is sequentially appended to and updated step by step by the service processing unit 2014.

[0069] The "Session ID" field stores an identifier that uniquely identifies a single test execution unit (session) performed by a single virtual user based on a single scenario. The "Step ID" field stores information such as a sequential number indicating the order of operations within that session. The "Operation Log" field stores the specific action history of the autonomous agent, such as elements clicked, scroll amounts, or entered text. The "User Interface Information" field stores data related to the user interface viewed by the virtual user at each step (screenshot images, HTML source code, DOM tree information, sensor values, information about the structure or state of the operation elements, the situation of the operator or the object being operated on, etc.). This information is used in the results report screen described later to reproduce and present to the user the screen situation at the time the operation was performed. The item "Internal State" stores a snapshot of at least one of the following from the virtual user's "knowledge" (e.g., learned site specifications such as "the shopping cart button is located in the upper right corner of the screen," or the site's business logic such as "shipping is free for orders over 3000 yen"), "emotions" (stress level, satisfaction level, etc.), and the "thought process" leading to the decision to perform the operation, at the time the operation was performed. This information is used by the presentation control unit 2013 when generating a timeline display of the test results. The knowledge and emotion data included in this internal state may be expressed and stored in JSON (JavaScript Object Notation), XML (Extensible Markup Language), or other structured data formats.

[0070] <6. Operation> Figure 10 is a sequence diagram showing the processing flow between the terminal device 10, the server 20, and the autonomous agent when the operability evaluation process according to this embodiment is executed.

[0071] First, triggered by an operation of the terminal device 10 by the user (evaluator), a test execution request (display request) is sent from the terminal device 10 to the server 20. This request includes test conditions, including the configured persona and action guidelines (goals, tasks, etc.). In step S11, when the server 20's receiving control unit 2011 receives the request, it processes a new project in the test management table 2021. Note that sending the test execution request (display request) to the server 20 does not necessarily have to be an explicit operation of the terminal device 10 by the user (evaluator); it may also be triggered by periodic execution at regular intervals, scheduled execution to run late at night when the server load is low, or by pushing source code to the repository or deploying a new version of the application being evaluated.

[0072] Next, in step S12, the service processing unit 2014 of the server 20 generates (instantiates) an autonomous agent in cooperation with the generation AI system 40 based on the received test conditions. The service processing unit 2014 then inputs a prompt (including action guidelines and initial settings) to the generated autonomous agent, which serves as a trigger for starting the test.

[0073] In response, the autonomous agent autonomously repeats the processes from step S13 to step S15 until the set test conditions are met or the predetermined interruption conditions are satisfied.

[0074] First, in step S13 (perception), the autonomous agent accesses the object to be evaluated and obtains information about the current user interface (screen data, DOM tree, sensor values, information about the structure or state of the operating elements, the situation in which the operator or the object being operated is located, etc.).

[0075] Next, in step S14 (thinking), the autonomous agent decides on the next action based on the acquired information and its current internal state (knowledge, emotions, history of past thought processes, etc.). At this time, the autonomous agent updates its own internal state based on "operation-related information" acquired in relation to the previous operation or the current situation. Here, "operation-related information" in this embodiment is a broad concept that includes not only explicit responses from the evaluation target (such as whether or not there is a screen transition), but also waiting time from operation to response (load time), asynchronous state changes that occur regardless of user operation (such as the appearance of a pop-up), ease of finding operation elements (visibility), or the autonomous agent's own situation (such as interruption of operation due to an interruption, inability to concentrate because a baby is crying nearby, etc.) and thought logs (detection of hesitation or errors). Here, thought logs refer to dynamic reasoning data that occurs in each operation step, and the autonomous agent updates its own internal state (thought process) by acquiring these thought logs as operation-related information. In other words, temporary observation data called thought logs are converted into continuously accumulated information called "internal state" through an update process. Specifically, in step S14 (thinking), the reasoning content and hesitation process generated by the autonomous agent are stored and updated as a "thinking process" in the internal state. This is then included in the prompts (internal state information P5) for subsequent steps, thereby enabling consistent operational decisions based on past judgments.

[0076] The autonomous agent evaluates the "difficulty or burden" of the operation based on this operation-related information and changes emotional parameters (stress level, etc.). Specifically, "difficulty" refers to cognitive and visual effort such as not being able to find the target button or the label being unclear, while "burden" refers to stress factors such as slow page loading (time burden), too many clicks (physical burden), or frequent errors. For example, if the autonomous agent determines that "it took more than 5 seconds to find the target button (difficulty: high)" or "there was no response for 3 seconds after clicking (burden: high)," it will increase its internal state of "frustration (stress value)." Regarding the stress value to be updated, for example, if the time taken for page loading or screen refresh during an operation exceeds the expected response time set in advance for each persona (e.g., 3.0 seconds), the stress value will be increased according to the excess time (e.g., 3). Alternatively, if the time it takes for a virtual user to recognize and identify the desired operation element on the screen exceeds a cognitive threshold set for each persona (e.g., 5.0 seconds), a fixed value (e.g., 1) is added as stress due to the difficulty of the operation, but this is not the only way to do so. The stress value is adjusted according to the persona's personality (for example, increasing the coefficient for an "impatient personality"). Furthermore, a process may be added where the stress value naturally decreases after a certain period of time, such as when the user is waiting without performing any operations. The stress value can also be expressed as a percentage, such as 0%-100%, or as a real number (including negative values), and the method of expression is not limited. In addition, other parameters (e.g., expected value) may be changed in conjunction with the change in the stress value.

[0077] Then, if the updated internal state meets a predetermined condition (e.g., stress value exceeds a threshold), the autonomous agent changes the "execution method" of the next operation. Here, "changing the execution method" means changing the quality or policy of the action rather than continuing the operation as originally planned. Specifically, one of the following (or a combination thereof) behaviors will be selected: (1) Operation interruption (withdrawal): The agent decides to "give up" without achieving the goal and forcibly terminates the session. (2) Change in precision (randomization): The agent stops carefully reading the explanation and switches to crude operations (reproducing human error), such as "randomly pressing buttons" or "agreeing without reading the terms and conditions." (3) Change of strategy (avoidance): The agent abandons the regular purchase route and takes other means (avoidance actions) such as "searching for a help page," "using site search," or "going back using the browser's back button." In this way, the autonomous agent takes unplanned actions (random operations or unexpected avoidance actions), functioning as an exploratory test that discovers potential bugs and errors that would be missed in normal script testing.

[0078] Then, in step S15 (execution), the autonomous agent executes the determined operation command (click, input, scroll, wait, etc.) on the target being evaluated.

[0079] When the loop processing is complete, the autonomous agent outputs the accumulated series of operation logs and internal state transition data to the server 20. In step S16, the presentation control unit 2013 of the server 20 generates evaluation report data (timeline, list of detected issues, etc.) based on the data received from the autonomous agent. Note that the generation process of this evaluation report data may be performed by the autonomous agent itself instead of the server 20. In that case, after the completion of the series of operations, the autonomous agent reflects on its own "experience" and autonomously creates report data including the issues discovered and an overall assessment, and outputs it to the server 20. In this embodiment, the report is generated after the completion of the series of operations, but it goes without saying that reports may be generated and output sequentially during the operation with the autonomous agent.

[0080] Then, the transmission control unit 2012 of the server 20 transmits the generated evaluation report data to the terminal device 10.

[0081] Finally, in step S17, the terminal device 10 displays the evaluation report data received from the server 20 on the screen of the output unit 105. This completes the series of operability evaluation processes.

[0082] <7. Screen example> Next, an example of a screen displayed on the terminal device 10 in the system according to this embodiment will be described.

[0083] Figure 11 shows an example of a project list screen for managing evaluation projects. This screen is displayed immediately after a user (evaluator) logs into the system. The screen displays a list of projects created for each website or application being evaluated, in a card-style display area (hereinafter referred to as "cards") 1102. Each project card displays the project name (e.g., "EC Site Renewal"), status (e.g., "In Progress"), and statistical information such as the number of test groups in the project, the number of sessions executed, and the last execution date. Users can create a new evaluation project by operating the "New Project" button 1101, and by selecting the area of ​​an existing project, they can transition to the details screen of that project (Figure 12).

[0084] Figure 12 shows an example of the test group list screen displayed on the terminal device 10. This screen is used to manage test scenarios (tasks) within a specific project and is displayed when a specific project is selected on the project list screen (Figure 11). The screen displays a list of test groups defined for each specific task (e.g., "purchase flow", "member registration flow", "product search") within the site or application being evaluated, in multiple card-style display areas (hereinafter referred to as "cards") 1202.

[0085] Each test group's card 1202 displays the group name (task name) and a "scenario summary." Furthermore, the number of problems detected as a result of the group's execution is displayed, categorized by severity (e.g., "Urgent," "High," "Medium," "Low"). Users can define and add new test groups (scenarios) by clicking the "+ New Test Group" button 1201 at the top of the screen. Users can also navigate to the detailed screen of an existing test group (Figure 13) by selecting an area on the card 1202 of that test group.

[0086] Figure 13 shows an example of the test group details screen displayed on terminal device 10. This screen is used to check the evaluation results for a specific test group and is displayed when a specific group (e.g., "Purchase Flow (Members)") is selected on the test group list screen (Figure 12).

[0087] Area 1301 in the "List of Detected Issues" section at the top of the screen displays usability issues that were commonly detected across multiple sessions. Each issue row includes information such as severity (e.g., "High", "Medium", "Low"), issue name (e.g., "Cart button visibility", "Response delay", "Unresponsive (freeze)"), impact (e.g., "Operation time +23 seconds"), and specific recommended actions (e.g., "Change button placement and make the color more prominent", "Improve server response speed").

[0088] Area 1302 of the "Test Session List" at the bottom of the screen displays the history of individual test sessions performed by different personas (e.g., "Beginner User (Female, 20s)", "General User (Male, 30s)"). Each session row displays the persona name, as well as quantitative data such as the number of clicks, screen transitions, and duration. Users can navigate to a session details screen to view the details of an individual session by clicking the "Details" button in each row.

[0089] Figure 14 shows an example of a session details screen displayed on terminal device 10. This screen is used for detailed analysis of individual test execution results and allows users to view the virtual user's behavior and internal state in a time series during a single session.

[0090] The session details display area 1401 at the top of the screen displays the overall evaluation results of the session. Specifically, it shows indicators such as the number of problems detected, the overall score, the time taken to complete the task, and satisfaction levels, allowing the user to grasp an overview of the session.

[0091] The operation history timeline display area 1402 located at the bottom of the screen is an area that displays the main functions of this system, showing the state of the screen for each step and the actions and thought processes of the virtual user at that point in time in chronological order. Each step in the timeline includes a thumbnail image of the operation screen that the virtual user is viewing at that time, as well as specific operation details such as "enter search keywords" and "click a button."

[0092] Furthermore, the display area 1402 shows internal state parameters that quantify the virtual user's emotions and psychological state at that moment (e.g., "learning (knowledge level)," "anxiety," "expectation," "impatience," "frustration," etc.), as well as AI thought process data that verbalizes the thoughts of the virtual user, explaining why they performed a particular action or what they felt. In addition, warnings for actions that take too long and detection icons for problems that are found are also displayed. This allows users to gain a deeper understanding of the causes of usability issues not only through a simple operation log, but also through the evolution of the AI's thoughts and emotions.

[0093] Figures 15 and 16, described below, are examples of screens related to additional functions (extension functions) of the system according to this embodiment. While these functions are not essential, including them enables more detailed verification and efficient test operation.

[0094] First, Figure 15 shows an example of a session restart settings screen displayed on the terminal device 10 for intervening in test execution. This screen is displayed, for example, when selecting a specific step on the timeline and changing the conditions to retest from that point. The current virtual user status display area 1501 on the screen displays the virtual user's status (knowledge level, emotion parameters, etc.) at that point (for example, step #12). The settings area 1502 also has tabs such as "Give advice" and "Change status".

[0095] In the example in Figure 15, the "Give Advice" tab is selected in the settings area 1502, allowing the evaluator to enter hints such as "The cart button is at the bottom of the page" in the hint input field, or to calm the emotional parameter by pressing the "Calm Down" button. On the other hand, although not shown in the illustration, if the "Change State" tab is selected, the evaluator can directly rewrite the values ​​of the virtual user's knowledge level (e.g., "20%") or emotional parameter (e.g., "Irritation: 35%"). This makes it possible to perform verifications that simulate the behavior of an experienced user without waiting for learning, such as "If the user had operational knowledge (knowledge: 80%), would they have stopped here?" Note that the units used to represent the internal state of the virtual user (knowledge, emotion, etc.) are not limited to these. For example, in this embodiment, the virtual user's knowledge level is expressed as a percentage, where 0% means they know nothing and 100% means they are well-versed in the subject of evaluation. However, knowledge could also be expressed in terms of data capacity, or knowledge items could be listed like slot filling, and the presence or absence of knowledge acquired during the operation process could be managed. Furthermore, it goes without saying that knowledge can be expressed using multiple parameters (e.g., site-specific knowledge, knowledge common to similar apps, knowledge common to mobile apps in general, etc.).

[0096] When the user presses the "Resume with these conditions" button, the service processing unit 2014 of server 20 reads the log data from that step and inserts the entered "advice text" or "modified internal state value (knowledge, emotion, etc.)" as additional information (context) into the prompt for deciding the next action. Then, it restarts (or updates the state of) the autonomous agent using this modified prompt and resumes the operation from the specified point in time, or executes the operation as a new branch different from the original operation log. In this case, server 20 may manage the operation log resulting from the re-execution as a "branch" different from the original timeline and display the operation history after the branch on the screen. This allows evaluators to simulate scenarios where the user consults help or receives support and verify subsequent changes in behavior.

[0097] Next, Figure 16 shows an example of a state export screen for saving the persona's state, which is displayed on the terminal device 10. On this screen, the internal state of the virtual user at a specific step can be summarized and saved with a unique persona ID. The internal information display area 1601 on the screen displays information such as the virtual user's knowledge level (e.g., "20% (initial)"), emotional state (e.g., "anxiety: 35%)", and past experiences (e.g., "error encountered") at the point in time to be saved (e.g., step #24). In addition, the export setting area 1602 allows the user to enter an arbitrary persona ID (e.g., "persona_experienced_A01") and a memo.

[0098] When the user clicks the "Export" button, the service processing unit 2014 serializes (converts into structured data) the internal state of the autonomous agent at that point in time (learned knowledge data, emotion parameters, short-term memory, etc.), links it to the entered persona ID, and saves it as a new record in the persona table 2022. By exporting this saved state, the evaluator can import it as a "knowledgeable user" in another test project (for example, evaluating a redesigned site) and start testing. This function can also be used for regression testing, which verifies that there is no functional degradation (regression) by saving and reusing specific scenarios and internal state (knowledge, emotion, etc.) combinations that were successful in past tests and automatically re-executing them on the redesigned system.

[0099] <8.Summary> As described above, according to this embodiment, an autonomous agent generated based on the set persona and goals can acquire information about the user interface to be evaluated, recognize the operation elements, and autonomously decide and execute operations. This automates a series of processes that would normally be performed by a human looking at the screen and operating it, significantly reducing the cost and time required for recruiting subjects and scheduling, while achieving realistic and high-quality evaluation results that are close to those of actual human testing.

[0100] Furthermore, by setting test conditions that include detailed persona information such as age and IT literacy, it is possible to reproduce behaviors that are not uniform but tailored to diverse user attributes. In addition, instead of following predetermined procedures, the user (evaluator) recognizes the operational elements contained in the information about the user interface in the moment and decides on the next action. This allows the user to discover dynamic usability issues that cannot be found with static verification tools and to devise more effective improvement measures.

[0101] <Variation> Next, we will describe some modified examples. The following will primarily focus on the differences from the embodiments described above.

[0102] (Prototype evaluation using image data) In the above embodiment, the evaluation target was a website (URL specified) running on the web server 30. However, in the modified example, the evaluation target may be website design images (still images), image data such as handwritten sketches, or data created with a prototyping tool such as Figma®. In this case, the service processing unit 2014 performs OCR (optical character recognition) or object recognition processing on the input image data to identify UI elements such as buttons and input forms. Then, it generates a virtual HTML structure (equivalent to a DOM tree) based on the identified elements and inputs this into the generation AI as prompt reference data, thereby realizing a pseudo-operability evaluation at the prototype stage before coding. If information other than images can be used in combination, that information may also be used in combination.

[0103] (AI interview function) Furthermore, this system may include an "AI interview function" that allows evaluators to ask questions to a virtual user (autonomous agent) in a conversational format after the test is completed or during the test execution. This function allows the virtual user to express, in their own words, the underlying psychology of "why they performed that action" and "how they felt at that time," which cannot be gleaned from simply reviewing logs.

[0104] Specifically, when the input unit 104 of the terminal device 10 receives a natural language question from the evaluator, such as "Why did it get stuck on the payment screen for a while?", the service processing unit 2014 of the server 20 performs inference processing to generate an answer to that question. In doing so, the service processing unit 2014 refers to the "attribute information (personality, IT literacy, etc.)" stored in the persona table 2022 and the "operation log" and "history of internal state (knowledge, emotion)" stored chronologically in the log table 2023 as the basis for generating the answer.

[0105] The service processing unit 2014 inputs a prompt to the generating AI system 40 that includes the following elements: (1) Definition of role: "You are Hanako Sato, a 38-year-old housewife. You have a slightly impatient personality." (2) Recollection of the situation: "You just noticed (knowledge) that the shipping fee was not displayed on the payment screen, and you stopped the operation due to distrust (emotion: anxiety)." (3) Instructions for answering: "Based on your feelings at that time, please answer the user's question, 'Why did you stop?' in your own words."

[0106] In this way, by providing the "internal state (emotional parameters and acquired knowledge)" at a specific point in the past as context for the prompt, the generating AI can output subjective answers that maintain the persona's personality, such as, "I couldn't find where the shipping cost was written, and I suddenly became anxious," rather than just a systemic explanation. This allows evaluators to intuitively understand usability issues that cannot be fully grasped by numerical data alone, as the user's "authentic voice."

[0107] (Applications to hardware products) Furthermore, while the evaluation target in the above embodiment was software such as a website or application displayed on a screen, it is not limited to this. The evaluation target may also be hardware products with a physical casing, such as a robot, home appliance (smart appliance), or automobile. In addition, System 1 may be equipped with hardware (actuators) such as a robot arm for performing physical operations on these evaluation targets, or the evaluation may be performed using a virtual operation target in a VR (Virtual Reality) space.

[0108] In this case, the autonomous agent in this disclosure functions as a "virtual user" that operates these hardware products. Here, an autonomous agent refers to an entity that utilizes generative AI (large-scale language model) and behaves autonomously with specific attributes (persona) and psychological state (context). This autonomous agent may not only execute programmed procedures, but may also have internal states that, for example, learn information obtained through operation and accumulate "knowledge," or dynamically change "emotions (stress level, etc.)" in response to the smoothness of operation and the product's response.

[0109] When evaluating a hardware product, the service processing unit 2014 acquires sensing data (camera images, microphone audio, or various sensor values, etc.) indicating the status of the hardware product, instead of screen information. The autonomous agent then determines and executes physical operations on the hardware product based on the acquired status information and its own internal state. Specifically, the service processing unit 2014 controls the robot arm based on the determined operation to press physical buttons on the target of evaluation, tap a touch panel, or issue voice commands to the target of evaluation.

[0110] This allows for simulations such as, "An autonomous agent, recognizing through sensors that a cleaning robot is stuck in a gap between furniture and unable to move, and therefore not functioning as expected, increases its internal stress level." In this way, the system can automate usability evaluations that take into account changes in the user's "knowledge" and "emotions" in products that involve physical interaction using robotic arms, not just software screen operations.

[0111] (Providing advanced evaluation instructions and reference information) Furthermore, in addition to the prompt generation process in the embodiments described above, processing to include more advanced instructions in the prompt may also be performed. For example, the service processing unit 2014 may include instructions such as "evaluate usability by comparing it with typical competitor sites" or "check whether it complies with accessibility guidelines such as WCAG (Web Content Accessibility Guidelines)" in the prompt. This processing is performed after the basic prompt has been generated. In this case, after receiving response data from the generating AI system 40, the service processing unit 2014 may extract the points raised in the response data (e.g., "insufficient contrast ratio," "missing labels") and search a guideline database (not shown) separately stored in the storage unit 202. Then, it adds a hyperlink to the relevant clause of the guideline obtained as a search result to the generating AI's evaluation comment and sends it to the terminal device 10. This allows the user to refer to the official standards that underlie the AI's evaluation results.

[0112] (Automatic scenario generation) Furthermore, while the above embodiment involved the user manually setting scenarios (goals, etc.) as test conditions, the system is not limited to this. When a URL to be evaluated is specified, the service processing unit 2014 may automatically crawl the website starting from that URL and perform processing to analyze the site structure and page content. Specifically, the service processing unit 2014 uses a generation AI or a dedicated crawler to follow links from the top page and construct a sitemap. Then, based on the elements included in the analyzed pages (for example, a "Add to Cart" button, a "Contact Us" form, etc.), it estimates the actions (goals) that the user is most likely to take and automatically generates test scenarios such as a "product purchase scenario" or an "inquiry submission scenario." The service processing unit 2014 presents the generated scenario candidates to the terminal device 10, accepts the user's (evaluator's) approval or selection, and then executes the evaluation process by the autonomous agent described above. This makes it possible to comprehensively and efficiently set test conditions even when the scale of the evaluation target is large or when it is unclear what kind of tests should be performed.

[0113] (Report output tailored to evaluator attributes) Furthermore, in addition to defining virtual users, Persona Table 2022 may also store output templates associated with the attributes of users (evaluators) who view the evaluation report (e.g., job title "Engineer", job title "Marketing"). In this case, when generating a prompt, Service Processing Unit 2014 first obtains the attribute information of the evaluator based on the evaluator's user ID. Next, it adjusts the instructions to the generating AI based on the obtained attribute information. For example, it may instruct an "Engineer" evaluator to generate a report that emphasizes technical details such as error logs and response speed, and a "Marketing" evaluator to generate a report that emphasizes psychological aspects such as the impact on brand image and changes in purchasing intent. This makes it possible to generate personalized reports that are more appropriate to the perspective or situation of the recipient of the evaluation results.

[0114] (Automatic generation and optimization of design proposals) Furthermore, the service processing unit 2014 may use a generating AI to automatically generate multiple high-fidelity design proposals (visual designs) with different button colors, shapes, sizes, or placements, based on low-fidelity image data such as wireframes or handwritten sketches input as evaluation targets. In this case, the service processing unit 2014 sets each of the generated design proposals as a new evaluation target and automatically executes the test scenario by the autonomous agent described above. The presentation control unit 2013 then compares and analyzes the virtual user's task completion rate, time taken, or emotional score (stress level, etc.) for each design proposal and outputs the design proposal estimated to have the highest usability, as well as information indicating directions for design improvement, to the terminal device 10. This makes it possible for the user to efficiently decide on the optimal design from among multiple design candidates based on objective evaluation data from the AI.

[0115] (Standalone configuration) Furthermore, although this embodiment describes a client-server system consisting of a terminal device 10 and a server 20, the modified system may instead be a standalone configuration in which all processing is completed on the terminal device 10. In this case, the memory unit 102 of the terminal device 10 stores an application program including a lightweight generative AI model (SLM: Small Language Models, etc.), and the control unit 101 executes a series of processes from prompt generation to virtual user action decision and evaluation report generation.

[0116] In the embodiments described above, the cases in which the units and means are implemented by a processor have been explained, but the invention is not limited thereto. The units and means may be any hardware known to perform the operation.

[0117] Furthermore, although the above embodiment describes an example in which the server 20 provides each function, the configuration is not limited to this. Some or all of the functions provided by the server 20 in this embodiment may be provided by the terminal device 10 instead of the server 20.

[0118] Although several embodiments of this disclosure have been described above, these embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. For example, configurations and processes in one embodiment may be combined with configurations and processes in another embodiment, or a modification of one embodiment may be applied to another embodiment. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0119] (Note) The details described in each of the above embodiments are noted below.

[0120] (Note 1) A program for operating a computer that includes a processor and memory, The program is directed to the processor, Steps include accepting the setting of a virtual user persona for the target of evaluation, and the test conditions for the virtual user, The steps include generating an autonomous agent that behaves as a virtual user based on test conditions, and The steps include having an autonomous agent acquire information about the user interface to be evaluated and recognize the operational elements included in the information about the user interface, The steps include having the autonomous agent determine and execute an operation on the recognized operational element, A program that executes something.

[0121] (Note 2) The processor also has, A step of updating the internal state of the virtual user, including at least one of the knowledge and emotions of the virtual user, based on operation-related information related to the execution of the operation, A step of reflecting the updated internal state in the criteria for determining subsequent operations, The program described in (Note 1) that executes the above.

[0122] (Note 3) The program described in (Appendix 2) further updates the thought process of the virtual user when deciding on an operation as part of the internal state during the step of updating the internal state.

[0123] (Note 4) In the step of updating the internal state, information regarding the specifications or structure of the subject being evaluated is learned and stored as knowledge. In the step of reflecting this in the judgment criteria, the operating procedure for achieving the objectives included in the test conditions is determined based on the information regarding the user interface and the accumulated knowledge. The program described in (Appendix 2).

[0124] (Note 5) In the step of updating the internal state, the internal state of the virtual user is changed based on the difficulty or burden of the operation on the evaluation target, or the content of the accumulated knowledge. In the step of reflecting the above judgment criteria, if the internal state satisfies predetermined conditions, the execution mode of the above operation is changed. The program described in any of (Appendix 2) to (Appendix 4).

[0125] (Note 6) The processor also has, The steps include: storing the operation history and the history of the internal state at each operation point in a memory unit in a chronological order; Based on the saved operation history and internal state history, the process generates and outputs timeline information showing the changes in the virtual user's behavior and emotions over time. A program described in any of (Appendix 2) to (Appendix 5) that executes the above.

[0126] (Note 7) The processor also has, A step to receive questions related to operations performed on the subject of evaluation, The steps include: generating and outputting an answer as an autonomous agent by referring to the saved operation history and internal state history in response to a question; The program described in (Appendix 6) that executes the above.

[0127] (Note 8) A program as described in any of (Appendix 1) to (Appendix 7), which, in the step of obtaining information about the user interface, accepts input information about the user interface of a prototype to be evaluated and recognizes the operation elements from the image.

[0128] (Note 9) The processor also has, A step to automatically generate multiple design proposals with different designs based on information about the user interface entered as the subject of evaluation, The process involves setting each of the generated design proposals as a new evaluation target and having an autonomous agent perform the evaluation, A program described in any of (Appendix 1) to (Appendix 8) that executes the above.

[0129] (Note 10) The processor also has, The steps include having an autonomous agent or crawler traverse the target of evaluation and analyze the screen transition structure or page content of the target of evaluation, Based on the analysis results, the system automatically generates a test scenario that includes goals that the virtual user should achieve, and sets it as a test condition. A program described in any of (Appendix 1) to (Appendix 9) that executes the above.

[0130] (Note 11) The aforementioned processor further includes, The steps include: specifying an arbitrary point in time in the execution history of the aforementioned operation, and receiving input of advice information for the autonomous agent at that point in time, or receiving an instruction to change the internal state; The steps include: restarting the operation from the specified time point or executing the operation as a new branch based on the input advice information or the changed internal state; The program described in (Appendix 2) that executes the above.

[0131] (Note 12) The aforementioned processor further includes, The steps include saving the internal state of the autonomous agent, which has been updated during the execution of the aforementioned operation, as learned persona information, When generating the autonomous agent under different evaluation targets or different test conditions, the steps include loading the saved trained persona information as the initial state, The program described in (Appendix 2) that executes the above.

[0132] (Note 13) An evaluation method performed by a computer comprising a processor and memory, A method by which a processor executes all steps in a program described in any of (Appendix 1) to (Appendix 12).

[0133] (Note 14) An information processing device comprising a processor and memory, An information processing device in which the processor executes all steps in a program described in any of (Appendix 1) to (Appendix 12).

[0134] (Note 15) A system comprising means for executing all steps in any of the programs described in (Appendix 1) to (Appendix 12). [Explanation of symbols]

[0135] 1... System 10…Terminal device 101... Control Unit 102...Storage section 103... Communications Department 104...Input section 105...Output section 20... Server 201... Control Unit 202...Storage section 203... Communications Department 204… Input / Output Interface 30…Web server 40…Generative AI system 80…Network

Claims

1. A program for operating a computer that includes a processor and memory, The program is provided to the processor: A step of accepting the setting of a virtual user persona for the target of evaluation, and the test conditions for said virtual user, A step of generating an autonomous agent that behaves as the virtual user based on the aforementioned test conditions, The steps include causing the autonomous agent to acquire information about the user interface to be evaluated and to recognize the operational elements included in the information about the user interface, The steps include having the autonomous agent determine and execute an operation to be performed on the recognized operation element, A step of updating the internal state of the virtual user, which includes at least one of the knowledge and feelings of the subject being evaluated, based on operation-related information related to the execution of the operation; A step of reflecting the updated internal state in the decision criteria in the step of determining the subsequent operation, The steps include storing the history of the operations and the history of the internal state at each operation point in a memory unit in a chronological order, Based on the saved operation history and the internal state history, the steps include generating and outputting timeline information that shows the changes in the virtual user's behavior and emotions in chronological order, A program that executes something.

2. The program according to claim 1, wherein in the step of updating the internal state, the program further updates the thought process of the virtual user when deciding on the operation as the internal state.

3. In the step of updating the internal state, information regarding the specifications or structure of the subject to evaluation is learned and stored as knowledge. In the step of reflecting the above judgment criteria, the operating procedure for achieving the objectives included in the test conditions is determined based on the information regarding the user interface and the accumulated knowledge. The program according to claim 1.

4. In the step of updating the internal state, the internal state of the virtual user is changed based on the difficulty or burden of the operation on the evaluation target, or the content of the accumulated knowledge. In the step of reflecting the above judgment criteria, if the internal state satisfies predetermined conditions, the execution mode of the above operation is changed. The program according to claim 1.

5. The aforementioned processor further includes, A step of receiving questions related to the operation on the subject of evaluation, The steps include: In response to the aforementioned question, generating and outputting an answer as the autonomous agent by referring to the saved operation history and internal state history; The program according to claim 1, which causes to execute.

6. The program according to claim 1, wherein, in the step of obtaining information about the user interface, it accepts input of information about the user interface of the prototype to be evaluated and recognizes the operation elements from the information about the user interface.

7. The aforementioned processor further includes, The steps include: automatically generating multiple design proposals with different designs based on the user interface information input as the subject of evaluation; The steps include setting each of the generated design proposals as a new evaluation target and having the autonomous agent perform the evaluation, The program according to claim 1, which causes to execute.

8. The aforementioned processor further includes, The steps include having the autonomous agent or crawler traverse the target of evaluation and analyze the screen transition structure or page content of the target of evaluation, Based on the results of the analysis, the system automatically generates a test scenario that includes guidelines for the actions the virtual user should follow, and sets it as the test condition. The program according to claim 1, which causes to execute.

9. The aforementioned processor further includes, The steps include: specifying an arbitrary point in the execution history of the aforementioned operation, and receiving input of advice information for the autonomous agent at that point in time, or receiving an instruction to change the internal state; The steps include: restarting the operation from the specified time point or executing the operation as a new branch based on the input advice information or the changed internal state; The program according to claim 1, which causes to execute.

10. The aforementioned processor further includes, The steps include saving the internal state of the autonomous agent, which has been updated during the execution of the aforementioned operation, as learned persona information, When generating the autonomous agent under different evaluation targets or different test conditions, the steps include loading the saved trained persona information as the initial state, The program according to claim 1, which causes to execute.

11. An evaluation method performed by a computer comprising a processor and memory, A method wherein the processor performs all steps in the program described in any one of claims 1 to 10.

12. An information processing device comprising a processor and memory, An information processing apparatus wherein the processor performs all steps in the program described in any one of claims 1 to 10.

13. A system comprising means for performing all steps in a program according to any one of claims 1 to 10.

Images