system

Abstract

We provide the system. [Solution] The present invention relates to a user test simulation system that automatically generates virtual agents with diverse user profiles using generative AI, simulates the operation procedures of a prototype, collects and analyzes detailed user interaction data, and automatically extracts problems and areas for improvement. The system comprises a prototype information acquisition unit, a virtual agent generation unit, a user interaction evaluation unit, and an analysis feedback generation unit, and is characterized by providing the development team with verification from diverse user perspectives and concrete feedback in a short time and at low cost, thereby simultaneously achieving increased efficiency and improved quality in the verification work of new product development.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method executed by at least one processor, the method including the steps of: receiving a user utterance; adding the user utterance to a prompt that includes an instruction sentence associated with a description related to the character of the chatbot; encoding the prompt; and inputting the encoded prompt into a language model to generate a chatbot utterance that responds to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In new product development, user testing and hypothesis verification of prototypes require considerable time and effort, and it is particularly difficult to comprehensively and efficiently evaluate the behavior and reactions of diverse user groups. Conventional methods require gathering actual users to conduct tests, which leads to increased human resources and costs, a longer verification cycle, and challenges such as subjective variability in user interaction analysis and problem identification, as well as limitations in the quality and quantity of feedback. This invention aims to address these issues by providing user test simulation and automated analysis functions using a virtual agent that utilizes generative AI, enabling prototype verification from diverse user perspectives in a short time and at low cost. It enables objective and comprehensive evaluation of user interactions, automatic extraction of problems and areas for improvement, and rapid provision of specific feedback to the development team, thereby simultaneously achieving increased efficiency and improved quality in verification work in new product development. [Means for solving the problem]

[0005] The present invention solves the problem by providing a user test simulation system comprising a prototype information acquisition unit, a virtual agent generation unit, a user interaction evaluation unit, and an analysis feedback generation unit. The prototype information acquisition unit acquires operation procedure information and user scenario information of a prototype to be verified in new product development. The virtual agent generation unit automatically generates multiple virtual agents based on different user profiles using a generative AI. The user interaction evaluation unit simulates user interaction by sequentially executing each operation procedure of the prototype using the virtual agents, recording actions such as success or failure for each procedure. The analysis feedback generation unit analyzes the operation results of all virtual agents using a generative AI, automatically extracts procedures with frequent failures and problems for each user group, and generates and outputs feedback information including points for improvement. This configuration enables prototype verification from diverse user perspectives in a shorter time and at a lower cost compared to conventional user testing, and provides a means to objectively and comprehensively evaluate user interaction, automatically extract problems and points for improvement, and quickly provide concrete feedback to the development team. [Brief explanation of the drawing]

[0006] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Modes for carrying out the invention]

[0007] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0008] First, let's explain the terminology used in the following explanation.

[0009] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0010] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0011] In the following embodiments, the storage with a reference sign is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include a flash memory (SSD (Solid State Drive)), a magnetic disk (for example, a hard disk), or a magnetic tape and the like.

[0012] In the following embodiments, the communication I / F (Interface) with a reference sign is an interface including a communication processor, an antenna, and the like. The communication I / F manages communication between a plurality of computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark) and the like.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, may be only B, or may be a combination of A and B. Further, in the present specification, when three or more matters are expressed in a connected manner by "and / or", the same concept as that for "A and / or B" is applied.

[0014] [First Embodiment]

[0015] FIG. 1 illustrates an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] As shown in FIG. 1, the data processing system 10 includes a data processing apparatus 12 and a smart device 14. An example of the data processing apparatus 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of the "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network), and the like.

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, camera 42, and communication I / F 44 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact of an indicator (for example, a pen or a finger, etc.), thereby receiving user input by the contact of the indicator. The microphone 38B detects a user's voice, thereby receiving voice-based user input. A control unit 46A in the processor 46 transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, a specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0022] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0024] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output processing program 60 executed on the RAM 48. The smart device 14 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0026] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device having the data generation model 58. The data processing device 12 may also be a server device or a terminal device owned by a user (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0027] (Example 1)

[0028] The flow of a specific process in Example 1 will now be described. Each part of the system described below is implemented by a data processing device 12 and a smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0029] (Embodiments for carrying out the present invention)

[0030] Embodiments of the present invention will be described in more specific and detail. The user test simulation system of the present invention is realized through the cooperative operation of a server and a terminal. The entire system comprises a prototype information acquisition unit, a virtual agent generation unit, a user interaction evaluation unit, and an analysis feedback generation unit, and these main processes are executed on the server. The terminal mainly plays a role in providing the user interface, inputting and editing prototype information, viewing simulation results, and receiving feedback reports.

[0031] First, the prototype information acquisition unit sends operation procedure information and user scenario information for the new product prototype, entered from the terminal, to the server, where it is centrally managed. For example, the terminal can input a wide variety of information, including screen transition diagrams, wireframes, prototype HTML files, operation videos, interaction specifications, user stories, use case diagrams, operation flow diagrams, detailed explanations of UI elements for each screen, validation specifications for input items, error message display conditions, responsive design behavior, and accessibility compliance status. This information is stored in a database on the server and becomes the basic data for subsequent simulation processing. For example, in the case of an e-commerce site prototype, detailed scenario data such as the transitions between each screen (product list screen, product details screen, shopping cart screen, payment screen, order completion screen, etc.) and the order completion screen, as well as the user's purpose and the order in which they operate the screens, are entered.

[0032] The virtual agent generation unit operates on a server using generative AI. This generative AI can utilize a combination of large-scale language models, reinforcement learning models, generative adversarial networks, and simulation agent generation algorithms. For example, it can generate virtual users with extremely diverse attributes, such as age, gender, IT literacy, purchase history, past experience with similar services, disability status, language settings, cultural background, digital divide status, visual, auditory, and motor function characteristics, learning style, risk aversion tendencies, emotional responses, stress tolerance, attention, memory, reaction speed, error tendencies, information-seeking behavior, and decision-making patterns. For instance, it can automatically generate dozens to thousands of virtual agents that accurately simulate real-world user demographics, such as a 60-year-old male smartphone novice, a 20-year-old female with extensive experience using e-commerce sites, a visually impaired user, a foreign language speaker, a housewife raising children, a student, a business person, a senior citizen, a person with a disability, a frequent user, or a first-time user. Generative AI automatically optimizes the attributes and behavioral patterns of virtual agents based on target user segment conditions entered from the terminal, persona information important to the development team, past user test data, usage logs of existing services, and publicly available user behavior databases. For example, in the case of testing new features for an e-commerce site, it can generate virtual agents with various purchasing behavioral patterns, such as frequent buyers, first-time users, coupon users, review posters, and users who have abandoned their shopping carts.

[0033] The user interaction evaluation unit operates on a server and sequentially executes the prototype's operation procedures using generated virtual agents. The generative AI simulates realistic user behavior and failure patterns according to the profile of each agent. For example, it can reproduce an extremely wide range of user behaviors, such as entering the wrong password on the login screen, getting confused when selecting a menu item, not being able to find the intended product in a product search, making a mistake when adding items to the cart, making input errors during the purchase process, abandoning the site because the meaning of the error message is not understood, becoming unable to operate due to a breakdown in responsive design, being unable to operate the site for screen reader users due to insufficient accessibility support, giving up midway due to overly strict validation of input forms, abandoning the site because the coupon input field cannot be found, and becoming anxious because the shipping cost is not known before confirming the order. The generative AI also estimates the user's psychological reactions, stress levels, satisfaction levels, points of abandonment, willingness to revisit, willingness to post reviews, and intention to switch to other services, and records in detail actions such as "success," "failure," "hesitation," "abandonment," "retry," "see help," and "inquiry generated" for each operation procedure. For example, even with the same procedure, it's possible to simulate differences that reflect the user profile, such as beginner agents taking longer to complete the operation while advanced agents proceed smoothly, disabled agents becoming unable to operate certain UI elements, and foreign language-speaking agents becoming confused by unnatural translations. Furthermore, generative AI can automatically generate and record detailed user interaction data, including eye movements, mouse movements, touch operation trajectories, input speed, accidental tap rates, scroll amounts, page dwell time, click heatmaps, error locations, and the number of times users transition to help pages.

[0034] The analysis and feedback generation unit operates on the server using a generative AI to aggregate and analyze the operation results of all virtual agents. The generative AI automatically calculates the number of failures, dropout rate, average operation time, frequency of confusion, number of retries, help referral rate, inquiry rate, error location, number of page transitions, operation completion rate, satisfaction score, etc. for each operation procedure, and automatically extracts problems such as frequent failures in specific procedures or concentrated dropouts among specific user groups. For example, it can identify specific issues such as "the failure rate of elderly agents is significantly higher than other groups in the product search procedure," "the overall dropout rate is high in the purchase procedure," "beginner agents frequently get confused when selecting menus," "disabled agents are unable to operate due to UI elements that do not support accessibility," and "foreign language speaking agents are making mistakes due to translation errors." Furthermore, the generative AI automatically generates specific improvement suggestions for the identified problems, such as "strengthen keyword completion in the search function," "simplify the input form for the purchase procedure," "redesign the menu structure," "strengthen accessibility," "improve translation accuracy," "make error messages easier to understand," "strengthen the navigation to the help page," "fix responsive design flaws," "place the coupon input field in a prominent position," and "display shipping costs earlier," and outputs these as a feedback report for the development team. The feedback report can be viewed from the device and is provided in a visually and intuitively understandable format, such as graphs, heatmaps, operation videos, detailed behavior logs for each agent, screenshots of failures, priority of improvement suggestions, problem matrices for each user group, comparison of A / B test results, and measurement of the effects before and after improvements.

[0035] Furthermore, generative AI can automatically optimize the number of simulation runs and agent variations, enabling a dynamic verification cycle. For example, it can automatically run additional simulations for a specific procedure if failures are concentrated in that procedure, automatically generate and re-evaluate new user profiles, modify parts of the operation flow and re-simulate, simultaneously compare multiple prototype versions, and automatically search for the optimal UI pattern for each user segment. In addition, if the development team uploads a new prototype or revised version on the terminal side, the server automatically runs a re-simulation, enabling comparative analysis before and after improvements and A / B testing. For example, it is possible to simultaneously test UI patterns A and B with a virtual agent and quantitatively compare which one is superior in terms of usability and user satisfaction.

[0036] Furthermore, generative AI can continuously learn and evolve its virtual agent's behavioral model and problem identification algorithm by utilizing external data such as past simulation results, actual user test data, publicly available user behavior databases, social media and review site user reviews, search engine query logs, and customer support inquiry history. For example, it can automatically analyze UI trends trending on social media, frequently occurring user questions on search engines, and common customer support inquiries, and reflect these in the virtual agent's behavioral patterns and feedback generation logic.

[0037] Thus, in this embodiment of the present invention, a generative AI is utilized on a server in a multi-stage and multi-purpose manner, automating everything from the generation of virtual agents, simulation of user interactions, analysis of operation results, automatic generation of feedback, utilization of external data, and optimization of the dynamic verification cycle. This significantly streamlines user testing that previously relied on human resources and enables the rapid provision of comprehensive verification from diverse user perspectives and concrete improvement suggestions. For example, it can be applied to prototype verification in all fields, such as new function development for e-commerce sites, user interface verification for financial services, operability evaluation for medical devices, user experience analysis for educational apps, accessibility verification for government services, route search experience evaluation for transportation apps, smart control panel verification for home appliances, tutorial experience analysis for game apps, input flow verification for reservation systems, and operational efficiency evaluation for B2B business systems. This embodiment is extremely practical and scalable, greatly contributing to shortening development cycles and improving quality.

[0038] (System Configuration)

[0039] The system according to this embodiment includes a prototype information acquisition unit, a virtual agent generation unit, a user interaction evaluation unit, an analysis feedback generation unit, a user interface unit, a database management unit, an external data linkage unit, and a prompt generation unit. The prototype information acquisition unit has the function of sending an extremely wide variety of information to the server, such as operation procedure information, user scenario information, screen transition diagrams, wireframes, prototype HTML files, operation videos, interaction specifications, user stories, use case diagrams, operation flow diagrams, detailed explanations of UI elements for each screen, validation specifications for input items, error message display conditions, responsive design behavior, and accessibility compliance status, and managing it centrally on the server side. For example, in the case of an e-commerce site prototype, detailed scenario data such as the transitions for each screen, including the product list screen, product details screen, cart screen, payment screen, and order completion screen, as well as the purpose and order in which the user operates the screens, are input. Furthermore, the prototype information acquisition unit can also accept screen transition diagrams, wireframes, operation videos, and interaction specifications of prototypes uploaded from the terminal and store them as prototype information.

[0040] The virtual agent generation unit operates on a server using generative AI to generate virtual users with extremely diverse attributes, including age, gender, IT literacy, purchase history, past experience using similar services, presence or absence of disabilities, language settings, cultural background, presence or absence of the digital divide, characteristics of visual, auditory, and motor functions, learning style, risk aversion tendencies, emotional reactions, stress tolerance, attention, memory, reaction speed, tendency to make errors, information-seeking behavior, and decision-making patterns. For example, it can automatically generate dozens to thousands of virtual agents that accurately simulate real-world user groups, such as a male in his 60s who is a smartphone novice, a female in her 20s who is an experienced e-commerce site user, a visually impaired user, a foreign language speaker, a housewife raising children, a student, a business person, a senior citizen, a person with a disability, a frequent user, and a first-time user. The virtual agent generation unit automatically optimizes the attributes and behavioral patterns of virtual agents based on target user group conditions entered from the terminal, persona information that the development team values, past user test data, usage logs of existing services, and publicly available user behavior databases. For example, when testing new features on an e-commerce site, it's possible to generate virtual agents with various purchasing behavior patterns, such as frequent buyers, first-time users, coupon users, review posters, and users who have abandoned their shopping carts.

[0041] The User Interaction Evaluation Unit operates on a server and sequentially executes the prototype's operation procedures using generated virtual agents. The User Interaction Evaluation Unit simulates realistic user behavior and failure patterns according to the profile of each agent. For example, it can reproduce an extremely wide range of user behaviors, such as incorrect password entry on the login screen, confusion when selecting a menu item, inability to find the intended product in a product search, errors in adding items to the cart, input errors during the purchase process, abandonment due to not understanding the meaning of error messages, inability to operate due to responsive design breakdowns, inability of screen reader users due to insufficient accessibility, giving up midway due to overly strict validation of input forms, abandonment because the coupon input field cannot be found, and anxiety due to not knowing the shipping cost before confirming the order. The User Interaction Evaluation Unit also estimates the user's psychological reactions, stress levels, satisfaction levels, points of abandonment, willingness to revisit, willingness to post reviews, and intention to switch to other services, and records in detail actions such as "success," "failure," "confusion," "abandonment," "retry," "see help," and "inquiry generated" for each operation procedure. Furthermore, it can automatically generate and record detailed user interaction data, such as eye movement during operation, mouse movements, touch operation trajectories, input speed, accidental tap rate, scroll amount, page dwell time, click heatmap, error locations, and the number of times users transitioned to help pages.

[0042] The analysis feedback generation unit operates on a server using generative AI and aggregates and analyzes the operation results of all virtual agents. For each operation procedure, the analysis feedback generation unit automatically calculates the number of failures, dropout rate, average operation time, frequency of confusion, number of retries, help referral rate, inquiry rate, error location, number of page transitions, operation completion rate, satisfaction score, etc., and automatically extracts problems such as frequent failures in specific procedures or concentrated dropouts among specific user groups. For example, it can identify specific issues such as "the failure rate of elderly agents is significantly higher than other groups in the product search procedure," "the overall dropout rate is high in the purchase procedure," "beginner agents frequently get confused when selecting menus," "disabled agents are unable to operate due to UI elements that do not support accessibility," and "foreign language speaking agents are making mistakes due to translation errors." Furthermore, the analysis and feedback generation unit automatically generates specific improvement suggestions for the identified problems, such as "strengthening keyword completion in the search function," "simplifying the input form for the purchase procedure," "redesigning the menu structure," "strengthening accessibility," "improving translation accuracy," "making error messages easier to understand," "strengthening the navigation to the help page," "correcting responsive design flaws," "placing the coupon input field in a prominent position," and "clearly displaying shipping costs earlier," and outputs these as a feedback report for the development team. The feedback report can be viewed from the terminal and is provided in a visually and intuitively understandable format, such as graphs, heatmaps, operation videos, detailed behavior logs for each agent, screenshots of failures, priority of improvement suggestions, a problem matrix for each user group, a comparison of A / B test results, and measurement of the effects before and after improvements.

[0043] The user interface operates on the terminal and provides an intuitive interface that allows development teams and stakeholders to perform a variety of operations, such as inputting and editing prototype information, instructing simulation execution, viewing feedback reports, confirming improvement suggestions, setting simulation parameters, customizing agent profiles, searching past simulation results, comparing A / B tests, playing operation videos, viewing heatmaps, and adding comments to problem areas. For example, it includes functions to meet the diverse needs of the development environment, such as creating screen transition diagrams using drag-and-drop, registering user scenarios using form input, selecting target user groups using checkboxes, adjusting the number of simulations using a slider, specifying test periods using a calendar, outputting feedback reports as PDFs, and linking improvement suggestions to a task management system.

[0044] The database management unit operates on the server and centrally manages and stores a vast amount of data handled by the entire system, including prototype information, virtual agent profile information, user interaction data, simulation results, feedback reports, past user test data, external data integration results, and system configuration information. For example, it performs version control for each prototype, allowing comparison with past simulation results, and saves agent behavior logs chronologically, enabling detailed analysis and reproduction simulations later. Furthermore, the database management unit also includes functions to ensure security and reliability, such as access rights management, data encryption, backup, and log auditing.

[0045] The external data integration unit operates on the server and integrates with a variety of external data sources, including publicly available user behavior databases, social media review data, search engine query logs, customer support inquiry history, industry-standard UI guidelines, accessibility standards, publicly available information on competing services, open data, and API integration partners. This contributes to improving the virtual agent's behavior model and problem extraction algorithm, optimizing feedback generation logic, and enhancing simulation accuracy. For example, it can automatically collect trending UI trends on social media and reflect them in the virtual agent's behavior patterns, or analyze frequently occurring customer support inquiries to utilize them for problem extraction during simulations.

[0046] The prompt generation unit operates on the server and automatically generates optimal prompt text for the generative AI. Based on prototype information, target user group, simulation objectives, past problem trends, and external data integration results, the prompt generation unit creates instruction text that enables the generative AI to generate virtual agents with high accuracy, simulate user interactions, and automatically generate problem identification and improvement suggestions. For example, it automatically generates prompt text such as: "This prototype is a new payment flow for an e-commerce site, targeting elderly people who are new to smartphones and young people with extensive experience using e-commerce. Each agent should perform each step of login, product search, cart addition, purchase procedure, and order confirmation, and meticulously record actions such as confusion, failure, abandonment, errors, help consultation, and inquiries that occur along the way. In particular, focus on accessibility, clarity of error messages, ease of use of input forms, and clarity of shipping cost display, extract problems for each user group, and generate specific improvement suggestions." Furthermore, it can generate specific prompts such as, "This prototype is a control panel for a medical device, targeting the elderly and healthcare professionals. Each agent should perform the following steps: initial startup, setting changes, measurement start, error handling, data saving, and shutdown. They should meticulously record actions such as inability to operate, errors, confusion, stress, and inquiries. They should identify issues focusing on the clarity of the UI, the intuitiveness of the operation flow, and the ease of error handling, and generate improvement suggestions." In addition, it can automatically generate a variety of prompts such as, "This prototype is a tutorial screen for an educational app, targeting elementary school students and their parents. Each agent should perform the following steps: account creation, tutorial start, task selection, hint use, task submission, and grade confirmation. They should meticulously record actions such as confusion, abandonment, help consultation, errors, and decreased satisfaction that occur along the way. They should identify issues focusing on the clarity of the tutorial, the ease of task selection, and the effectiveness of the hint function, and generate improvement suggestions."

[0047] As described above, the system according to this embodiment, through the coordinated operation of its various components, enables the acquisition and management of prototype information, high-precision generation of virtual agents, multifaceted simulation of user interactions, detailed data recording and analysis, utilization of external data, automatic generation of optimal prompt statements, provision of an intuitive user interface, centralized management of vast amounts of data, security assurance, and flexible response to the diverse needs of the development site. This significantly streamlines user testing, which previously relied on human resources, and enables the rapid provision of comprehensive verification from diverse user perspectives and concrete improvement suggestions.

[0048] (Implementation Steps)

[0049] Step 1: Inputting and obtaining prototype information This step involves the development team and stakeholders using terminals to input detailed information about the new product prototype into the system. A wide variety of information is entered here, including screen transition diagrams, wireframes, HTML files, operation videos, interaction specifications, user stories, use case diagrams, operation flow diagrams, descriptions of UI elements for each screen, validation specifications for input fields, error message display conditions, responsive design behavior, and accessibility compliance status. For example, in the case of an e-commerce site prototype, detailed data is entered regarding the transitions between each screen, such as the product list screen, product details screen, shopping cart screen, payment screen, and order completion screen, as well as scenario data describing the user's purpose and the order in which they operate the screens. This information is stored in a database on the server and serves as the basis for subsequent processing.

[0050] Step 2: Prompt generation and AI utilization for virtual agent generation. Based on prototype information, target user base, simulation objectives, past problem trends, and external data integration results, the prompt generation unit automatically generates optimal prompt text for the generation AI. Specific examples of prompt text to be fed into the generation AI include: "This prototype is a new payment flow for an e-commerce site, targeting elderly people who are new to smartphones and young people with extensive e-commerce experience. Each agent should perform each step—login, product search, cart addition, purchase procedure, and order confirmation—and meticulously record any actions that occur along the way, such as hesitation, failure, abandonment, errors, help consultations, and inquiries. Pay particular attention to accessibility, clarity of error messages, ease of use of input forms, and clarity of shipping cost displays, extract issues for each user base, and generate specific improvement suggestions." Based on this prompt, the generating AI automatically creates dozens to thousands of virtual agents with diverse attributes, including age, gender, IT literacy, purchase history, disability status, language settings, cultural background, visual, auditory, and motor function characteristics, learning style, risk aversion tendencies, emotional responses, stress tolerance, attention, memory, reaction speed, tendency to make errors, information-seeking behavior, and decision-making patterns.

[0051] Step 3: Run the user interaction simulation. This step involves the generated virtual agent sequentially executing the prototype's operating procedures. The generated AI simulates realistic user behavior and failure patterns according to each agent's profile. For example, it reproduces a variety of user behaviors such as entering the wrong password on the login screen, getting confused when selecting a menu item, not being able to find the intended product in a product search, making a mistake when adding items to the cart, making input errors during the purchase process, abandoning the site because the error message is not understood, becoming unable to operate due to a broken responsive design, being unable to operate the site for screen reader users due to insufficient accessibility, giving up midway due to overly strict validation of input forms, abandoning the site because the coupon input field cannot be found, and becoming anxious because the shipping cost is not known before confirming the order. Furthermore, it estimates the user's psychological reactions, stress levels, satisfaction levels, points of abandonment, willingness to revisit, willingness to post reviews, and intention to switch to other services, and records in detail actions such as "success," "failure," "hesitation," "abandonment," "retry," "see help," and "inquiry generated" for each operating procedure.

[0052] Step 4: Detailed recording and aggregation of user interaction data The user interaction evaluation unit automatically generates and records detailed user interaction data, including eye movement, mouse movements, touch operation trajectories, input speed, accidental tap rate, scroll amount, page dwell time, click heatmap, error locations, and the number of times users transitioned to help pages. This allows for the chronological storage of behavioral logs for each agent, enabling detailed analysis and reproduction simulations later on.

[0053] Step 5: Analysis of simulation results and identification of issues (utilizing generation AI) The analysis and feedback generation unit uses a generation AI to aggregate and analyze the operation results of all virtual agents. Specific examples of prompt text to be fed into the generation AI include: "Based on the operation history of all agents, calculate the number of failures, dropout rate, average operation time, frequency of confusion, number of retries, help referral rate, inquiry rate, error location, page transition count, operation completion rate, satisfaction score, etc., for each operation procedure, and automatically extract problems such as frequent failures in specific procedures or concentrated dropouts among specific user groups. Furthermore, generate specific improvement suggestions for the extracted problems." For example, it can identify specific issues such as: "The failure rate of elderly agents in the product search procedure is significantly higher than other groups," "The overall dropout rate is high in the purchase procedure," "Beginner agents frequently get confused when selecting menus," "Disabled agents are unable to operate due to UI elements that are not accessible," and "Foreign language-speaking agents are making errors due to translation mistakes."

[0054] Step 6: Automatic generation and provision of feedback reports The analysis and feedback generation unit automatically generates feedback reports for the development team based on the extracted issues and improvement suggestions. The feedback reports include graphs, heatmaps, operation videos, detailed behavior logs for each agent, screenshots of failures, priority rankings of improvement suggestions, issue matrices for each user group, A / B test results comparisons, and measurement of the effects before and after improvements. These reports can be viewed intuitively from the terminal, allowing the development team to quickly grasp specific improvement measures and reflect them in the next development cycle.

[0055] Step 7: System optimization through external data integration and continuous learning The External Data Integration Department integrates with a diverse range of external data sources, including social media review data, search engine query logs, customer support inquiry history, industry-standard UI guidelines, accessibility standards, publicly available information from competitor services, open data, and API integration partners. This contributes to the advancement of virtual agent behavior models and problem identification algorithms, the optimization of feedback generation logic, and the improvement of simulation accuracy. Furthermore, the generating AI utilizes past simulation results and actual user test data to continuously learn and evolve, achieving more realistic and accurate user simulations and problem identification.

[0056] (Specific use cases)

[0057] For example, when applying the system of the present invention to prototype verification of a new payment flow for an e-commerce service, the development team first uses a terminal to input into the system screen transition diagrams for screens such as the product list screen, product details screen, cart screen, payment screen, and order completion screen, UI elements for each screen, validation specifications for input items, display conditions for error messages, responsive design behavior, accessibility compliance status, a detailed operation flow from when a user searches for a product, adds it to the cart, and completes the purchase procedure, as well as user scenarios assuming diverse user groups such as "elderly people," "young people," "people with disabilities," and "foreign language speakers." The prototype information acquisition unit centrally manages this information on the server, and the virtual agent generation unit uses generation AI to generate a large number of virtual agents that accurately simulate real user groups, such as "elderly people who are new to smartphones," "young people with extensive experience using e-commerce," "users with visual impairments," "foreign language speakers," "people who have abandoned their carts before," and "users who use coupons." The User Interaction Evaluation Unit has these virtual agents sequentially perform each step of the process: login, product search, adding to cart, checkout, and order confirmation. This simulates realistic user behavior and failure patterns, such as "entering the wrong password on the login screen," "not being able to find the intended product in the product search," "making a mistake when adding to the cart," "making an input error during the checkout process," "abandoning the process because the error message is not understood," "being unable to operate due to UI elements that are not accessible," and "abandoning the process because the shipping cost display is unclear." For each operation step, the unit meticulously records actions such as "success," "failure," "hesitation," "abandonment," "retry," "refer to help," and "inquiry." Furthermore, it automatically records detailed user interaction data, including eye movements, mouse movements, touch operation trajectories, input speed, accidental tap rate, scroll amount, page dwell time, click heatmap, error locations, and the number of times users transitioned to the help page.The analysis and feedback generation unit uses generation AI to aggregate and analyze the operation results of all virtual agents, automatically extracting specific issues such as "the failure rate of elderly agents in the product search procedure is significantly higher than other groups," "the overall dropout rate in the purchase procedure is high," "beginner agents frequently get confused when selecting menus," "disabled agents are unable to operate due to UI elements that are not accessible," and "foreign language speaking agents are making mistakes due to translation errors." Furthermore, for the extracted problems, it automatically generates specific improvement suggestions such as "strengthen keyword completion in the search function," "simplify the input form in the purchase procedure," "redesign the menu structure," "strengthen accessibility support," "improve translation accuracy," "make error messages easier to understand," "strengthen the link to the help page," "correct responsive design flaws," "place the coupon input field in a prominent position," and "clearly display shipping costs earlier," and outputs these as a feedback report for the development team. The feedback report includes graphs, heatmaps, operation videos, detailed behavior logs for each agent, screenshots of failure points, priority of improvement suggestions, a problem matrix for each user group, a comparison of A / B test results, and measurement of the effects before and after improvements. These reports can be viewed intuitively from the terminal, allowing the development team to quickly grasp specific improvement measures and incorporate them into the next development cycle.

[0058] Specific examples of prompt text to be fed into the generating AI include: "This prototype is a new payment flow for an e-commerce service, targeting a diverse user base including elderly people who are new to smartphones, young people with extensive e-commerce experience, visually impaired users, and foreign language speakers. Each agent should perform the steps of logging in, searching for products, adding to cart, completing the purchase, and confirming the order, and meticulously record any actions that occur along the way, such as confusion, failures, abandonment, errors, referencing help, and inquiries. Pay particular attention to accessibility, clarity of error messages, ease of use of input forms, and clarity of shipping cost displays, extract challenges for each user group, and generate specific improvement suggestions."

[0059] (Application Example 1) The flow of a specific process in Application Example 1 will be explained below. Each part of the system described below is implemented by a data processing device 12 and a smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0060] (Embodiments for carrying out the present invention)

[0061] Embodiments for carrying out the present invention will be described in more specific and detail regarding the configuration and operation of the virtual store experience evaluation system. This system comprises a store scenario information acquisition unit, a virtual agent generation unit, a user interaction simulation unit, an analysis feedback generation unit, a user interface unit, a database management unit, an external data linkage unit, and a prompt generation unit, and is realized through the cooperative operation of a server and a terminal.

[0062] First, the store scenario information acquisition unit acquires and manages the virtual store experience scenarios in an extremely multifaceted and detailed manner. For example, avatar operation (movement, jumping, viewpoint switching, gestures, clothing changes), product search (category selection, keyword input, voice search, image search, ranking viewing), product description viewing (multilingual switching, video playback, zoom display, review viewing, specification comparison), conversation with customer service AI (text chat, voice conversation, FAQ automated response, product recommendation, complaint handling), use of accessibility functions (screen zoom, color vision correction, text-to-speech, keyboard operation, subtitle display), payment (adding to cart, using coupons, using points, selecting multiple payment methods, identity verification), promotion display (banners, pop-ups, limited-time sales, lottery participation, coupon distribution), in-store movement (moving between floors, elevators, escalators, shortcut movement, guidance during crowded times), cart operation (adding multiple items simultaneously, deleting, changing quantities, checking inventory), return procedures (selecting return reason, selecting return method, refund procedures) The system allows users to input diverse operational flows and user scenarios unique to virtual stores from their terminals and manage them centrally on the server. These include: checking return history, posting reviews (star ratings, comment input, image attachment, anonymous posting, editing posts), participating in events (stamp rallies, quizzes, live streaming viewing, lottery entries, purchasing limited-edition items), virtual try-on (wearing on an avatar, size selection, color change, mirror display, comparison with other users), using voice guidance (navigation, product descriptions, event announcements, emergency announcements, accessibility assistance), real-time chat (conversations with other users, conversations with store staff, group chats, stamp sending, reporting function), in-store mini-games (point acquisition, ranking, cooperative play, reward acquisition, difficulty selection), actions during crowded times (queue formation, entry restrictions, congestion status display, priority entry, suggested routes to avoid congestion), and purchasing limited-edition items (lottery, first-come, first-served, reservation, purchase restrictions, notification function).Furthermore, it is possible to input detailed scenario data such as UI elements for each screen (buttons, links, images, videos, input fields, lists, tabs, sliders, checkboxes, radio buttons), validation specifications for input fields (required fields, character limits, format restrictions, real-time error display), conditions for displaying error messages (input errors, communication errors, out-of-stock items, payment failures, authentication errors), responsive design behavior (layout changes depending on screen size, portrait / landscape switching, zoom support), accessibility support status (screen reader support, keyboard operation, color vision correction, text-to-speech, subtitle display), and the purpose and order in which users operate the system.

[0063] The virtual agent generation unit automatically generates virtual agents with diverse user profiles using a generative AI. For example, it can include: beginners (first-time users, haven't read the operation guide, low IT literacy), foreign language speakers (non-native Japanese speakers, English speakers, Chinese speakers, dependent on translation accuracy), people with disabilities (visual impairment, hearing impairment, motor impairment, color vision deficiency, cognitive impairment), advanced users (frequent users, shortcut users, customization-oriented), children (elementary school students, junior high school students, accompanied by a guardian, age restrictions), seniors (65 years and older, slow operation speed, frequent errors), users with low IT literacy (smartphone beginners, no PC experience), users who frequently use virtual stores (repeat customers, VIP members, high point reward orientation), first-time users (account not registered, guide not read, unfamiliar with operation), users strongly inclined towards coupon use (discount-focused, coupon collection, multiple coupon use), and review-focused users. The system automatically generates dozens to thousands of virtual agents that accurately simulate real-world user demographics, including: type users (those who refer to other users' ratings and frequently post reviews), multi-device users (those who use smartphones, tablets, and PCs), voice guidance users (those with visual impairments, those who use the system while driving, and those who rely on voice recognition accuracy), AR function users (those who operate the system, recognize spatial information, and are sensitive to AR display errors), multilingual speakers (those who switch between languages ​​and compare translation accuracy), left-handed users (those who prioritize UI layout and usability), users with color vision deficiencies (those who are blind to red-green or blue-yellow, and those who prioritize contrast), impatient users (those who require shorter waiting times and immediate responses), long-stay users (those who visit multiple stores, participate in events, and collect points), and users who use the system simultaneously with multiple people (family use, group use, and simultaneous operation). Generative AI automatically sets an extremely wide range of attributes, including age, gender, language, disability status, IT literacy, purchase history, past virtual store usage experience, cultural background, stress tolerance, tendency to make mistakes, information-seeking behavior, decision-making patterns, accessibility usage tendencies, reaction speed, emotional responses, attention span, memory, preferred UI design, responsiveness to promotions, review posting tendencies, event participation frequency, limited-edition product purchase experience, return experience, chat usage frequency, and mini-game participation tendencies. For example, a foreign language-speaking agent is more likely to fail to understand product descriptions and customer service AI conversations when using a Japanese UI, while a disabled agent is strongly influenced by the presence or absence of accessibility features and the visibility and operability of the UI.Beginner agents tend to get confused with avatar operation and payment, while advanced agents are sensitive to minor UI flaws and response delays. Child agents have different behavioral patterns when accompanied by a guardian compared to when alone, and agents who stay for extended periods tend to visit multiple stores and participate in events and mini-games.

[0064] The user interaction simulation unit uses the generated virtual agent to sequentially execute each operation step of the store scenario. For example, avatars get stuck on walls, jumping is impossible, switching viewpoints is confusing, errors occur when changing outfits, incorrect category selection or keyword input in product searches, voice search recognition errors, similar products cannot be found in image searches, getting lost when viewing rankings, failing to switch languages ​​when viewing product descriptions, videos stopping midway, inability to zoom in, being misled by misinformation when viewing reviews, getting confused when comparing specifications, unnatural responses in conversations with customer service AI, irrelevant FAQ automated responses, product recommendations not matching needs, dissatisfaction with complaint handling, screen zoom not working with accessibility features, insufficient color correction, voice reading stopping midway, keyboard operation impossible, delayed subtitle display, failure to add items to cart during checkout, inability to use coupons, confusing point usage, confusion with multiple payment method selection, errors during identity verification, banners obstructing promotion displays, inability to close pop-ups, unclear methods for participating in limited-time sales, errors when participating in raffles, missed coupon distributions, difficulty navigating between floors in the store, elevators... Unable to operate the turntable, misoperation on escalators, unable to find shortcuts, insufficient guidance during crowded times, failure to add multiple items to the cart simultaneously, deletion or quantity changes not reflected, incorrect information in inventory check, incorrect reason selection during return procedures, unsure how to return items, error during refund procedures, return history not displayed, star ratings not reflected in reviews, exceeding character limits when entering comments, unable to attach images, unable to post anonymously, post edits not reflected, unable to progress through stamp rallies during events, unable to submit quiz answers, playback errors during live streaming, unsure if entry is complete in lottery applications, exceeding purchase limits when purchasing limited items, notification function not working, items not reflected on avatars in virtual try-on, size selection errors, unable to change colors, mirror display bugs, unable to compare with other users, navigation stopping midway when using voice guidance, product descriptions inaudible, delayed event announcements, emergency announcements not working, insufficient accessibility assistance, conversations with other users interrupted in real-time chat.The system replicates a variety of problems that real users might encounter, such as inability to communicate with staff, accidental sending of messages in group chats, stamps not being reflected, reporting functions not working, inability to earn points in in-store mini-games, rankings not updating, inability to play cooperatively, rewards not being reflected, inability to select difficulty levels, inability to form queues during peak hours, entry restrictions not functioning, delayed display of congestion status, priority entry not being reflected, insufficient suggestions for avoiding congestion, inability to participate in lotteries for limited-edition product purchases, errors with first-come, first-served systems, reservations not being reflected, purchase restrictions malfunctioning, and delayed notification functions. Each agent meticulously records actions such as "success," "failure," "hesitation," "abandonment," "retry," "refer to help," and "inquiry generated" for each step, and automatically generates and records detailed user interaction data such as eye movement during operation, mouse movements, touch operation trajectories, input speed, accidental tap rate, scroll amount, page dwell time, click heatmap, error locations, and number of transitions to help pages.

[0065] The analysis and feedback generation unit uses a generational AI to aggregate and analyze the operation results of all virtual agents. For example, it automatically calculates the number of failures, dropout rate, average operation time, frequency of confusion, number of retries, help referral rate, inquiry rate, error location, page transition count, operation completion rate, satisfaction score, etc., for each operation procedure, and automatically extracts issues such as when failures or dropouts are frequent in a particular procedure, or when problems are concentrated in a particular user group. For example, it extracts issues as specific numerical data, such as an 80% failure rate for disabled agents when operating avatars, a 60% failure rate for foreign language-speaking agents when viewing product descriptions, a high dropout rate for novice agents during checkout, a high stress score for advanced agents when displaying promotions, frequent errors by child agents when adding items to the cart, frequent confusion by elderly agents during return procedures, inability to operate agents with color vision deficiency during virtual try-on, many failures by hearing-impaired agents with voice guidance, frequent mis-sentences by foreign language-speaking agents in real-time chat, and a high overall error rate when purchasing limited-edition products. Furthermore, for each identified problem, the system automatically generates specific improvement suggestions, such as simplifying the avatar operation UI, adding multilingual support to the product search function, improving the translation accuracy of product descriptions, improving the natural language processing accuracy of the customer service AI, strengthening accessibility features, improving error handling in the payment flow, optimizing promotion displays, adding operation guides for adding items to the cart, improving the visibility of coupon input fields, strengthening explanations of point accrual, simplifying the return procedure flow, improving the UI for posting reviews, clarifying the operation procedures for participating in events, strengthening navigation for moving between multiple floors, improving the usability of virtual try-ons, adding multilingual support for voice guidance, adding a function to prevent accidental sending of real-time chat messages, adding operation guides for in-store mini-games, optimizing avatar control during crowded times, and implementing error prevention measures for purchasing limited-edition products. These suggestions are then output as a feedback report for the development team. The feedback report includes graphs and heatmaps, operation videos, detailed behavior logs for each agent, screenshots of failures, prioritization of improvement suggestions, a problem matrix for each user group, a comparison of A / B test results, and measurement of the effects before and after improvements.

[0066] The user interface operates on the terminal and provides an intuitive interface that allows development teams and stakeholders to perform a variety of operations, such as inputting and editing store scenario information, instructing simulation execution, viewing feedback reports, confirming improvement suggestions, setting simulation parameters, customizing agent profiles, searching past simulation results, comparing A / B tests, playing operation videos, viewing heatmaps, and adding comments to problem areas. For example, it includes functions that meet the diverse needs of the development environment, such as creating screen transition diagrams using drag-and-drop, registering user scenarios using form input, selecting target user groups using checkboxes, adjusting the number of simulations using a slider, specifying test periods using a calendar, outputting feedback reports as PDFs, and linking improvement suggestions to a task management system.

[0067] The database management unit operates on the server and centrally manages and stores vast amounts of data handled by the entire system, including store scenario information, virtual agent profile information, user interaction data, simulation results, feedback reports, past user test data, external data integration results, and system configuration information. For example, it performs version control for each prototype, allowing comparison with past simulation results, and saves agent behavior logs chronologically, enabling detailed analysis and reproduction simulations later. Furthermore, the database management unit also includes functions to ensure security and reliability, such as access rights management, data encryption, backup, and log auditing.

[0068] The external data integration unit operates on the server and integrates with a variety of external data sources, including publicly available user behavior databases, social media review data, search engine query logs, customer support inquiry history, industry-standard UI guidelines, accessibility standards, publicly available information on competing services, open data, and API integration partners. This contributes to improving the virtual agent's behavior model and problem extraction algorithm, optimizing feedback generation logic, and enhancing simulation accuracy. For example, it can automatically collect trending UI trends on social media and reflect them in the virtual agent's behavior patterns, or analyze frequently occurring customer support inquiries to utilize them for problem extraction during simulations.

[0069] The prompt generation unit operates on the server and automatically generates the optimal prompt text for the generation AI. For example, "This virtual store prototype consists of the following steps: avatar operation, product search, product description viewing, conversation with customer service AI, use of accessibility features, payment, promotion display, cart addition, coupon use, point accrual, return procedures, review posting, event participation, movement between multiple floors, virtual try-on, use of voice guidance, real-time chat, in-store mini-games, behavior during crowded times, and purchase of limited-edition products. The target users are diverse, including beginners, foreign language speakers, people with disabilities, advanced users, children, and the elderly. Each agent executes each step sequentially, handling any confusion, failures, abandonment, errors, help requests, and inquiries that may occur along the way." Record the user's actions in detail. Pay particular attention to the ease of avatar operation, ease of product search, multilingual support for product descriptions, naturalness of customer service AI, effectiveness of accessibility features, security and clarity of payment flow, appropriateness of promotion display, ease of adding to cart and using coupons, clarity of point accrual and return procedures, ease of posting reviews and participating in events, usability of moving between floors and virtual try-ons, effectiveness of voice guidance and real-time chat, in-store mini-games and avatar control during busy periods, and error prevention measures for purchasing limited-edition products. Identify the challenges for each user segment and generate specific improvement proposals. The system will automatically generate prompts such as "[...].

[0070] As described above, the virtual store experience evaluation system according to this embodiment utilizes generative AI in a multi-stage and multi-purpose manner, automating everything from the generation of virtual agents, simulation of user interactions, analysis of operation results, automatic generation of feedback, utilization of external data, and optimization of the dynamic verification cycle. This significantly streamlines user testing that previously relied on human resources and enables the rapid provision of comprehensive verification from diverse user perspectives and concrete improvement suggestions. For example, this embodiment is extremely practical and highly scalable in all aspects of virtual store operation, such as designing new virtual shopping malls, adding functions to existing virtual stores, comparing multiple UI versions, verifying accessibility and multilingual support, evaluating congestion handling during events, and verifying error prevention measures when selling limited-edition products.

[0071] (System Configuration)

[0072] The system according to this embodiment includes a store scenario information acquisition unit, a virtual agent generation unit, a user interaction simulation unit, an analysis feedback generation unit, a user interface unit, a database management unit, an external data linkage unit, and a prompt generation unit. The store scenario information acquisition unit acquires and manages the experience scenarios in the virtual store from multiple perspectives and in detail. For example, this includes avatar operation (movement, jumping, viewpoint switching, gestures, clothing changes), product search (category selection, keyword input, voice search, image search, ranking viewing), product description viewing (multilingual switching, video playback, zoom display, review viewing, specification comparison), conversation with customer service AI (text chat, voice conversation, FAQ automated response, product recommendation, complaint handling), use of accessibility functions (screen zoom, color vision correction, text-to-speech, keyboard operation, subtitle display), payment (adding to cart, using coupons, using points, selecting multiple payment methods, identity verification), promotion display (banners, pop-ups, limited sales, lottery participation, coupon distribution), in-store movement (moving between floors, elevators, escalators, shortcut movement, guidance during crowded times), and cart operation (adding multiple items simultaneously, deleting, changing quantities, checking inventory). The system allows users to input diverse operational flows and user scenarios unique to virtual stores from their terminals and manage them centrally on the server. These include: return procedures (selection of return reason, selection of return method, refund procedure, return history confirmation), review posting (star rating, comment input, image attachment, anonymous posting, post editing), event participation (stamp rally, quiz, live streaming viewing, lottery entry, purchase of limited items), virtual try-on (wearing on an avatar, size selection, color change, mirror display, comparison with other users), use of voice guidance (navigation, product description, event guidance, emergency guidance, accessibility assistance), real-time chat (conversation with other users, conversation with store staff, group chat, stamp sending, reporting function), in-store mini-games (point acquisition, ranking, cooperative play, reward acquisition, difficulty selection), actions during crowded times (queue formation, entry restrictions, congestion status display, priority entry, suggested routes to avoid congestion), and purchase of limited items (lottery, first-come-first-served, reservation, purchase restrictions, notification function).Furthermore, it is possible to input detailed scenario data such as UI elements for each screen (buttons, links, images, videos, input fields, lists, tabs, sliders, checkboxes, radio buttons), validation specifications for input fields (required fields, character limits, format restrictions, real-time error display), conditions for displaying error messages (input errors, communication errors, out-of-stock items, payment failures, authentication errors), responsive design behavior (layout changes depending on screen size, portrait / landscape switching, zoom support), accessibility support status (screen reader support, keyboard operation, color vision correction, text-to-speech, subtitle display), and the purpose and order in which users operate the system.

[0073] The virtual agent generation unit automatically generates virtual agents with diverse user profiles using a generative AI. Examples include: beginners (first-time users, unread operation guides, low IT literacy), foreign language speakers (non-native Japanese speakers, English speakers, Chinese speakers, dependent on translation accuracy), people with disabilities (visual impairment, hearing impairment, motor impairment, color vision deficiency, cognitive impairment), advanced users (frequent users, shortcut users, customization-oriented), children (elementary school students, junior high school students, accompanied by a guardian, age restrictions), seniors (65 years and older, slow operation speed, frequent errors), users with low IT literacy (smartphone beginners, no PC experience), users who frequently use virtual stores (repeat customers, VIP members, high point reward orientation), first-time users (account not registered, guide unread, unfamiliar with operation), and users who strongly favor coupon use. The system automatically generates dozens to thousands of virtual agents that accurately simulate real-world user demographics, such as users who prioritize discounts, collect coupons, and use multiple coupons; review-focused users (referencing other users' ratings, frequent review posting); users with multiple devices (smartphones, tablets, and PCs); users of voice guidance (visually impaired, using while driving, dependent on voice recognition accuracy); users of AR features (camera operation, spatial recognition, AR display errors); multilingual speakers (multilingual switching, translation accuracy comparison); left-handed users (UI layout, usability); users with color vision deficiencies (red-green color blindness, blue-yellow color blindness, emphasis on contrast); impatient users (reduced waiting time, immediate response requests); users who stay for extended periods (visiting multiple stores, participating in events, collecting points); and users with multiple users simultaneously (family use, group use, simultaneous operation). Generative AI automatically sets an extremely wide range of attributes, including age, gender, language, disability status, IT literacy, purchase history, past virtual store usage experience, cultural background, stress tolerance, tendency to make mistakes, information seeking behavior, decision-making patterns, accessibility usage tendencies, reaction speed, emotional response, attention, memory, preferred UI design, responsiveness to promotions, review posting tendencies, event participation frequency, limited-edition product purchase experience, return experience, chat usage frequency, and mini-game participation tendencies.

[0074] The user interaction simulation unit uses a generated virtual agent to sequentially execute each operation step of the store scenario. For example, the avatar may get stuck on a wall, be unable to jump, the viewpoint switching may be unclear, errors may occur when changing clothes, the user may select the wrong category or enter the wrong keyword in the product search, there may be recognition errors in the voice search, similar products may not be found in the image search, the user may get lost when viewing rankings, the user may fail to switch to multiple languages ​​when viewing product descriptions, video playback may stop midway, the user may not be able to zoom in, the user may be misled by misinformation when viewing reviews, and the user may get confused when comparing specifications. Disruptive, unnatural responses in conversations with customer service AI, irrelevant FAQ automated responses, product recommendations that don't match needs, unsatisfactory handling of complaints, screen zoom not working with accessibility features, insufficient color correction, text-to-speech stopping midway, keyboard operation impossible, delayed subtitle display, failure to add items to cart during checkout, inability to use coupons, unclear point usage, confusion with multiple payment method selection, errors during identity verification, banners obstructing promotional displays, inability to close pop-ups, unclear how to participate in limited-time sales, errors when participating in raffles, coupons Distribution is missed, it's difficult to navigate between floors in the store, elevators are unusable, escalators are operated incorrectly, shortcuts are not found, guidance during peak hours is insufficient, adding multiple items to the cart simultaneously fails, deletions and quantity changes are not reflected, incorrect information is displayed in inventory checks, incorrect reason selection is made during the return process, the return method is unclear, errors occur during the refund process, return history is not displayed, star ratings are not reflected in reviews, comments are limited by character count, images cannot be attached, anonymous posting is not possible, edits to posts are not reflected, standing during event participation I can't understand the progress of the plaline, I can't submit quiz answers, there are playback errors when watching live streams, I can't confirm that my entry for the lottery is complete, I hit the purchase limit when buying limited items, the notification function doesn't work, the clothes don't appear on my avatar when I try on clothes in virtual try-on mode, there are errors when selecting sizes, I can't change colors, the mirror display is buggy, I can't compare myself to other users, the navigation stops midway when using voice guidance, I can't hear the product description, event announcements are delayed, emergency announcements don't work, accessibility assistance is insufficient, conversations with other users are interrupted in real-time chat.The system replicates a variety of problems that real users might encounter, such as inability to communicate with staff, accidental sending of messages in group chats, stamps not being reflected, reporting functions not working, inability to earn points in in-store mini-games, rankings not updating, inability to play cooperatively, rewards not being reflected, inability to select difficulty levels, inability to form queues during peak hours, entry restrictions not functioning, delayed display of congestion status, priority entry not being reflected, insufficient suggestions for avoiding congestion, inability to participate in lotteries for limited-edition product purchases, errors with first-come, first-served systems, reservations not being reflected, purchase restrictions malfunctioning, and delayed notification functions. Each agent meticulously records actions such as "success," "failure," "hesitation," "abandonment," "retry," "refer to help," and "inquiry generated" for each step, and automatically generates and records detailed user interaction data such as eye movement during operation, mouse movements, touch operation trajectories, input speed, accidental tap rate, scroll amount, page dwell time, click heatmap, error locations, and number of transitions to help pages.

[0075] The analysis and feedback generation unit uses a generational AI to aggregate and analyze the operation results of all virtual agents. For example, it automatically calculates the number of failures, dropout rate, average operation time, frequency of confusion, number of retries, help referral rate, inquiry rate, error location, number of page transitions, operation completion rate, and satisfaction score for each operation procedure. It then automatically extracts issues such as frequent failures or dropouts in specific procedures, or issues concentrated in specific user groups. Furthermore, for each identified problem, the system automatically generates specific improvement suggestions, such as simplifying the avatar operation UI, adding multilingual support to the product search function, improving the translation accuracy of product descriptions, improving the natural language processing accuracy of the customer service AI, strengthening accessibility features, improving error handling in the payment flow, optimizing promotion displays, adding operation guides for adding items to the cart, improving the visibility of coupon input fields, strengthening explanations of point accrual, simplifying the return procedure flow, improving the UI for posting reviews, clarifying the operation procedures for participating in events, strengthening navigation for moving between multiple floors, improving the usability of virtual try-ons, adding multilingual support for voice guidance, adding a function to prevent accidental sending of real-time chat messages, adding operation guides for in-store mini-games, optimizing avatar control during crowded times, and implementing error prevention measures for purchasing limited-edition products. These suggestions are then output as a feedback report for the development team. The feedback report includes graphs and heatmaps, operation videos, detailed behavior logs for each agent, screenshots of failures, prioritization of improvement suggestions, a problem matrix for each user group, a comparison of A / B test results, and measurement of the effects before and after improvements.

[0076] The user interface operates on the terminal and provides an intuitive interface that allows development teams and stakeholders to perform a variety of operations, such as inputting and editing store scenario information, instructing simulation execution, viewing feedback reports, confirming improvement suggestions, setting simulation parameters, customizing agent profiles, searching past simulation results, comparing A / B tests, playing operation videos, viewing heatmaps, and adding comments to problem areas. For example, it includes functions that meet the diverse needs of the development environment, such as creating screen transition diagrams using drag-and-drop, registering user scenarios using form input, selecting target user groups using checkboxes, adjusting the number of simulations using a slider, specifying test periods using a calendar, outputting feedback reports as PDFs, and linking improvement suggestions to a task management system.

[0077] The database management unit operates on the server and centrally manages and stores vast amounts of data handled by the entire system, including store scenario information, virtual agent profile information, user interaction data, simulation results, feedback reports, past user test data, external data integration results, and system configuration information. For example, it performs version control for each prototype, allowing comparison with past simulation results, and saves agent behavior logs chronologically, enabling detailed analysis and reproduction simulations later. Furthermore, the database management unit also includes functions to ensure security and reliability, such as access rights management, data encryption, backup, and log auditing.

[0078] The external data integration unit operates on the server and integrates with a variety of external data sources, including publicly available user behavior databases, social media review data, search engine query logs, customer support inquiry history, industry-standard UI guidelines, accessibility standards, publicly available information on competing services, open data, and API integration partners. This contributes to improving the virtual agent's behavior model and problem extraction algorithm, optimizing feedback generation logic, and enhancing simulation accuracy. For example, it can automatically collect trending UI trends on social media and reflect them in the virtual agent's behavior patterns, or analyze frequently occurring customer support inquiries to utilize them for problem extraction during simulations.

[0079] The prompt generation unit operates on the server and automatically generates the optimal prompt text for the generation AI. For example, "This virtual store prototype consists of the following steps: avatar operation, product search, product description viewing, conversation with customer service AI, use of accessibility features, payment, promotion display, cart addition, coupon use, point accrual, return procedures, review posting, event participation, movement between multiple floors, virtual try-on, use of voice guidance, real-time chat, in-store mini-games, behavior during crowded times, and purchase of limited-edition products. The target users are diverse, including beginners, foreign language speakers, people with disabilities, advanced users, children, and the elderly. Each agent executes each step sequentially, handling any confusion, failures, abandonment, errors, help requests, and inquiries that may occur along the way." Record the user's actions in detail. Pay particular attention to the ease of avatar operation, ease of product search, multilingual support for product descriptions, naturalness of customer service AI, effectiveness of accessibility features, security and clarity of payment flow, appropriateness of promotion display, ease of adding to cart and using coupons, clarity of point accrual and return procedures, ease of posting reviews and participating in events, usability of moving between floors and virtual try-ons, effectiveness of voice guidance and real-time chat, in-store mini-games and avatar control during busy periods, and error prevention measures for purchasing limited-edition products. Identify the challenges for each user segment and generate specific improvement proposals. The system will automatically generate prompts such as "[...].

[0080] Thus, the system according to this embodiment, through the coordinated operation of its various components, can comprehensively and multifaceted simulate user interactions in all aspects of the virtual store experience, accurately simulate diverse real-world user segments, automate problem identification and improvement suggestions based on vast amounts of behavioral data, and provide highly practical and scalable feedback to the development team.

[0081] (Implementation Steps)

[0082] Step 1: Input and retrieve virtual store scenario information This step involves the development team and other stakeholders using terminals to input highly multifaceted and detailed user experience scenarios for the virtual store into the system. Here, users can input detailed information about the diverse operation flows and user scenarios unique to virtual stores, such as avatar control (movement, jumping, viewpoint switching, gestures, clothing changes), product search (category selection, keyword input, voice search, image search, ranking viewing), product description viewing (multilingual switching, video playback, zoom display, review viewing, specification comparison), conversation with customer service AI (text chat, voice conversation, FAQ auto-response, product recommendation, complaint handling), use of accessibility features (screen zoom, color vision correction, text-to-speech, keyboard operation, subtitle display), payment (adding to cart, using coupons, using points, selecting multiple payment methods, identity verification), promotion display (banners, pop-ups, limited-time sales, lottery participation, coupon distribution), in-store movement (moving between floors, elevators, escalators, shortcut movement, guidance during crowded times), cart operation, return procedures, review posting, event participation, virtual try-on, voice guidance, real-time chat, in-store mini-games, behavior during crowded times, and purchase of limited-edition products, and these are centrally managed on the server. Furthermore, it is possible to input detailed scenario data such as UI elements for each screen, validation specifications for input fields, conditions for displaying error messages, responsive design behavior, accessibility compliance status, and the order in which users operate the system for what purpose.

[0083] Step 2: Create a virtual agent generation prompt and generate agents using the AI. Based on store scenario information, target user base, simulation objectives, past challenge trends, and external data integration results, the prompt generation unit automatically generates optimal prompt text for the generation AI, and the virtual agent generation unit automatically generates virtual agents with diverse user profiles. A concrete example of prompt text to be fed into the generation AI is: "This virtual store prototype consists of the following steps: avatar operation, product search, product description viewing, conversation with customer service AI, use of accessibility features, payment, promotion display, cart addition, coupon use, point accrual, return procedures, review posting, event participation, multi-floor movement, virtual try-on, voice guidance use, real-time chat, in-store mini-games, behavior during crowded times, and purchase of limited-edition products. The target users are diverse, including beginners, foreign language speakers, people with disabilities, advanced users, children, and the elderly. Each agent executes each step sequentially, encountering confusion, failures, abandonment, and errors along the way." Record in detail actions such as referring to help and making inquiries. Focus particularly on ease of avatar operation, ease of product search, multilingual support for product descriptions, naturalness of customer service AI, effectiveness of accessibility features, security and clarity of payment flow, appropriateness of promotion display, ease of adding to cart and using coupons, clarity of point accrual and return procedures, ease of posting reviews and participating in events, usability of moving between multiple floors and virtual try-ons, effectiveness of voice guidance and real-time chat, in-store mini-games and avatar control during busy periods, and error prevention measures for purchasing limited-edition products. Extract challenges for each user group and generate specific improvement proposals. Based on this prompt, the generating AI will automatically create dozens to thousands of virtual agents with diverse user profiles, such as beginners, foreign language speakers, people with disabilities, advanced users, children, and the elderly.

[0084] Step 3: Run the user interaction simulation. This step involves the generated virtual agent sequentially executing each operational step of the store scenario. Each agent experienced various issues, including: getting stuck on walls while controlling their avatar, being unable to jump, difficulty switching viewpoints, errors when changing outfits, incorrect category selection and keyword input during product searches, voice search recognition errors, inability to find similar products in image searches, getting lost while viewing rankings, failure to switch languages ​​when viewing product descriptions, videos stopping midway, inability to zoom in, being misled by misinformation when viewing reviews, confusion when comparing specifications, unnatural responses in conversations with customer service AI, irrelevant FAQ automated responses, product recommendations not matching needs, dissatisfaction with complaint handling, screen zoom not working with accessibility features, insufficient color correction, voice reading stopping midway, inability to use keyboard, delayed subtitle display, failure to add items to cart during checkout, inability to use coupons, difficulty using points, confusion with multiple payment method selections, errors during identity verification, banners obstructing promotional displays, inability to close pop-ups, unclear methods for participating in limited-time sales, errors when participating in raffles, missing coupon distributions, difficulty navigating between floors within the store, and difficulty operating elevators. Unable to do something, misoperation on escalators, unable to find shortcuts, insufficient guidance during crowded times, failure to add multiple items to the cart simultaneously, deletion or quantity changes not reflected, incorrect information in inventory check, incorrect reason selection in return procedure, unsure how to return items, error in refund procedure, return history not displayed, star rating not reflected in review submission, exceeding character limit in comment input, unable to attach images, unable to post anonymously, post edits not reflected, unable to progress in stamp rally during event participation, unable to submit quiz answers, playback error during live stream viewing, unsure if entry is complete in lottery applications, exceeding purchase limits when purchasing limited items, notification function not working, clothes not reflected on avatars in virtual try-on, size selection error, unable to change colors, mirror display bug, unable to compare with other users, navigation stopping midway when using voice guidance, unable to hear product descriptions, delayed event announcements, emergency announcements not working, insufficient accessibility assistance, conversations with other users interrupted in real-time chat, unable to communicate with store staff.The system replicates a variety of problems that real users might encounter, such as accidental sending of group chat messages, stamps not being reflected, reporting functions not working, inability to earn points in in-store mini-games, rankings not updating, cooperative play not being established, rewards not being reflected, inability to select difficulty levels, queues not forming during peak hours, entry restrictions not functioning, congestion status display being delayed, priority entry not being reflected, insufficient suggestions for avoiding congestion, inability to participate in lotteries for limited-edition product purchases, errors with first-come, first-served systems, reservations not being reflected, purchase restrictions malfunctioning, and delayed notification functions. Each agent meticulously records actions such as "success," "failure," "hesitation," "abandonment," "retry," "refer to help," and "inquiry generated" for each step, and automatically generates and records detailed user interaction data such as eye movement during operation, mouse movements, touch operation trajectories, input speed, accidental tap rate, scroll amount, page dwell time, click heatmap, error locations, and number of transitions to help pages.

[0085] Step 4: Analysis of simulation results and identification of issues (utilizing generation AI) The analysis and feedback generation unit uses a generation AI to aggregate and analyze the operation results of all virtual agents. Specific examples of prompts to be fed into the generation AI include: "Based on the operation history of all agents, calculate the number of failures, dropout rate, average operation time, frequency of confusion, number of retries, help referral rate, inquiry rate, error location, page transition count, operation completion rate, satisfaction score, etc., for each operation procedure, and automatically extract issues such as frequent failures or dropouts in specific procedures, or issues concentrated in specific user groups. Furthermore, generate specific improvement suggestions for the extracted problems." For example, we extract issues as specific numerical data, such as an 80% failure rate for disabled agents when operating avatars, a 60% failure rate for foreign language-speaking agents when viewing product descriptions, a high dropout rate for novice agents during checkout, high stress scores for advanced agents when displaying promotions, frequent errors by child agents when adding items to the cart, frequent confusion among elderly agents during return procedures, inability to operate virtual try-ons for agents with color vision deficiencies, many failures by hearing-impaired agents with voice guidance, frequent mis-sentences by foreign language-speaking agents in real-time chat, and a high overall error rate when purchasing limited-edition items. Furthermore, for each identified problem, the system automatically generates specific improvement suggestions, such as simplifying the avatar operation UI, adding multilingual support to the product search function, improving the translation accuracy of product descriptions, improving the natural language processing accuracy of the customer service AI, strengthening the accessibility feature guide, improving error handling in the payment flow, optimizing promotion displays, adding operation guides for adding items to the cart, improving the visibility of the coupon input field, strengthening the explanation of point accrual, simplifying the return procedure flow, improving the UI for posting reviews, clarifying the operation procedures for participating in events, strengthening navigation for moving between multiple floors, improving the usability of virtual try-ons, adding multilingual support for voice guidance, adding a function to prevent accidental sending of real-time chat messages, adding operation guides for in-store mini-games, optimizing avatar control during crowded times, and implementing error prevention measures for purchasing limited-edition products. These suggestions are then output as a feedback report for the development team.

[0086] Step 5: Automatic generation and provision of feedback reports The analysis and feedback generation unit automatically generates feedback reports for the development team based on the extracted issues and improvement suggestions. The feedback reports include graphs, heatmaps, operation videos, detailed behavior logs for each agent, screenshots of failures, priority rankings of improvement suggestions, issue matrices for each user group, A / B test results comparisons, and measurement of the effects before and after improvements. These reports can be viewed intuitively from the terminal, allowing the development team to quickly grasp specific improvement measures and reflect them in the next development cycle.

[0087] Step 6: System optimization through external data integration and continuous learning The External Data Integration Department integrates with a diverse range of external data sources, including social media review data, search engine query logs, customer support inquiry history, industry-standard UI guidelines, accessibility standards, publicly available information from competitor services, open data, and API integration partners. This contributes to the advancement of virtual agent behavior models and problem identification algorithms, the optimization of feedback generation logic, and the improvement of simulation accuracy. Furthermore, the generating AI utilizes past simulation results and actual user test data to continuously learn and evolve, achieving more realistic and accurate user simulations and problem identification.

[0088] (Specific use cases)

[0089] For example, when applying the virtual store experience evaluation system of the present invention to the opening of a new virtual shopping mall, the development team uses a terminal to register in detail various operation flows and user scenarios unique to virtual stores into the system, such as avatar operation (movement, jumping, viewpoint switching, gestures, changing clothes), product search (category selection, keyword input, voice search, image search, ranking viewing), product description viewing (multilingual switching, video playback, zoom display, review viewing, specification comparison), conversation with customer service AI (text chat, voice conversation, FAQ automatic response, product recommendation, complaint handling), use of accessibility functions (screen zoom, color vision correction, text-to-speech, keyboard operation, subtitle display), payment (adding to cart, using coupons, using points, selecting multiple payment methods, identity verification), promotion display (banners, pop-ups, limited sales, lottery participation, coupon distribution), in-store movement (moving between floors, elevators, escalators, shortcut movement, guidance during crowded times), cart operation, return procedures, review posting, event participation, virtual try-on, use of voice guidance, real-time chat, in-store mini-games, behavior during crowded times, purchase of limited-edition products, etc. Furthermore, detailed scenario data is also entered, including UI elements for each screen, validation specifications for input fields, conditions for displaying error messages, responsive design behavior, accessibility compliance status, and how and in what order users interact with the system.

[0090] The virtual agent generation unit uses a generation AI to automatically generate hundreds of virtual agents with diverse user profiles, including beginners, foreign language speakers, people with disabilities (visual impairment, hearing impairment, motor impairment, color blindness, cognitive impairment), advanced users, children, the elderly, users with low IT literacy, frequent users, first-time users, coupon-oriented users, review-focused users, multiple device users, voice guidance users, AR function users, multilingual users, left-handed users, impatient users, long-stay users, and users with multiple simultaneous users. Each agent possesses extremely diverse attributes, such as age, gender, language, disability status, IT literacy, purchase history, past virtual store usage experience, cultural background, stress tolerance, tendency to make mistakes, information-seeking behavior, decision-making patterns, accessibility usage tendencies, reaction speed, emotional response, attention span, memory, preferred UI design, responsiveness to promotions, review posting tendencies, event participation frequency, limited-edition product purchase experience, return experience, chat usage frequency, and mini-game participation tendencies.

[0091] The user interaction simulation unit uses the generated virtual agent to sequentially execute each operation step of the store scenario. For example, avatars get stuck on walls, jumping is impossible, switching viewpoints is confusing, errors occur when changing outfits, incorrect category selection or keyword input in product searches, voice search recognition errors, similar products cannot be found in image searches, getting lost when viewing rankings, failing to switch languages ​​when viewing product descriptions, videos stopping midway, inability to zoom in, being misled by misinformation when viewing reviews, getting confused when comparing specifications, unnatural responses in conversations with customer service AI, irrelevant FAQ automated responses, product recommendations not matching needs, dissatisfaction with complaint handling, screen zoom not working with accessibility features, insufficient color correction, voice reading stopping midway, keyboard operation impossible, delayed subtitle display, failure to add items to cart during checkout, inability to use coupons, confusing point usage, confusion with multiple payment method selection, errors during identity verification, banners obstructing promotion displays, inability to close pop-ups, unclear methods for participating in limited-time sales, errors when participating in raffles, missed coupon distributions, difficulty navigating between floors in the store, elevators... Unable to operate the turntable, misoperation on escalators, unable to find shortcuts, insufficient guidance during crowded times, failure to add multiple items to the cart simultaneously, deletion or quantity changes not reflected, incorrect information in inventory check, incorrect reason selection during return procedures, unsure how to return items, error during refund procedures, return history not displayed, star ratings not reflected in reviews, exceeding character limits when entering comments, unable to attach images, unable to post anonymously, post edits not reflected, unable to progress through stamp rallies during events, unable to submit quiz answers, playback errors during live streaming, unsure if entry is complete in lottery applications, exceeding purchase limits when purchasing limited items, notification function not working, items not reflected on avatars in virtual try-on, size selection errors, unable to change colors, mirror display bugs, unable to compare with other users, navigation stopping midway when using voice guidance, product descriptions inaudible, delayed event announcements, emergency announcements not working, insufficient accessibility assistance, conversations with other users interrupted in real-time chat.The system replicates a variety of problems that real users might encounter, such as inability to communicate with staff, accidental sending of messages in group chats, stamps not being reflected, reporting functions not working, inability to earn points in in-store mini-games, rankings not updating, inability to play cooperatively, rewards not being reflected, inability to select difficulty levels, inability to form queues during peak hours, entry restrictions not functioning, delayed display of congestion status, priority entry not being reflected, insufficient suggestions for avoiding congestion, inability to participate in lotteries for limited-edition product purchases, errors with first-come, first-served systems, reservations not being reflected, purchase restrictions malfunctioning, and delayed notification functions. Each agent meticulously records actions such as "success," "failure," "hesitation," "abandonment," "retry," "refer to help," and "inquiry generated" for each step, and automatically generates and records detailed user interaction data such as eye movement during operation, mouse movements, touch operation trajectories, input speed, accidental tap rate, scroll amount, page dwell time, click heatmap, error locations, and number of transitions to help pages.

[0092] The analysis and feedback generation unit uses generation AI to aggregate and analyze the operation results of all virtual agents, extracting issues as specific numerical data. For example, it might reveal that the failure rate for disabled agents is 80% when operating avatars, the failure rate for foreign language-speaking agents when viewing product descriptions is 60%, the dropout rate for novice agents is high when making payments, advanced agents have high stress scores when displaying promotions, child agents frequently make mistakes when adding items to the cart, elderly agents frequently get confused during the return process, agents with color vision deficiencies become unable to operate virtual try-ons, hearing-impaired agents make many mistakes with voice guidance, foreign language-speaking agents frequently send messages to the wrong recipients in real-time chat, and the overall error rate is high when purchasing limited-edition products. Furthermore, for each identified problem, the system automatically generates specific improvement suggestions, such as simplifying the avatar operation UI, adding multilingual support to the product search function, improving the translation accuracy of product descriptions, improving the natural language processing accuracy of the customer service AI, strengthening accessibility features, improving error handling in the payment flow, optimizing promotion displays, adding operation guides for adding items to the cart, improving the visibility of coupon input fields, strengthening explanations of point accrual, simplifying the return procedure flow, improving the UI for posting reviews, clarifying the operation procedures for participating in events, strengthening navigation for moving between multiple floors, improving the usability of virtual try-ons, adding multilingual support for voice guidance, adding a function to prevent accidental sending of real-time chat messages, adding operation guides for in-store mini-games, optimizing avatar control during crowded times, and implementing error prevention measures for purchasing limited-edition products. These suggestions are then output as a feedback report for the development team. The feedback report includes graphs and heatmaps, operation videos, detailed behavior logs for each agent, screenshots of failures, prioritization of improvement suggestions, a problem matrix for each user group, a comparison of A / B test results, and measurement of the effects before and after improvements.

[0093] Specific examples of prompt sentences to be fed into the generating AI to realize this invention include: "This virtual store prototype consists of the following steps: avatar operation, product search, product description viewing, conversation with customer service AI, use of accessibility features, payment, promotion display, cart addition, coupon use, point accrual, return procedure, review posting, event participation, movement between multiple floors, virtual try-on, use of voice guidance, real-time chat, in-store mini-games, behavior during crowded times, and purchase of limited-edition products. The target users are diverse, including beginners, foreign language speakers, people with disabilities, advanced users, children, and the elderly. Each agent will execute each step sequentially, and any confusion or failures that occur along the way, Record in detail actions such as abandonment, errors, help requests, and inquiries. Focus particularly on ease of avatar operation, clarity of product search, multilingual support for product descriptions, naturalness of customer service AI, effectiveness of accessibility features, security and clarity of payment flow, appropriateness of promotional displays, ease of adding items to cart and using coupons, clarity of point accrual and return procedures, ease of posting reviews and participating in events, usability of moving between floors and virtual try-ons, effectiveness of voice guidance and real-time chat, in-store mini-games and avatar control during busy periods, and error prevention measures for purchasing limited-edition products. Identify challenges for each user segment and generate specific improvement proposals.

[0094] The specific processing unit 290 transmits the result of the specific processing obtained in the embodiment 1 or application example 1 described above to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0095] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include text generation AI, image generation AI, and multimodal generation AI. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats from audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each of the above parts is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example.Furthermore, processing performed by AI, including generative AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI, including generative AI.

[0096] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0097] For example, the data collection unit is implemented by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit acquires step count data using the camera 42 and communication I / F 44 of the smart device 14 and processes it with the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 and analyzes the data from the data collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 and generates a cooking menu using generation AI. For example, the serving unit is implemented by the output device 40 of the smart device 14 or the specific processing unit 290 of the data processing device 12 and provides the generated cooking menu to the user. The correspondence between each unit and the devices and control units is not limited to the examples described above and can be modified in various ways.

[0098] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0099] [Second Embodiment]

[0100] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0101] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0102] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0103] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and communication interface 44 are also connected to the bus 52.

[0104] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0105] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0106] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0107] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0108] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0109] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0110] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the identification processing unit 290 using these models.

[0111] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is realized by the data processing device 12 and the smart glasses 214. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0112] (Example 1)

[0113] Since the process flow is the same as in Example 1 described in the First Embodiment above, the explanation will be omitted.

[0114] (Application Example 1) Since the process flow is the same as in Application Example 1 described in the first embodiment above, the explanation will be omitted.

[0115] The specific processing unit 290 transmits the result of the specific processing obtained in the embodiment 1 or application example 1 described above to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0116] Furthermore, the processing by the data processing system 10 described above is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0117] For example, the data collection unit is implemented by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit acquires step count data using the camera 42 and communication I / F 44 of the smart glasses 214 and processes it using the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 and analyzes the data from the data collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 and generates a cooking menu using generation AI. For example, the serving unit is implemented by the speaker 240 of the smart glasses 214 or the specific processing unit 290 of the data processing device 12 and provides the generated cooking menu to the user. The correspondence between each unit and the device or control unit is not limited to the examples described above and can be modified in various ways.

[0118] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0119] [Third Embodiment]

[0120] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0121] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0122] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0123] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, display 343, and communication interface 44 are also connected to the bus 52.

[0124] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0125] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0126] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0127] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0128] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0129] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0130] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0131] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the headset terminal 314. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0132] (Example 1) Since the process flow is the same as in Example 1 described in the First Embodiment above, the explanation will be omitted.

[0133] (Application Example 1) Since the process flow is the same as in Application Example 1 described in the first embodiment above, the explanation will be omitted.

[0134] The specific processing unit 290 transmits the result of the specific processing obtained in the embodiment 1 or application example 1 described above to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0135] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset terminal 314, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the headset terminal 314 or an external device, and the headset terminal 314 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0136] For example, the data collection unit is implemented by the control unit 46A of the headset terminal 314 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit acquires step count data using the camera 42 and communication I / F 44 of the headset terminal 314 and processes it using the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 and analyzes the data from the data collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 and generates a cooking menu using generation AI. For example, the serving unit is implemented by the speaker 240 and display 343 of the headset terminal 314 or the specific processing unit 290 of the data processing device 12 and provides the generated cooking menu to the user. The correspondence between each unit and the device or control unit is not limited to the examples described above and can be modified in various ways.

[0137] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0138] [Fourth Embodiment]

[0139] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0140] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0141] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0142] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, controlled object 443, and communication interface 44 are also connected to the bus 52.

[0143] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0144] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0145] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0146] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0147] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0148] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0149] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0150] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0151] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the robot 414. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0152] (Example 1) Since the process flow is the same as in Example 1 described in the First Embodiment above, the explanation will be omitted.

[0153] (Application Example 1) Since the process flow is the same as in Application Example 1 described in the first embodiment above, the explanation will be omitted.

[0154] The specific processing unit 290 transmits the result of the specific processing obtained in the embodiment 1 or application example 1 described above to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0155] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the robot 414 or external devices, and the robot 414 acquires or collects information necessary for processing from the data processing device 12 or external devices.

[0156] For example, the data collection unit is implemented by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit acquires step count data using the camera 42 and communication I / F 44 of the robot 414 and processes it using the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 and analyzes the data from the data collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 and generates a cooking menu using a generation AI. For example, the serving unit is implemented by the speaker 240 and controlled object 443 of the robot 414 and the specific processing unit 290 of the data processing device 12 and provides the generated cooking menu to the user. The correspondence between each unit and the device or control unit is not limited to the examples described above and can be modified in various ways.

[0157] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0158] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0159] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0160] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0161] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0162] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0163] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0164] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0165] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0166] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0167] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0168] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0169] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0170] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using this memory.

[0171] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0172] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0173] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0174] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0175] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0176] Furthermore, the following additional information is disclosed regarding the above explanation.

[0177] (Claim 1)

[0178] The system is a virtual store experience evaluation system comprising a store scenario information acquisition unit, a virtual agent generation unit, a user interaction simulation unit, and an analysis feedback generation unit. The store scenario information acquisition unit acquires specific operation flows and user scenario information in the virtual store, such as avatar operation, product search, product description viewing, conversation with customer service AI, use of accessibility functions, and payment. The virtual agent generation unit uses a generative AI to automatically generate virtual agents with diverse user profiles, such as beginners, foreign language speakers, people with disabilities, and advanced users. The user interaction simulation unit uses the virtual agents to sequentially execute each operation step of the store scenario, recording actions such as success or failure, confusion, abandonment, errors, help referral, and inquiry occurrence for each step. The analysis feedback generation unit uses a generative AI to aggregate and analyze the operation results of all virtual agents, automatically extracting steps with frequent failures or abandonment and issues specific to each user group, and generating and outputting feedback information that includes specific improvement suggestions such as UI improvement for avatar operation, multilingual support for product search, improved natural language processing accuracy for customer service AI, enhanced accessibility functions, and optimized error handling in the payment flow. (Claim 2)

[0179] The system according to claim 1, characterized in that the virtual agent generation unit automatically sets detailed attributes such as different behavioral patterns, operational proficiency, language comprehension, accessibility usage tendencies, stress tolerance, error tendencies, information-seeking behavior, and decision-making patterns for each user group using a generative AI, and when each virtual agent executes each operation procedure of the store scenario, it generates probabilistic success or failure, confusion, abandonment, errors, help referral, and inquiry generation according to the profile, thereby accurately simulating diverse real-world user experiences. (Claim 3)

[0180] The system according to claim 1, characterized in that the aforementioned analysis feedback generation unit aggregates the operation history of all virtual agents, automatically calculates the number of failures, dropout rate, average operation time, frequency of confusion, number of retries, help referral rate, inquiry rate, error location, number of page transitions, operation completion rate, satisfaction score, etc. for each operation procedure, automatically extracts issues such as when failures or dropouts occur frequently in a particular procedure or when issues are concentrated in a particular user group, automatically generates specific improvement suggestions for the extracted issues, such as simplifying the avatar operation UI, adding multilingual support to the product search function, improving the natural language processing accuracy of the customer service AI, strengthening the accessibility function guide, and improving error handling in the payment flow, and outputs these as a feedback report for the development team. [Explanation of symbols]

[0181] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots

Claims

1. A user test simulation system comprising a prototype information acquisition unit, a virtual agent generation unit, a user interaction evaluation unit, and an analysis feedback generation unit, wherein the prototype information acquisition unit acquires operation procedure information and related user scenario information of a prototype to be verified in new product development; the virtual agent generation unit automatically generates multiple virtual agents based on different user profiles using a generation AI; the user interaction evaluation unit simulates user interaction by sequentially executing each operation procedure of the prototype using the virtual agents, recording actions such as success or failure for each procedure; and the analysis feedback generation unit analyzes the operation results of all virtual agents using a generation AI, automatically extracts procedures with frequent failures and problems for each user group, and generates and outputs feedback information including points for improvement.

2. The system according to claim 1, characterized in that the generative AI sets different behavioral patterns and operational proficiency levels for each user profile of a virtual agent, and when each virtual agent executes the operation procedure of the prototype, it generates a probabilistic success or failure action according to the profile, thereby accurately simulating the diversity of the actual user base.

3. The system according to claim 1, characterized in that the analysis feedback generation unit aggregates the operation history of all virtual agents, counts the number of failures for each operation procedure, extracts procedures in which failures occurred more than a predetermined number of times as problematic, further analyzes the failure trends for each user profile to identify in detail which user group faced what kind of problems in which procedure, and automatically generates and outputs a feedback report for the development team that includes specific improvement suggestions.

Images