system

Abstract

Provide a system. 【Solution means】 Means for automatically detecting components of a screen display device and generating related information; Means for creating a test procedure table based on the detected components; Means for automatically preparing test information based on the test procedure table; Means for autonomously executing verification using the prepared test information; Means for automatically analyzing verification results and providing feedback; Means for recognizing the operation of a work device using a camera and assisting the operation test process; Means for transmitting an instruction to a device via a communication technology and executing verification content; A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the test process of system development, it is a problem that a large amount of man-hours and time are required. In particular, manual acquisition of screen elements, creation of a test item table, preparation of accurate test data, and implementation of tests involve multiple personnel, which are factors delaying the progress of the entire project. Also, the accuracy of the test is often compromised due to human errors and incomplete preparations. In such a situation, automation of system testing is required by achieving both efficiency and accuracy improvement.

Means for Solving the Problems

[0005] This invention streamlines the acquisition of elements to be tested by providing means for automatically identifying elements of a screen interface and generating metadata. It also introduces means for generating a test item list based on the identified elements, preventing omissions of test items. Furthermore, it provides an accurate and consistent dataset by using means for automatically preparing test data based on the test item list. By autonomously conducting tests using this prepared test data, the entire testing process is automated, and by efficiently analyzing test results and providing feedback, improved test accuracy and rapid response are achieved.

[0006] "Screen interface elements" refer to the visual and functional components that are interacted with, such as input fields, buttons, and dropdown menus, displayed on the user interface.

[0007] "Metadata" refers to data that describes information about other data, and in this invention, it refers to detailed information about the properties and arrangement of screen elements.

[0008] A "test item list" is a document that lists the operations and functions to be verified in a system test, and includes test inputs, test procedures, and expected results.

[0009] "Test data" refers to a dataset that specifies the input values ​​and scenarios used when conducting system tests.

[0010] "Means of autonomously conducting tests" refers to a method or mechanism for a system to automatically execute a test process and obtain results without human intervention.

[0011] "Feedback" refers to information and improvement suggestions obtained based on test results, which clearly identify system defects and areas for improvement. [Brief explanation of the drawing]

[0012] [Figure 1]This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0013] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0018] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] This invention provides a method for automating and efficiently executing the system testing process. Here, the program's processing is explained in natural language, and the overall operation is illustrated with concrete examples.

[0034] In this invention, the server first automatically acquires the screen elements of the user interface of the target system. The server identifies elements such as input fields and buttons from the screen and generates metadata related to them. This allows the interface information necessary for the testing process to be accumulated.

[0035] Next, the user specifies the type of test, and the server generates a test item list based on the identified elements and the specified test content. This test item list includes the specific procedures and expected results for each test case.

[0036] Subsequently, the server automatically prepares the test data based on the test item list. This includes the actual input data and processes used, and optimized data is provided for each test case.

[0037] Once ready, the server autonomously performs the tests. The server accesses the system through a terminal, inputs data according to the test item list, and performs operations. It also checks the system's response to each operation and evaluates whether the tests were performed as planned.

[0038] After the test, the server analyzes the results and reports successful test cases and problems. The server analyzes the test results as feedback and notifies the user of details of areas for improvement and anomalies. The results displayed on the terminal allow the user to understand the test status and modify the system or test items as needed.

[0039] As a concrete example, consider testing an online money transfer system for a bank. The user selects internal integration testing and specifies testing the money transfer function. The server scans the UI and obtains elements such as "transfer amount" and "transfer button." Based on this, the server generates a test item list and sets up test cases that include verifying the correctness of the transfer amount and checking the operation of the transfer button. Next, the server prepares the transfer test data and conducts the test. After the server checks whether the transfer process is successful and confirms the system's response, it provides feedback to the user with the analyzed results.

[0040] This implementation is expected to improve the overall efficiency and accuracy of system testing.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The server retrieves the UI screen of the target system and automatically identifies screen elements using image analysis and DOM analysis. The server extracts elements such as input fields, buttons, and dropdown menus, and generates metadata to record these elements.

[0044] Step 2:

[0045] The user selects the type of test and specifies the test objective (e.g., unit test, integration test). Based on this, the server automatically generates a test item list in conjunction with the screen element data it has analyzed. The test item list includes the specific procedure and expected results for each test case.

[0046] Step 3:

[0047] The server prepares the necessary test data for each case listed in the test item sheet. Using the system's database and existing datasets, it automatically generates datasets, including dummy data as needed, to provide the optimal data for each test case.

[0048] Step 4:

[0049] The server autonomously conducts tests using prepared test data. It accesses the system via a terminal and automatically performs operations according to the test item list. Specifically, it performs actions such as entering appropriate values ​​into screen elements and clicking designated buttons.

[0050] Step 5:

[0051] During the test, the server records the system's response to each operation and monitors the results in real time. It collects logs to compare the expected results with the actual responses and determine success or failure.

[0052] Step 6:

[0053] The server analyzes the test results and compiles the success rate and errors for all test cases. A detailed test report is provided to the terminal, allowing the user to review the test results. If a defect is found, details and improvement suggestions are included as feedback.

[0054] (Example 1)

[0055] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0056] Traditional system testing processes require manual verification of screen interfaces and setting of test items, which is time-consuming and labor-intensive, and has limitations in terms of test accuracy and efficiency. Furthermore, because the preparation and optimization of test data relied on manual processes, a system that could flexibly adapt to changes in test content was needed. To address these problems, it is necessary to automate the testing process while improving accuracy and efficiency.

[0057] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0058] In this invention, the server includes means for automatically identifying elements of a screen interface and generating metadata, means for generating a test item list based on the identified elements, and means for automatically preparing test data based on the test item list. This reduces manual work and enables automation and improved accuracy of the testing process.

[0059] A "screen interface" is a visual component of a computer program that a user can operate visually.

[0060] An "element" refers to an individual component or part within a screen interface that has a specific function, including input fields and buttons.

[0061] "Metadata" is data that describes information about an element, and may include the element's ID, class name, and location information.

[0062] A "test item list" is a table that lists the specific procedures required for the test and the expected results.

[0063] "Test data" refers to input information such as specific numerical values ​​and strings necessary to carry out the test items.

[0064] "Autonomously" refers to the ability to act automatically based on one's own judgment without waiting for external instructions.

[0065] "Test results" refers to the record of the outcomes and conditions obtained through the testing process.

[0066] "Feedback" refers to evaluations, notifications, or reports of areas for improvement provided based on test results.

[0067] "Optimization" refers to adjusting the resources of a system or data to utilize them in the most efficient way possible in order to achieve a specific objective.

[0068] "Access control" refers to the procedure of controlling access rights to a system or parts of it, allowing only authorized operations to be performed.

[0069] This invention provides a method for automating and efficiently executing the testing process of a system. The system consists of the interaction of a server, a terminal, and a user.

[0070] First, the server scans the target system's screen interface using web scraping or automation tools (e.g., Selenium). The server automatically identifies elements such as input fields and buttons from the scanned screen and generates metadata about them. This aggregates the interface information necessary for the testing process.

[0071] The user accesses the system through a terminal and specifies the type of test. For example, they can choose from UI tests, functional tests, stress tests, etc. Based on the user's specifications, the server generates a test item list according to the identified elements and specified test content. These test item lists include specific procedures and expected results.

[0072] Next, the server automatically prepares test data from the generated AI model and database. This involves leveraging AI technology for data generation and preparing a dataset optimized for the test case. The test data is organized according to the test item list and provided in an immediately usable state.

[0073] During testing, the server autonomously performs the tests via the terminal. Based on a pre-configured script, the server inputs test data to the target interface and verifies the operation of each operation. Subsequently, the server analyzes the obtained test results and evaluates and reports successful test cases and any problems that occurred.

[0074] As a concrete example, consider testing a bank's online money transfer system. The user selects internal integration testing and specifies testing the money transfer function. The server retrieves screen elements such as "transfer amount" and "transfer button" and generates a test item list based on them. The test includes test cases that verify the correctness of the transfer amount and the operation of the transfer button. The server then prepares transfer test data and conducts the test. The test results are analyzed and fed back to the user, improving the efficiency and accuracy of the test.

[0075] An example of a prompt message could be: "The user wants to perform an internal integration test of the online money transfer function. Please describe the process by which the server scans the UI elements of the target system and automatically generates a test item list."

[0076] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0077] Step 1:

[0078] The server scans the target system's screen interface. Using web scraping techniques, the server analyzes the page's HTML structure and identifies elements such as input fields and buttons. The input is the entire screen's HTML data, and the output is a list of identified elements and their associated metadata. Specifically, the server analyzes the HTML document and extracts the necessary UI elements.

[0079] Step 2:

[0080] The user specifies the type of test using the system's terminal. The user selects options such as UI testing or functional testing from the interface. The input is the type of test selected by the user, and the output is the transmission of that selection to the server. Specifically, this involves the user selecting from a menu and confirming their selection.

[0081] Step 3:

[0082] The server generates a test item list based on the specified test type. The input is the test type selected by the user and metadata of previously retrieved elements. The output is a test item list, containing a list of procedures and expected results for each test case. As a data processing step, the server determines the appropriate test cases by applying rules specific to the test type.

[0083] Step 4:

[0084] The server prepares test data using a generated AI model. The input is a test item list, and the output is the specific dataset required for the test. Specifically, the AI ​​model automatically generates the optimal data for each test case and filters the data based on pre-set conditions.

[0085] Step 5:

[0086] The server autonomously performs the tests via the terminal. The inputs are prepared test data and a test item list, and the output is the result of each test step. Specifically, the server follows a script, inputs test data into the target interface, and records the system's response at each step.

[0087] Step 6:

[0088] The server analyzes the test results and provides feedback to the user. The input is the test execution result, and the output is a report of the analyzed results. As a data calculation, the server aggregates the test results, classifies them into success cases and errors, and creates a report. Specifically, the server uses a particular analysis algorithm to generate feedback based on the obtained data.

[0089] (Application Example 1)

[0090] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0091] In recent years, there has been a growing demand for improved efficiency and accuracy in the complex operational testing of factory equipment. However, current testing processes rely heavily on manual labor, resulting in significant time and cost issues associated with operational verification. Furthermore, quickly identifying malfunctions and defects presents challenges. In this context, there is a need to provide a method for conducting more rapid and accurate operational testing of equipment.

[0092] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0093] In this invention, the server includes means for automatically detecting components of a screen display device and generating related information, means for creating a test procedure table based on the detected components, and means for autonomously performing verification using the prepared test information. This makes it possible to quickly and accurately perform operational tests on factory equipment and efficiently identify faulty parts.

[0094] A "screen display device" refers to equipment or devices used to present visual information. This includes interfaces designed for user operation and confirmation.

[0095] "Components" refer to basic units such as operable buttons, text input fields, and display labels placed on a screen display device. These form the user interface.

[0096] "Related information" refers to additional data generated about the detected components. This may include metadata such as location, size, and function.

[0097] A "test procedure sheet" refers to a document or dataset that details the series of steps required to perform operational tests on factory equipment. It is automatically generated based on the detected components.

[0098] "Test information" refers to the specific data and instructions prepared based on the test procedure sheet and used in the verification process. This information is customized according to the purpose of the test.

[0099] "Autonomously performing verification" refers to a server or system following pre-configured procedures and carrying out the testing process without human intervention.

[0100] "Factory equipment" refers to general equipment such as robots, machines, and systems used in manufacturing. It plays a role in automating various production processes.

[0101] A "defective point" refers to a point in a factory's equipment or system where it does not function as expected or generates an error. These are problems that should be identified through testing.

[0102] According to this invention, the server provides a system for autonomously performing operational tests on factory equipment. First, the server uses a camera mounted on a smartphone or tablet to capture images of the factory equipment's screen display. Using OpenCV as image recognition software, it identifies the elements on the screen and generates related information. This related information includes metadata about the location and function of each element.

[0103] Based on this metadata, the server automatically generates a test procedure table. This table details the steps required for operational testing, providing a foundation for efficiently verifying the operational processes of the relevant factory equipment.

[0104] Furthermore, the server prepares the necessary test information based on the test procedure sheet. At this stage, AWS Lambda is used as a cloud service to create and prepare the test scenario. The test information includes specific input data and control commands.

[0105] The server transmits the prepared test information to the factory equipment via Bluetooth or Wi-Fi and autonomously performs the verification. During testing, the equipment's response is monitored in real time, and the collected data is sent to the server for analysis.

[0106] Once verification is complete, the server analyzes the test results and provides feedback to the user. Important information is displayed on the terminal's screen, and a detailed report is provided, especially regarding any anomalies or malfunctions. This report is analyzed using a generative AI model to generate specific improvement suggestions.

[0107] A concrete example is the operational testing of screw tightening operations in a specific assembly process. The server automatically recognizes screw tightening buttons and confirmation indicators, and generates a test procedure table, significantly improving the efficiency of the actual test. In this way, improvements in both the accuracy and efficiency of the test are expected.

[0108] An example of a prompt statement could be, "Please tell me how to automate the test process and identify faults when a factory machine performs a series of complex operations." Using this prompt statement, a generative AI model can derive detailed test procedures and improvement suggestions.

[0109] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0110] Step 1:

[0111] The terminal uses a camera to capture images of the factory equipment's screen display. It collects image data as input and sends it to a server. It provides clear screen image data as output. This data is used for subsequent image recognition processing.

[0112] Step 2:

[0113] The server uses image recognition software (e.g., OpenCV) to detect screen elements from the input image data. Specifically, the server calculates the position and size of each element and labels them. The output generates a list of the detected elements and their corresponding metadata.

[0114] Step 3:

[0115] The server automatically generates a test procedure table based on the generated metadata. Here, it organizes the components based on a predefined test scenario and assembles the specific steps required for the test. The output is a detailed test procedure table.

[0116] Step 4:

[0117] The server uses cloud services such as AWS Lambda to prepare test information based on the test procedure table. It references the test procedure table as input and generates input data and control commands appropriate for the test target. The output includes the prepared test information.

[0118] Step 5:

[0119] The server transmits test information prepared for the factory equipment via Bluetooth or Wi-Fi. Here, control commands are sent to the equipment as input, initiating operational testing. The equipment's response data is returned to the server in real time as output.

[0120] Step 6:

[0121] The server analyzes the response data from the operational test. Specifically, it automatically checks whether the execution results meet the test conditions. It takes response data as input and generates analyzed test results as output.

[0122] Step 7:

[0123] The user receives feedback on the problems based on the analysis results. The server uses a generative AI model to create a detailed report of the test results, which is displayed on the terminal's screen and includes information such as the locations of anomalies and suggestions for improvement.

[0124] Through these steps, operational testing of factory equipment can be performed efficiently and accurately, enabling the rapid identification and correction of any malfunctions.

[0125] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0126] This invention achieves a more user-friendly testing process by incorporating an emotion engine that recognizes user emotions, in addition to a system that automates the testing process. This system can flexibly modify the testing process by utilizing emotion data to enhance the effectiveness of the tests.

[0127] The system first automatically retrieves elements from the UI screen and generates metadata. Next, the user selects the type of test, and the server automatically generates a test item list based on this. During the item list creation process, the emotion engine starts up. The emotion engine analyzes the user's voice patterns, facial expressions, and operation methods to identify their current emotional state.

[0128] The server automatically adjusts the preparation of test data and the prioritization of test items based on the emotions recognized by the emotion engine. For example, if the system determines that the user is stressed, it reduces the amount and complexity of the test data and selects a test scenario that is less burdensome for the user. Conversely, if the user is relaxed or interested, the system can change the settings to allow for more detailed testing.

[0129] Even during autonomous testing, the emotion engine monitors the user's emotions in real time and provides appropriate feedback. If the user shows signs of frustration or fatigue, the device supports them by displaying encouraging messages or brief explanations of their progress.

[0130] In analyzing test results, the server takes emotional data into account and personalizes feedback. By incorporating emotional elements, users can receive improvement suggestions that are sensitive to their own feelings. Even if a problem occurs, it will be reported in a way that aligns with the user's emotions, thus reducing stress.

[0131] In this way, an intelligent system that enhances user engagement can be realized. The present invention provides an efficient and highly accurate testing process while also being able to respond flexibly to user emotions.

[0132] The following describes the processing flow.

[0133] Step 1:

[0134] The server analyzes the UI screen of the target system and automatically retrieves screen elements. The server identifies interface components such as input fields and buttons, generates metadata related to them, and prepares the foundation for test preparation.

[0135] Step 2:

[0136] The user logs into the system and specifies the type and purpose of the test. For example, they can select unit testing or integration testing, and the scope of the test is determined based on the user's specifications.

[0137] Step 3:

[0138] The emotion engine analyzes the user's emotions in real time. The device captures the user's voice and facial expression data, which the server processes to determine the user's current emotional state.

[0139] Step 4:

[0140] The server automatically generates a test item list based on the emotional information obtained and the metadata of the screen elements. The test content is customized according to the user's emotions; for example, simpler items are prioritized when the user is stressed, while more detailed items are prioritized when the user is concentrating.

[0141] Step 5:

[0142] The server prepares test data based on the test item list. It selects and generates the necessary data while reflecting emotional information, and assembles a dataset for conducting the test.

[0143] Step 6:

[0144] An autonomous test is conducted. The server accesses the system via a terminal and sequentially executes the procedures listed in the checklist. During the operation, the emotion engine continuously monitors the user's emotions, and feedback appropriate to the user's state is displayed on the screen.

[0145] Step 7:

[0146] Once the test is complete, the server analyzes the test results and generates a detailed report that includes emotional information. The results are presented to the user on their device, and emotionally sensitive improvement suggestions are provided as feedback.

[0147] Step 8:

[0148] Based on the information provided, users can readjust the system and test settings. They can rerun the test as needed and continue to optimize the test process with support from the emotion engine.

[0149] (Example 2)

[0150] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0151] Traditional testing systems struggled to provide a flexible testing process that took into account the emotional state of test-takers, and lacked measures to mitigate stress during testing. Furthermore, feedback was typically based solely on test results, lacking a personalized approach that reflected the user's emotions. This resulted in reduced accuracy and efficiency of testing, and insufficient user engagement.

[0152] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0153] In this invention, the server includes means for automatically acquiring elements of the screen interface and generating information, means for generating a list of test items based on the acquired elements, and means for identifying the user's emotional state using emotion recognition technology when creating the list of test items. This makes it possible to provide a flexible testing process that responds to the user's emotional state, enabling highly accurate testing while reducing stress.

[0154] "Screen interface elements" refer to interactive components such as buttons and text fields that are displayed on a user interface.

[0155] "Means for generating information" refers to a mechanism for analyzing elements of the acquired screen interface and automatically generating metadata and other supplementary information.

[0156] "Means for generating a list of test items" refers to a process for automatically creating a list of tasks, questions, and other items for a test based on predefined criteria.

[0157] "Emotion recognition technology" is a technology that analyzes a user's voice, facial expressions, and actions to identify their emotional state.

[0158] "Means for identifying the user's emotional state" refers to a mechanism that uses emotion recognition technology to analyze and understand the user's psychological state and feedback.

[0159] "Means for dynamically adjusting test data" refers to a function that automatically changes the difficulty level and amount of test content based on the user's current emotional state.

[0160] A "means of providing feedback" is a system for conveying information, advice, and improvement measures to the other party, taking into account test results and the user's emotional state.

[0161] The system of this invention automates the testing process performed by the user and flexibly adjusts the process based on the user's emotional state. The system is implemented using the following hardware and software.

[0162] The core of the system consists of a server with powerful data processing and analysis capabilities. The server uses specific interface analysis software to automatically acquire elements of the screen interface. This software has the ability to scan the components on the screen and generate metadata.

[0163] When a user selects the type of test according to the purpose of the test, the server automatically generates a list of test items based on that information. This item generation incorporates a list creation algorithm that instantly generates appropriate items for the selected test content.

[0164] Based on this generated list of test items, the emotion recognition technology begins to operate. The device analyzes the user's voice, facial expressions, and operation patterns using an emotion engine to identify the user's emotional state in real time. For this purpose, speech recognition software and facial recognition algorithms are used.

[0165] The server has the ability to dynamically adjust test data according to the user's emotional state. Specifically, if the user's emotional state indicates stress, the test content is simplified; conversely, if they are relaxed, the test content is made more detailed. This procedure ensures that the test is conducted in a way that is less burdensome for the user.

[0166] Data obtained through real-time emotion recognition is also reflected in the feedback system. The device displays encouraging messages and progress reports to the user at appropriate times, reducing stress during the test.

[0167] For example, when a user tests a new software system, the server can scan UI elements and set up test items to start with simple operations. If the user shows signs of stress, the test content can be further simplified accordingly. Another example of a prompt might be, "Please tell me how to optimize test items and suggest stress reduction measures based on user emotion data."

[0168] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0169] Step 1:

[0170] The server retrieves elements of the screen interface used by the user. The user's interface screen is provided as input, and the server uses interface analysis software to scan UI elements such as buttons and text fields, generating metadata. This metadata is used in a later stage to generate test items.

[0171] Step 2:

[0172] The user selects the type of test. The user specifies the purpose and category of the test as input. The server receives this input and uses a list generation algorithm to automatically generate a list of test items according to the selected test type. The output is a list of test items.

[0173] Step 3:

[0174] The server initiates emotion recognition based on the generated list of test items. Inputs include the user's voice, facial expressions, and action patterns. The terminal analyzes this data through an emotion engine to identify the user's emotional state. As a result, information regarding the current emotional state is output.

[0175] Step 4:

[0176] The server adjusts the test data according to the identified emotional state. The input includes the user's emotional state and a list of test items. The server simplifies the test content for stressed users and elaborates on it for relaxed users. The output of this process is the adjusted test data.

[0177] Step 5:

[0178] The device continuously monitors the user's emotions in real time and provides feedback. Its input is updated emotion data from an emotion engine. The device supports the user by displaying encouraging messages and progress reports when signs of frustration or fatigue are detected. Its output is specific feedback messages for the user.

[0179] Step 6:

[0180] The server analyzes the test results and generates personalized feedback that takes emotional data into account. Test result data and emotional state data are used as input. The server analyzes this data to provide improvement suggestions and issue reports, and even provides positive feedback if the results are unfavorable. The output consists of improvement suggestions and feedback for the user.

[0181] (Application Example 2)

[0182] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0183] In automating the testing process, conventional systems followed a fixed process without considering the user's emotional state, resulting in a heavy burden on users and low satisfaction. Furthermore, in analyzing test results, the inability to provide feedback that reflected user emotions limited the effectiveness of improvement suggestions. To address these challenges, it is necessary to provide a flexible testing process that responds to the user's emotional state and personalized feedback.

[0184] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0185] In this invention, the server includes means for automatically identifying components of the screen interface and generating information data, means for generating a list of test items based on the identified components, and means for analyzing the user's voice characteristics and facial expression data to identify their emotional state and adjust the priority of the test items. This provides an adaptive testing process that responds to the user's emotional state, enabling a less burdensome test for the user and allowing for the provision of personalized feedback that takes emotions into account.

[0186] "Screen interface components" refer to various elements on the display screen that users interact with, including visually represented elements such as buttons, links, and text fields.

[0187] "Information data" refers to metadata generated based on the components of the screen interface, and is used for generating and conducting test items.

[0188] A "test item list" is a list of specific items for conducting a test, constructed based on identified components.

[0189] "User voice characteristics and facial expression data" refers to data on voice and facial expressions collected during interactions with the user, and is used to analyze the user's emotional state.

[0190] "Emotional state" refers to psychological states such as stress, relaxation, and interest, identified based on the analysis of the user's voice characteristics and facial expression data.

[0191] "Adjusting the priority of test items" means changing the order and importance of each item in the list of test items, taking into account the user's emotional state, in order to provide an appropriate test scenario.

[0192] The system for realizing this invention provides a new testing process that improves convenience by recognizing the user's emotional state and dynamically adjusting the testing process. It primarily uses the following hardware and software:

[0193] The server first automatically identifies the components of the screen interface and generates information data based on this. This information data is used to generate a list of test items, and test information is automatically prepared based on the list. Here, the user's voice characteristics and facial expression data are collected, and the emotional state is analyzed in real time using a generating AI model. For this, a standard camera and microphone are required as hardware. For emotion recognition software, an emotion analysis API, for example, is used.

[0194] The device automatically adjusts the priority of test items according to the user's emotional state, providing a test scenario optimized for the user's current psychological condition. If the analysis indicates that the user is stressed, the test is adjusted to reduce the burden; if the user is relaxed, the test can proceed with more detailed testing.

[0195] As a concrete example, suppose a user is beta testing a new application. By analyzing data obtained from the user's camera and microphone, if the system determines that the user is slightly excited, it considers the user to be highly focused and sets the test items to allow them to continue with complex tasks.

[0196] An example of a prompt message used is: "Using the user's facial expression data and voice patterns as input, determine if the user needs rest and generate appropriate relaxation suggestions."

[0197] In this way, the system provides a testing environment that takes user emotions into consideration, and realizes a personalized testing and feedback mechanism.

[0198] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0199] Step 1:

[0200] The server identifies the components of the screen interface and generates informational data. Specifically, it analyzes the UI elements displayed on the screen and collects metadata such as buttons and text fields. This process takes screen capture data as input and outputs informational data about each UI element.

[0201] Step 2:

[0202] The server generates a list of test items based on the generated informational data. It configures the test items from the metadata and determines the initial priority for each item. The input for this step is informational data, and the output is a list of test items.

[0203] Step 3:

[0204] The device collects voice characteristics and facial expression data from the user's camera and microphone. This data is sent to a server in real time, and a generative AI model is used to analyze the user's emotional state. In this step, video and audio data are the inputs, and the output is the classification result of the emotional state.

[0205] Step 4:

[0206] The server adjusts the priority of test items based on the acquired emotional state. If the user is stressed, it prioritizes simplified items to reduce the burden; if relaxed, it sets more complex items. The input is the emotional state and a list of test items, and the output is a list of adjusted test items.

[0207] Step 5:

[0208] The terminal performs tests according to a pre-configured list of test items. It notifies the user of the start and end of the tests and visually displays the progress. In this step, the pre-configured list of test items is the input, and the progress data is the output.

[0209] Step 6:

[0210] The server collects and analyzes test results and provides feedback that takes into account the user's emotional state. This feedback includes suggestions for improvement and messages tailored to the user's situation. The input for this step is test results and emotional data, while the output is personalized feedback.

[0211] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0212] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0213] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0214] [Second Embodiment]

[0215] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0216] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0217] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0218] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0219] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0220] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0221] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0222] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0223] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0224] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0225] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0226] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0227] This invention provides a method for automating and efficiently executing the system testing process. Here, the program's processing is explained in natural language, and the overall operation is illustrated with concrete examples.

[0228] In this invention, the server first automatically acquires the screen elements of the user interface of the target system. The server identifies elements such as input fields and buttons from the screen and generates metadata related to them. This allows the interface information necessary for the testing process to be accumulated.

[0229] Next, the user specifies the type of test, and the server generates a test item list based on the identified elements and the specified test content. This test item list includes the specific procedures and expected results for each test case.

[0230] Subsequently, the server automatically prepares the test data based on the test item list. This includes the actual input data and processes used, and optimized data is provided for each test case.

[0231] Once ready, the server autonomously performs the tests. The server accesses the system through a terminal, inputs data according to the test item list, and performs operations. It also checks the system's response to each operation and evaluates whether the tests were performed as planned.

[0232] After the test, the server analyzes the results and reports successful test cases and problems. The server analyzes the test results as feedback and notifies the user of details of areas for improvement and anomalies. The results displayed on the terminal allow the user to understand the test status and modify the system or test items as needed.

[0233] As a concrete example, consider testing an online money transfer system for a bank. The user selects internal integration testing and specifies testing the money transfer function. The server scans the UI and obtains elements such as "transfer amount" and "transfer button." Based on this, the server generates a test item list and sets up test cases that include verifying the correctness of the transfer amount and checking the operation of the transfer button. Next, the server prepares the transfer test data and conducts the test. After the server checks whether the transfer process is successful and confirms the system's response, it provides feedback to the user with the analyzed results.

[0234] This implementation is expected to improve the overall efficiency and accuracy of system testing.

[0235] The following describes the processing flow.

[0236] Step 1:

[0237] The server retrieves the UI screen of the target system and automatically identifies screen elements using image analysis and DOM analysis. The server extracts elements such as input fields, buttons, and dropdown menus, and generates metadata to record these elements.

[0238] Step 2:

[0239] The user selects the type of test and specifies the test objective (e.g., unit test, integration test). Based on this, the server automatically generates a test item list in conjunction with the screen element data it has analyzed. The test item list includes the specific procedure and expected results for each test case.

[0240] Step 3:

[0241] The server prepares the necessary test data for each case listed in the test item sheet. Using the system's database and existing datasets, it automatically generates datasets, including dummy data as needed, to provide the optimal data for each test case.

[0242] Step 4:

[0243] The server autonomously conducts tests using prepared test data. It accesses the system via a terminal and automatically performs operations according to the test item list. Specifically, it performs actions such as entering appropriate values ​​into screen elements and clicking designated buttons.

[0244] Step 5:

[0245] During the test, the server records the system's response to each operation and monitors the results in real time. It collects logs to compare the expected results with the actual responses and determine success or failure.

[0246] Step 6:

[0247] The server analyzes the test results and compiles the success rate and errors for all test cases. A detailed test report is provided to the terminal, allowing the user to review the test results. If a defect is found, details and improvement suggestions are included as feedback.

[0248] (Example 1)

[0249] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0250] Traditional system testing processes require manual verification of screen interfaces and setting of test items, which is time-consuming and labor-intensive, and has limitations in terms of test accuracy and efficiency. Furthermore, because the preparation and optimization of test data relied on manual processes, a system that could flexibly adapt to changes in test content was needed. To address these problems, it is necessary to automate the testing process while improving accuracy and efficiency.

[0251] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0252] In this invention, the server includes means for automatically identifying elements of a screen interface and generating metadata, means for generating a test item list based on the identified elements, and means for automatically preparing test data based on the test item list. This reduces manual work and enables automation and improved accuracy of the testing process.

[0253] A "screen interface" is a visual component of a computer program that a user can operate visually.

[0254] An "element" refers to an individual component or part within a screen interface that has a specific function, including input fields and buttons.

[0255] "Metadata" is data that describes information about an element, and may include the element's ID, class name, and location information.

[0256] A "test item list" is a table that lists the specific procedures required for the test and the expected results.

[0257] "Test data" refers to input information such as specific numerical values ​​and strings necessary to carry out the test items.

[0258] "Autonomously" refers to the ability to act automatically based on one's own judgment without waiting for external instructions.

[0259] "Test results" refers to the record of the outcomes and conditions obtained through the testing process.

[0260] "Feedback" refers to evaluations, notifications, or reports of areas for improvement provided based on test results.

[0261] "Optimization" refers to adjusting the resources of a system or data to utilize them in the most efficient way possible in order to achieve a specific objective.

[0262] "Access control" refers to the procedure of controlling access rights to a system or parts of it, allowing only authorized operations to be performed.

[0263] This invention provides a method for automating and efficiently executing the testing process of a system. The system consists of the interaction of a server, a terminal, and a user.

[0264] First, the server scans the target system's screen interface using web scraping or automation tools (e.g., Selenium). The server automatically identifies elements such as input fields and buttons from the scanned screen and generates metadata about them. This aggregates the interface information necessary for the testing process.

[0265] The user accesses the system through a terminal and specifies the type of test. For example, they can choose from UI tests, functional tests, stress tests, etc. Based on the user's specifications, the server generates a test item list according to the identified elements and specified test content. These test item lists include specific procedures and expected results.

[0266] Next, the server automatically prepares test data from the generated AI model and database. This involves leveraging AI technology for data generation and preparing a dataset optimized for the test case. The test data is organized according to the test item list and provided in an immediately usable state.

[0267] During testing, the server autonomously performs the tests via the terminal. Based on a pre-configured script, the server inputs test data to the target interface and verifies the operation of each operation. Subsequently, the server analyzes the obtained test results and evaluates and reports successful test cases and any problems that occurred.

[0268] As a concrete example, consider testing a bank's online money transfer system. The user selects internal integration testing and specifies testing the money transfer function. The server retrieves screen elements such as "transfer amount" and "transfer button" and generates a test item list based on them. The test includes test cases that verify the correctness of the transfer amount and the operation of the transfer button. The server then prepares transfer test data and conducts the test. The test results are analyzed and fed back to the user, improving the efficiency and accuracy of the test.

[0269] An example of a prompt message could be: "The user wants to perform an internal integration test of the online money transfer function. Please describe the process by which the server scans the UI elements of the target system and automatically generates a test item list."

[0270] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0271] Step 1:

[0272] The server scans the target system's screen interface. Using web scraping techniques, the server analyzes the page's HTML structure and identifies elements such as input fields and buttons. The input is the entire screen's HTML data, and the output is a list of identified elements and their associated metadata. Specifically, the server analyzes the HTML document and extracts the necessary UI elements.

[0273] Step 2:

[0274] The user specifies the type of test using the system's terminal. The user selects options such as UI testing or functional testing from the interface. The input is the type of test selected by the user, and the output is the transmission of that selection to the server. Specifically, this involves the user selecting from a menu and confirming their selection.

[0275] Step 3:

[0276] The server generates a test item list based on the specified test type. The input is the test type selected by the user and metadata of previously retrieved elements. The output is a test item list, containing a list of procedures and expected results for each test case. As a data processing step, the server determines the appropriate test cases by applying rules specific to the test type.

[0277] Step 4:

[0278] The server prepares test data using a generated AI model. The input is a test item list, and the output is the specific dataset required for the test. Specifically, the AI ​​model automatically generates the optimal data for each test case and filters the data based on pre-set conditions.

[0279] Step 5:

[0280] The server autonomously conducts tests via the terminal. The input is the prepared test data and the test item table, and the output is the execution result of each test step. As a specific operation, the server inputs test data to the target interface according to the script, operates it, and records the system response at each step.

[0281] Step 6:

[0282] The server analyzes the test results and provides feedback to the user. The input is the execution result of the test, and the output is a report of the analyzed results. As data calculation, the server aggregates the test results, classifies the successful cases and errors, and creates a report. The specific operation is that the server uses a specific analysis algorithm to generate feedback based on the obtained data.

[0283] (Application Example 1)

[0284] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0285] In recent years, there has been a demand for improving the efficiency and accuracy in the complex operation tests of factory equipment. However, the current test process relies heavily on manual work, and the time and cost required for operation confirmation have become problems. There is also an issue that it is difficult to quickly identify malfunction and defective parts. In such a situation, it is necessary to provide a method for more quickly and accurately conducting the operation test of the equipment.

[0286] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0287] In this invention, the server includes means for automatically detecting components of a screen display device and generating related information, means for creating a test procedure table based on the detected components, and means for autonomously performing verification using the prepared test information. This makes it possible to quickly and accurately perform operational tests on factory equipment and efficiently identify faulty parts.

[0288] A "screen display device" refers to equipment or devices used to present visual information. This includes interfaces designed for user operation and confirmation.

[0289] "Components" refer to basic units such as operable buttons, text input fields, and display labels placed on a screen display device. These form the user interface.

[0290] "Related information" refers to additional data generated about the detected components. This may include metadata such as location, size, and function.

[0291] A "test procedure sheet" refers to a document or dataset that details the series of steps required to perform operational tests on factory equipment. It is automatically generated based on the detected components.

[0292] "Test information" refers to the specific data and instructions prepared based on the test procedure sheet and used in the verification process. This information is customized according to the purpose of the test.

[0293] "Autonomously performing verification" refers to a server or system following pre-configured procedures and carrying out the testing process without human intervention.

[0294] "Factory equipment" refers to general equipment such as robots, machines, and systems used in manufacturing. It plays a role in automating various production processes.

[0295] A "defective point" refers to a point in a factory's equipment or system where it does not function as expected or generates an error. These are problems that should be identified through testing.

[0296] According to this invention, the server provides a system for autonomously performing operational tests on factory equipment. First, the server uses a camera mounted on a smartphone or tablet to capture images of the factory equipment's screen display. Using OpenCV as image recognition software, it identifies the elements on the screen and generates related information. This related information includes metadata about the location and function of each element.

[0297] Based on this metadata, the server automatically generates a test procedure table. This table details the steps required for operational testing, providing a foundation for efficiently verifying the operational processes of the relevant factory equipment.

[0298] Furthermore, the server prepares the necessary test information based on the test procedure sheet. At this stage, AWS Lambda is used as a cloud service to create and prepare the test scenario. The test information includes specific input data and control commands.

[0299] The server transmits the prepared test information to the factory equipment via Bluetooth or Wi-Fi and autonomously performs the verification. During testing, the equipment's response is monitored in real time, and the collected data is sent to the server for analysis.

[0300] Once verification is complete, the server analyzes the test results and provides feedback to the user. Important information is displayed on the terminal's screen, and a detailed report is provided, especially regarding any anomalies or malfunctions. This report is analyzed using a generative AI model to generate specific improvement suggestions.

[0301] As a specific example, there is an operation test of the screwing work in a specific assembly process. The server can automatically recognize the screwing button and the confirmation display, and by generating a test procedure table, the efficiency of the actual test can be greatly improved. In this way, it is expected that the accuracy and efficiency of the test will be improved.

[0302] As an example of a prompt sentence, "Please tell me how to automate the test process and identify defective parts when the factory equipment performs a series of complex operations" can be considered. Using this prompt sentence, detailed test procedures and improvement suggestions can be derived by the generative AI model.

[0303] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0304] Step 1:

[0305] The terminal captures the screen display device of the factory equipment with a camera. It collects image data as input and transmits this to the server. It provides clear screen image data as output. This data is used for subsequent image recognition processing.

[0306] Step 2:

[0307] The server uses image recognition software (e.g., OpenCV) to detect the components of the screen from the input image data. As a specific operation, the server calculates the position and size of each component and labels them. As output, it generates a list of the detected components and the corresponding metadata.

[0308] Step 3:

[0309] The server automatically generates a test procedure table based on the generated metadata. Here, the components are sorted based on a pre-defined test scenario, and the specific steps required for the test are assembled. As output, a detailed test procedure table is obtained.

[0310] Step 4:

[0311] The server uses cloud services such as AWS Lambda to prepare test information based on the test procedure table. It references the test procedure table as input and generates input data and control commands appropriate for the test target. The output includes the prepared test information.

[0312] Step 5:

[0313] The server transmits test information prepared for the factory equipment via Bluetooth or Wi-Fi. Here, control commands are sent to the equipment as input, initiating operational testing. The equipment's response data is returned to the server in real time as output.

[0314] Step 6:

[0315] The server analyzes the response data from the operational test. Specifically, it automatically checks whether the execution results meet the test conditions. It takes the response data as input and generates the analyzed test results as output.

[0316] Step 7:

[0317] The user receives feedback on the problems based on the analysis results. The server uses a generative AI model to create a detailed report of the test results, which is displayed on the terminal's screen and includes information such as the locations of anomalies and suggestions for improvement.

[0318] Through these steps, operational testing of factory equipment can be performed efficiently and accurately, enabling the rapid identification and correction of any malfunctions.

[0319] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0320] This invention achieves a more user-friendly testing process by incorporating an emotion engine that recognizes user emotions, in addition to a system that automates the testing process. This system can flexibly modify the testing process by utilizing emotion data to enhance the effectiveness of the tests.

[0321] The system first automatically retrieves elements from the UI screen and generates metadata. Next, the user selects the type of test, and the server automatically generates a test item list based on this. During the item list creation process, the emotion engine starts up. The emotion engine analyzes the user's voice patterns, facial expressions, and operation methods to identify their current emotional state.

[0322] The server automatically adjusts the preparation of test data and the prioritization of test items based on the emotions recognized by the emotion engine. For example, if the system determines that the user is stressed, it reduces the amount and complexity of the test data and selects a test scenario that is less burdensome for the user. Conversely, if the user is relaxed or interested, the system can change the settings to allow for more detailed testing.

[0323] Even during autonomous testing, the emotion engine monitors the user's emotions in real time and provides appropriate feedback. If the user shows signs of frustration or fatigue, the device supports them by displaying encouraging messages or brief explanations of their progress.

[0324] In analyzing test results, the server takes emotional data into account and personalizes feedback. By incorporating emotional elements, users can receive improvement suggestions that are sensitive to their own feelings. Even if a problem occurs, it will be reported in a way that aligns with the user's emotions, thus reducing stress.

[0325] In this way, an intelligent system that enhances user engagement can be realized. The present invention provides an efficient and highly accurate testing process while also being able to respond flexibly to user emotions.

[0326] The following describes the processing flow.

[0327] Step 1:

[0328] The server analyzes the UI screen of the target system and automatically retrieves screen elements. The server identifies interface components such as input fields and buttons, generates metadata related to them, and prepares the foundation for test preparation.

[0329] Step 2:

[0330] The user logs into the system and specifies the type and purpose of the test. For example, they can select unit testing or integration testing, and the scope of the test is determined based on the user's specifications.

[0331] Step 3:

[0332] The emotion engine analyzes the user's emotions in real time. The device captures the user's voice and facial expression data, which the server processes to determine the user's current emotional state.

[0333] Step 4:

[0334] The server automatically generates a test item list based on the emotional information obtained and the metadata of the screen elements. The test content is customized according to the user's emotions; for example, simpler items are prioritized when the user is stressed, while more detailed items are prioritized when the user is concentrating.

[0335] Step 5:

[0336] The server prepares test data based on the test item list. It selects and generates the necessary data while reflecting emotional information, and assembles a dataset for conducting the test.

[0337] Step 6:

[0338] An autonomous test is conducted. The server accesses the system via a terminal and sequentially executes the procedures listed in the checklist. During the operation, the emotion engine continuously monitors the user's emotions, and feedback appropriate to the user's state is displayed on the screen.

[0339] Step 7:

[0340] Once the test is complete, the server analyzes the test results and generates a detailed report that includes emotional information. The results are presented to the user on their device, and emotionally sensitive improvement suggestions are provided as feedback.

[0341] Step 8:

[0342] Based on the information provided, users can readjust the system and test settings. They can rerun the test as needed and continue to optimize the test process with support from the emotion engine.

[0343] (Example 2)

[0344] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0345] Traditional testing systems struggled to provide a flexible testing process that took into account the emotional state of test-takers, and lacked measures to mitigate stress during testing. Furthermore, feedback was typically based solely on test results, lacking a personalized approach that reflected the user's emotions. This resulted in reduced accuracy and efficiency of testing, and insufficient user engagement.

[0346] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0347] In this invention, the server includes means for automatically acquiring elements of the screen interface and generating information, means for generating a list of test items based on the acquired elements, and means for identifying the user's emotional state using emotion recognition technology when creating the list of test items. This makes it possible to provide a flexible testing process that responds to the user's emotional state, enabling highly accurate testing while reducing stress.

[0348] "Screen interface elements" refer to interactive components such as buttons and text fields that are displayed on a user interface.

[0349] "Means for generating information" refers to a mechanism for analyzing elements of the acquired screen interface and automatically generating metadata and other supplementary information.

[0350] "Means for generating a list of test items" refers to a process for automatically creating a list of tasks, questions, and other items for a test based on predefined criteria.

[0351] "Emotion recognition technology" is a technology that analyzes a user's voice, facial expressions, and actions to identify their emotional state.

[0352] "Means for identifying the user's emotional state" refers to a mechanism that uses emotion recognition technology to analyze and understand the user's psychological state and feedback.

[0353] "Means for dynamically adjusting test data" refers to a function that automatically changes the difficulty level and amount of test content based on the user's current emotional state.

[0354] A "means of providing feedback" is a system for conveying information, advice, and improvement measures to the other party, taking into account test results and the user's emotional state.

[0355] The system of this invention automates the testing process performed by the user and flexibly adjusts the process based on the user's emotional state. The system is implemented using the following hardware and software.

[0356] The core of the system consists of a server with powerful data processing and analysis capabilities. The server uses specific interface analysis software to automatically acquire elements of the screen interface. This software has the ability to scan the components on the screen and generate metadata.

[0357] When a user selects the type of test according to the purpose of the test, the server automatically generates a list of test items based on that information. This item generation incorporates a list creation algorithm that instantly generates appropriate items for the selected test content.

[0358] Based on this generated list of test items, the emotion recognition technology begins to operate. The device analyzes the user's voice, facial expressions, and operation patterns using an emotion engine to identify the user's emotional state in real time. For this purpose, speech recognition software and facial recognition algorithms are used.

[0359] The server has the ability to dynamically adjust test data according to the user's emotional state. Specifically, if the user's emotional state indicates stress, the test content is simplified; conversely, if they are relaxed, the test content is made more detailed. This procedure ensures that the test is conducted in a way that is less burdensome for the user.

[0360] Data obtained through real-time emotion recognition is also reflected in the feedback system. The device displays encouraging messages and progress reports to the user at appropriate times, reducing stress during the test.

[0361] For example, when a user tests a new software system, the server can scan UI elements and set up test items to start with simple operations. If the user shows signs of stress, the test content can be further simplified accordingly. Another example of a prompt might be, "Please tell me how to optimize test items and suggest stress reduction measures based on user emotion data."

[0362] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0363] Step 1:

[0364] The server retrieves elements of the screen interface used by the user. The user's interface screen is provided as input, and the server uses interface analysis software to scan UI elements such as buttons and text fields, generating metadata. This metadata is used in a later stage to generate test items.

[0365] Step 2:

[0366] The user selects the type of test. The user specifies the purpose and category of the test as input. The server receives this input and uses a list generation algorithm to automatically generate a list of test items according to the selected test type. The output is a list of test items.

[0367] Step 3:

[0368] The server initiates emotion recognition based on the generated list of test items. Inputs include the user's voice, facial expressions, and action patterns. The terminal analyzes this data through an emotion engine to identify the user's emotional state. As a result, information regarding the current emotional state is output.

[0369] Step 4:

[0370] The server adjusts the test data according to the identified emotional state. The input includes the user's emotional state and a list of test items. The server simplifies the test content for stressed users and elaborates on it for relaxed users. The output of this process is the adjusted test data.

[0371] Step 5:

[0372] The device continuously monitors the user's emotions in real time and provides feedback. Its input is updated emotion data from an emotion engine. The device supports the user by displaying encouraging messages and progress reports when signs of frustration or fatigue are detected. Its output is specific feedback messages for the user.

[0373] Step 6:

[0374] The server analyzes the test results and generates personalized feedback that takes emotional data into account. Test result data and emotional state data are used as input. The server analyzes this data to provide improvement suggestions and issue reports, and even provides positive feedback if the results are unfavorable. The output consists of improvement suggestions and feedback for the user.

[0375] (Application Example 2)

[0376] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0377] In automating the testing process, conventional systems followed a fixed process without considering the user's emotional state, resulting in a heavy burden on users and low satisfaction. Furthermore, in analyzing test results, the inability to provide feedback that reflected user emotions limited the effectiveness of improvement suggestions. To address these challenges, it is necessary to provide a flexible testing process that responds to the user's emotional state and personalized feedback.

[0378] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0379] In this invention, the server includes means for automatically identifying components of the screen interface and generating information data, means for generating a list of test items based on the identified components, and means for analyzing the user's voice characteristics and facial expression data to identify their emotional state and adjust the priority of the test items. This provides an adaptive testing process that responds to the user's emotional state, enabling a less burdensome test for the user and allowing for the provision of personalized feedback that takes emotions into account.

[0380] "Screen interface components" refer to various elements on the display screen that users interact with, including visually represented elements such as buttons, links, and text fields.

[0381] "Information data" refers to metadata generated based on the components of the screen interface, and is used for generating and conducting test items.

[0382] A "test item list" is a list of specific items for conducting a test, constructed based on identified components.

[0383] "User voice characteristics and facial expression data" refers to data on voice and facial expressions collected during interactions with the user, and is used to analyze the user's emotional state.

[0384] "Emotional state" refers to psychological states such as stress, relaxation, and interest, which are identified based on the analysis of the user's voice characteristics and facial expression data.

[0385] "Adjusting the priority of test items" means changing the order and importance of each item in the list of test items, taking into account the user's emotional state, in order to provide an appropriate test scenario.

[0386] The system for realizing this invention provides a new testing process that improves convenience by recognizing the user's emotional state and dynamically adjusting the testing process. It primarily uses the following hardware and software:

[0387] The server first automatically identifies the components of the screen interface and generates information data based on this. This information data is used to generate a list of test items, and test information is automatically prepared based on the list. Here, the user's voice characteristics and facial expression data are collected, and the emotional state is analyzed in real time using a generating AI model. For this, a standard camera and microphone are required as hardware. For emotion recognition software, an emotion analysis API, for example, is used.

[0388] The device automatically adjusts the priority of test items according to the user's emotional state, providing a test scenario optimized for the user's current psychological condition. If the analysis indicates that the user is stressed, the test is adjusted to reduce the burden; if the user is relaxed, the test can proceed with more detailed testing.

[0389] As a concrete example, suppose a user is beta testing a new application. By analyzing data obtained from the user's camera and microphone, if the system determines that the user is slightly excited, it considers the user to be highly focused and sets the test items to allow them to continue with complex tasks.

[0390] An example of a prompt message used is: "Using the user's facial expression data and voice patterns as input, determine if the user needs rest and generate appropriate relaxation suggestions."

[0391] In this way, the system provides a testing environment that takes user emotions into consideration, and realizes a personalized testing and feedback mechanism.

[0392] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0393] Step 1:

[0394] The server identifies the components of the screen interface and generates informational data. Specifically, it analyzes the UI elements displayed on the screen and collects metadata such as buttons and text fields. This process takes screen capture data as input and outputs informational data about each UI element.

[0395] Step 2:

[0396] The server generates a list of test items based on the generated informational data. It configures the test items from the metadata and determines the initial priority for each item. The input for this step is informational data, and the output is a list of test items.

[0397] Step 3:

[0398] The device collects voice characteristics and facial expression data from the user's camera and microphone. This data is sent to a server in real time, and a generative AI model is used to analyze the user's emotional state. In this step, video and audio data are the inputs, and the output is the classification result of the emotional state.

[0399] Step 4:

[0400] The server adjusts the priority of test items based on the acquired emotional state. If the user is stressed, it prioritizes simplified items to reduce the burden; if relaxed, it sets more complex items. The input is the emotional state and a list of test items, and the output is a list of adjusted test items.

[0401] Step 5:

[0402] The terminal performs tests according to a pre-configured list of test items. It notifies the user of the start and end of the tests and visually displays the progress. In this step, the pre-configured list of test items is the input, and the progress data is the output.

[0403] Step 6:

[0404] The server collects and analyzes test results and provides feedback that takes into account the user's emotional state. This feedback includes suggestions for improvement and messages tailored to the user's situation. The input for this step is test results and emotional data, while the output is personalized feedback.

[0405] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0406] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0407] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0408] [Third Embodiment]

[0409] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0410] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0411] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0412] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0413] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0414] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0415] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0416] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0417] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0418] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0419] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0420] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0421] This invention provides a method for automating and efficiently executing the system testing process. Here, the program's processing is explained in natural language, and the overall operation is illustrated with concrete examples.

[0422] In this invention, the server first automatically acquires the screen elements of the user interface of the target system. The server identifies elements such as input fields and buttons from the screen and generates metadata related to them. This allows the interface information necessary for the testing process to be accumulated.

[0423] Next, the user specifies the type of test, and the server generates a test item list based on the identified elements and the specified test content. This test item list includes the specific procedures and expected results for each test case.

[0424] Subsequently, the server automatically prepares the test data based on the test item list. This includes the actual input data and processes used, and optimized data is provided for each test case.

[0425] Once ready, the server autonomously performs the tests. The server accesses the system through a terminal, inputs data according to the test item list, and performs operations. It also checks the system's response to each operation and evaluates whether the tests were performed as planned.

[0426] After the test, the server analyzes the results and reports successful test cases and problems. The server analyzes the test results as feedback and notifies the user of details of areas for improvement and anomalies. The results displayed on the terminal allow the user to understand the test status and modify the system or test items as needed.

[0427] As a concrete example, consider testing an online money transfer system for a bank. The user selects internal integration testing and specifies testing the money transfer function. The server scans the UI and obtains elements such as "transfer amount" and "transfer button." Based on this, the server generates a test item list and sets up test cases that include verifying the correctness of the transfer amount and checking the operation of the transfer button. Next, the server prepares the transfer test data and conducts the test. After the server checks whether the transfer process is successful and confirms the system's response, it provides feedback to the user with the analyzed results.

[0428] This implementation is expected to improve the overall efficiency and accuracy of system testing.

[0429] The following describes the processing flow.

[0430] Step 1:

[0431] The server retrieves the UI screen of the target system and automatically identifies screen elements using image analysis and DOM analysis. The server extracts elements such as input fields, buttons, and dropdown menus, and generates metadata to record these elements.

[0432] Step 2:

[0433] The user selects the type of test and specifies the test objective (e.g., unit test, integration test). Based on this, the server automatically generates a test item list in conjunction with the screen element data it has analyzed. The test item list includes the specific procedure and expected results for each test case.

[0434] Step 3:

[0435] The server prepares the necessary test data for each case listed in the test item sheet. Using the system's database and existing datasets, it automatically generates datasets, including dummy data as needed, to provide the optimal data for each test case.

[0436] Step 4:

[0437] The server autonomously conducts tests using prepared test data. It accesses the system via a terminal and automatically performs operations according to the test item list. Specifically, it performs actions such as entering appropriate values ​​into screen elements and clicking designated buttons.

[0438] Step 5:

[0439] During the test, the server records the system's response to each operation and monitors the results in real time. It collects logs to compare the expected results with the actual responses and determine success or failure.

[0440] Step 6:

[0441] The server analyzes the test results and compiles the success rate and errors for all test cases. A detailed test report is provided to the terminal, allowing the user to review the test results. If a defect is found, details and improvement suggestions are included as feedback.

[0442] (Example 1)

[0443] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0444] Traditional system testing processes require manual verification of screen interfaces and setting of test items, which is time-consuming and labor-intensive, and has limitations in terms of test accuracy and efficiency. Furthermore, because the preparation and optimization of test data relied on manual processes, a system that could flexibly adapt to changes in test content was needed. To address these problems, it is necessary to automate the testing process while improving accuracy and efficiency.

[0445] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0446] In this invention, the server includes means for automatically identifying elements of a screen interface and generating metadata, means for generating a test item list based on the identified elements, and means for automatically preparing test data based on the test item list. This reduces manual work and enables automation and improved accuracy of the testing process.

[0447] A "screen interface" is a visual component of a computer program that a user can operate visually.

[0448] An "element" refers to an individual component or part within a screen interface that has a specific function, including input fields and buttons.

[0449] "Metadata" is data that describes information about an element, and may include the element's ID, class name, and location information.

[0450] A "test item list" is a table that lists the specific procedures required for the test and the expected results.

[0451] "Test data" refers to input information such as specific numerical values ​​and strings necessary to carry out the test items.

[0452] "Autonomously" refers to the ability to act automatically based on one's own judgment without waiting for external instructions.

[0453] "Test results" refers to the record of the outcomes and conditions obtained through the testing process.

[0454] "Feedback" refers to evaluations, notifications, or reports of areas for improvement provided based on test results.

[0455] "Optimization" refers to adjusting the resources of a system or data to utilize them in the most efficient way possible in order to achieve a specific objective.

[0456] "Access control" refers to the procedure of controlling access rights to a system or parts of it, allowing only authorized operations to be performed.

[0457] This invention provides a method for automating and efficiently executing the testing process of a system. The system consists of the interaction of a server, a terminal, and a user.

[0458] First, the server scans the target system's screen interface using web scraping or automation tools (e.g., Selenium). The server automatically identifies elements such as input fields and buttons from the scanned screen and generates metadata about them. This aggregates the interface information necessary for the testing process.

[0459] The user accesses the system through a terminal and specifies the type of test. For example, they can choose from UI tests, functional tests, stress tests, etc. Based on the user's specifications, the server generates a test item list according to the identified elements and specified test content. These test item lists include specific procedures and expected results.

[0460] Next, the server automatically prepares test data from the generated AI model and database. This involves leveraging AI technology for data generation and preparing a dataset optimized for the test case. The test data is organized according to the test item list and provided in an immediately usable state.

[0461] During testing, the server autonomously performs the tests via the terminal. Based on a pre-configured script, the server inputs test data to the target interface and verifies the operation of each operation. Subsequently, the server analyzes the obtained test results and evaluates and reports successful test cases and any problems that occurred.

[0462] As a concrete example, consider testing a bank's online money transfer system. The user selects internal integration testing and specifies testing the money transfer function. The server retrieves screen elements such as "transfer amount" and "transfer button" and generates a test item list based on them. The test includes test cases that verify the correctness of the transfer amount and the operation of the transfer button. The server then prepares transfer test data and conducts the test. The test results are analyzed and fed back to the user, improving the efficiency and accuracy of the test.

[0463] An example of a prompt message could be: "The user wants to perform an internal integration test of the online money transfer function. Please describe the process by which the server scans the UI elements of the target system and automatically generates a test item list."

[0464] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0465] Step 1:

[0466] The server scans the target system's screen interface. Using web scraping techniques, the server analyzes the page's HTML structure and identifies elements such as input fields and buttons. The input is the entire screen's HTML data, and the output is a list of identified elements and their associated metadata. Specifically, the server analyzes the HTML document and extracts the necessary UI elements.

[0467] Step 2:

[0468] The user specifies the type of test using the system's terminal. The user selects options such as UI testing or functional testing from the interface. The input is the type of test selected by the user, and the output is the transmission of that selection to the server. Specifically, this involves the user selecting from a menu and confirming their selection.

[0469] Step 3:

[0470] The server generates a test item list based on the specified test type. The input is the test type selected by the user and metadata of previously retrieved elements. The output is a test item list, containing a list of procedures and expected results for each test case. As a data processing step, the server determines the appropriate test cases by applying rules specific to the test type.

[0471] Step 4:

[0472] The server prepares test data using a generated AI model. The input is a test item list, and the output is the specific dataset required for the test. Specifically, the AI ​​model automatically generates the optimal data for each test case and filters the data based on pre-set conditions.

[0473] Step 5:

[0474] The server autonomously performs the tests via the terminal. The inputs are prepared test data and a test item list, and the output is the result of each test step. Specifically, the server follows a script, inputs test data into the target interface, and records the system's response at each step.

[0475] Step 6:

[0476] The server analyzes the test results and provides feedback to the user. The input is the test execution result, and the output is a report of the analyzed results. As a data calculation, the server aggregates the test results, classifies them into success cases and errors, and creates a report. Specifically, the server uses a particular analysis algorithm to generate feedback based on the obtained data.

[0477] (Application Example 1)

[0478] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0479] In recent years, there has been a growing demand for improved efficiency and accuracy in the complex operational testing of factory equipment. However, current testing processes rely heavily on manual labor, resulting in significant time and cost issues associated with operational verification. Furthermore, quickly identifying malfunctions and defects presents challenges. In this context, there is a need to provide a method for conducting more rapid and accurate operational testing of equipment.

[0480] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0481] In this invention, the server includes means for automatically detecting components of a screen display device and generating related information, means for creating a test procedure table based on the detected components, and means for autonomously performing verification using the prepared test information. This makes it possible to quickly and accurately perform operational tests on factory equipment and efficiently identify faulty parts.

[0482] A "screen display device" refers to equipment or devices used to present visual information. This includes interfaces designed for user operation and confirmation.

[0483] "Components" refer to basic units such as operable buttons, text input fields, and display labels placed on a screen display device. These form the user interface.

[0484] "Related information" refers to additional data generated about the detected components. This may include metadata such as location, size, and function.

[0485] A "test procedure sheet" refers to a document or dataset that details the series of steps required to perform operational tests on factory equipment. It is automatically generated based on the detected components.

[0486] "Test information" refers to the specific data and instructions prepared based on the test procedure sheet and used in the verification process. This information is customized according to the purpose of the test.

[0487] "Autonomously performing verification" refers to a server or system following pre-configured procedures and carrying out the testing process without human intervention.

[0488] "Factory equipment" refers to general equipment such as robots, machines, and systems used in manufacturing. It plays a role in automating various production processes.

[0489] A "defective point" refers to a point in a factory's equipment or system where it does not function as expected or generates an error. These are problems that should be identified through testing.

[0490] According to this invention, the server provides a system for autonomously performing operational tests on factory equipment. First, the server uses a camera mounted on a smartphone or tablet to capture images of the factory equipment's screen display. Using OpenCV as image recognition software, it identifies the elements on the screen and generates related information. This related information includes metadata about the location and function of each element.

[0491] Based on this metadata, the server automatically generates a test procedure table. This table details the steps required for operational testing, providing a foundation for efficiently verifying the operational processes of the relevant factory equipment.

[0492] Furthermore, the server prepares the necessary test information based on the test procedure sheet. At this stage, AWS Lambda is used as a cloud service to create and prepare the test scenario. The test information includes specific input data and control commands.

[0493] The server transmits the prepared test information to the factory equipment via Bluetooth or Wi-Fi and autonomously performs the verification. During testing, the equipment's response is monitored in real time, and the collected data is sent to the server for analysis.

[0494] Once verification is complete, the server analyzes the test results and provides feedback to the user. Important information is displayed on the terminal's screen, and a detailed report is provided, especially regarding any anomalies or malfunctions. This report is analyzed using a generative AI model to generate specific improvement suggestions.

[0495] A concrete example is the operational testing of screw tightening operations in a specific assembly process. The server automatically recognizes screw tightening buttons and confirmation indicators, and generates a test procedure table, significantly improving the efficiency of the actual test. In this way, improvements in both the accuracy and efficiency of the test are expected.

[0496] An example of a prompt statement could be, "Please tell me how to automate the test process and identify faults when a factory machine performs a series of complex operations." Using this prompt statement, a generative AI model can derive detailed test procedures and improvement suggestions.

[0497] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0498] Step 1:

[0499] The terminal uses a camera to capture images of the factory equipment's screen display. It collects image data as input and sends it to a server. It provides clear screen image data as output. This data is used for subsequent image recognition processing.

[0500] Step 2:

[0501] The server uses image recognition software (e.g., OpenCV) to detect screen elements from the input image data. Specifically, the server calculates the position and size of each element and labels them. The output generates a list of the detected elements and their corresponding metadata.

[0502] Step 3:

[0503] The server automatically generates a test procedure table based on the generated metadata. Here, it organizes the components based on a predefined test scenario and assembles the specific steps required for the test. The output is a detailed test procedure table.

[0504] Step 4:

[0505] The server uses cloud services such as AWS Lambda to prepare test information based on the test procedure table. It references the test procedure table as input and generates input data and control commands appropriate for the test target. The output includes the prepared test information.

[0506] Step 5:

[0507] The server transmits test information prepared for the factory equipment via Bluetooth or Wi-Fi. Here, control commands are sent to the equipment as input, initiating operational testing. The equipment's response data is returned to the server in real time as output.

[0508] Step 6:

[0509] The server analyzes the response data from the operational test. Specifically, it automatically checks whether the execution results meet the test conditions. It takes the response data as input and generates the analyzed test results as output.

[0510] Step 7:

[0511] The user receives feedback on the problems based on the analysis results. The server uses a generative AI model to create a detailed report of the test results, which is displayed on the terminal's screen and includes information such as the locations of anomalies and suggestions for improvement.

[0512] Through these steps, operational testing of factory equipment can be performed efficiently and accurately, enabling the rapid identification and correction of any malfunctions.

[0513] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0514] This invention achieves a more user-friendly testing process by incorporating an emotion engine that recognizes user emotions, in addition to a system that automates the testing process. This system can flexibly modify the testing process by utilizing emotion data to enhance the effectiveness of the tests.

[0515] The system first automatically retrieves elements from the UI screen and generates metadata. Next, the user selects the type of test, and the server automatically generates a test item list based on this. During the item list creation process, the emotion engine starts up. The emotion engine analyzes the user's voice patterns, facial expressions, and operation methods to identify their current emotional state.

[0516] The server automatically adjusts the preparation of test data and the prioritization of test items based on the emotions recognized by the emotion engine. For example, if the system determines that the user is stressed, it reduces the amount and complexity of the test data and selects a test scenario that is less burdensome for the user. Conversely, if the user is relaxed or interested, the system can change the settings to allow for more detailed testing.

[0517] Even during autonomous testing, the emotion engine monitors the user's emotions in real time and provides appropriate feedback. If the user shows signs of frustration or fatigue, the device supports them by displaying encouraging messages or brief explanations of their progress.

[0518] In analyzing test results, the server takes emotional data into account and personalizes feedback. By incorporating emotional elements, users can receive improvement suggestions that are sensitive to their own feelings. Even if a problem occurs, it will be reported in a way that aligns with the user's emotions, thus reducing stress.

[0519] In this way, an intelligent system that enhances user engagement can be realized. The present invention provides an efficient and highly accurate testing process while also being able to respond flexibly to user emotions.

[0520] The following describes the processing flow.

[0521] Step 1:

[0522] The server analyzes the UI screen of the target system and automatically retrieves screen elements. The server identifies interface components such as input fields and buttons, generates metadata related to them, and prepares the foundation for test preparation.

[0523] Step 2:

[0524] The user logs into the system and specifies the type and purpose of the test. For example, they can select unit testing or integration testing, and the scope of the test is determined based on the user's specifications.

[0525] Step 3:

[0526] The emotion engine analyzes the user's emotions in real time. The device captures the user's voice and facial expression data, which the server processes to determine the user's current emotional state.

[0527] Step 4:

[0528] The server automatically generates a test item list based on the emotional information obtained and the metadata of the screen elements. The test content is customized according to the user's emotions; for example, simpler items are prioritized when the user is stressed, while more detailed items are prioritized when the user is concentrating.

[0529] Step 5:

[0530] The server prepares test data based on the test item list. It selects and generates the necessary data while reflecting emotional information, and assembles a dataset for conducting the test.

[0531] Step 6:

[0532] An autonomous test is conducted. The server accesses the system via a terminal and sequentially executes the procedures listed in the checklist. During the operation, the emotion engine continuously monitors the user's emotions, and feedback appropriate to the user's state is displayed on the screen.

[0533] Step 7:

[0534] Once the test is complete, the server analyzes the test results and generates a detailed report that includes emotional information. The results are presented to the user on their device, and emotionally sensitive improvement suggestions are provided as feedback.

[0535] Step 8:

[0536] Based on the information provided, users can readjust the system and test settings. They can rerun the test as needed and continue to optimize the test process with support from the emotion engine.

[0537] (Example 2)

[0538] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0539] Traditional testing systems struggled to provide a flexible testing process that took into account the emotional state of test-takers, and lacked measures to mitigate stress during testing. Furthermore, feedback was typically based solely on test results, lacking a personalized approach that reflected the user's emotions. This resulted in reduced accuracy and efficiency of testing, and insufficient user engagement.

[0540] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0541] In this invention, the server includes means for automatically acquiring elements of the screen interface and generating information, means for generating a list of test items based on the acquired elements, and means for identifying the user's emotional state using emotion recognition technology when creating the list of test items. This makes it possible to provide a flexible testing process that responds to the user's emotional state, enabling highly accurate testing while reducing stress.

[0542] "Screen interface elements" refer to interactive components such as buttons and text fields that are displayed on a user interface.

[0543] "Means for generating information" refers to a mechanism for analyzing elements of the acquired screen interface and automatically generating metadata and other supplementary information.

[0544] "Means for generating a list of test items" refers to a process for automatically creating a list of tasks, questions, and other items for a test based on predefined criteria.

[0545] "Emotion recognition technology" is a technology that analyzes a user's voice, facial expressions, and actions to identify their emotional state.

[0546] "Means for identifying the user's emotional state" refers to a mechanism that uses emotion recognition technology to analyze and understand the user's psychological state and feedback.

[0547] "Means for dynamically adjusting test data" refers to a function that automatically changes the difficulty level and amount of test content based on the user's current emotional state.

[0548] A "means of providing feedback" is a system for conveying information, advice, and improvement measures to the other party, taking into account test results and the user's emotional state.

[0549] The system of this invention automates the testing process performed by the user and flexibly adjusts the process based on the user's emotional state. The system is implemented using the following hardware and software.

[0550] The core of the system consists of a server with powerful data processing and analysis capabilities. The server uses specific interface analysis software to automatically acquire elements of the screen interface. This software has the ability to scan the components on the screen and generate metadata.

[0551] When a user selects the type of test according to the purpose of the test, the server automatically generates a list of test items based on that information. This item generation incorporates a list creation algorithm that instantly generates appropriate items for the selected test content.

[0552] Based on this generated list of test items, the emotion recognition technology begins to operate. The device analyzes the user's voice, facial expressions, and operation patterns using an emotion engine to identify the user's emotional state in real time. For this purpose, speech recognition software and facial recognition algorithms are used.

[0553] The server has the ability to dynamically adjust test data according to the user's emotional state. Specifically, if the user's emotional state indicates stress, the test content is simplified; conversely, if they are relaxed, the test content is made more detailed. This procedure ensures that the test is conducted in a way that is less burdensome for the user.

[0554] Data obtained through real-time emotion recognition is also reflected in the feedback system. The device displays encouraging messages and progress reports to the user at appropriate times, reducing stress during the test.

[0555] For example, when a user tests a new software system, the server can scan UI elements and set up test items to start with simple operations. If the user shows signs of stress, the test content can be further simplified accordingly. Another example of a prompt might be, "Please tell me how to optimize test items and suggest stress reduction measures based on user emotion data."

[0556] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0557] Step 1:

[0558] The server retrieves elements of the screen interface used by the user. The user's interface screen is provided as input, and the server uses interface analysis software to scan UI elements such as buttons and text fields, generating metadata. This metadata is used in a later stage to generate test items.

[0559] Step 2:

[0560] The user selects the type of test. The user specifies the purpose and category of the test as input. The server receives this input and uses a list generation algorithm to automatically generate a list of test items according to the selected test type. The output is a list of test items.

[0561] Step 3:

[0562] The server initiates emotion recognition based on the generated list of test items. Inputs include the user's voice, facial expressions, and action patterns. The terminal analyzes this data through an emotion engine to identify the user's emotional state. As a result, information regarding the current emotional state is output.

[0563] Step 4:

[0564] The server adjusts the test data according to the identified emotional state. The input includes the user's emotional state and a list of test items. The server simplifies the test content for stressed users and elaborates on it for relaxed users. The output of this process is the adjusted test data.

[0565] Step 5:

[0566] The device continuously monitors the user's emotions in real time and provides feedback. Its input is updated emotion data from an emotion engine. The device supports the user by displaying encouraging messages and progress reports when signs of frustration or fatigue are detected. Its output is specific feedback messages for the user.

[0567] Step 6:

[0568] The server analyzes the test results and generates personalized feedback that takes emotional data into account. Test result data and emotional state data are used as input. The server analyzes this data to provide improvement suggestions and issue reports, and even provides positive feedback if the results are unfavorable. The output consists of improvement suggestions and feedback for the user.

[0569] (Application Example 2)

[0570] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0571] In automating the testing process, conventional systems followed a fixed process without considering the user's emotional state, resulting in a heavy burden on users and low satisfaction. Furthermore, in analyzing test results, the inability to provide feedback that reflected user emotions limited the effectiveness of improvement suggestions. To address these challenges, it is necessary to provide a flexible testing process that responds to the user's emotional state and personalized feedback.

[0572] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0573] In this invention, the server includes means for automatically identifying components of the screen interface and generating information data, means for generating a list of test items based on the identified components, and means for analyzing the user's voice characteristics and facial expression data to identify their emotional state and adjust the priority of the test items. This provides an adaptive testing process that responds to the user's emotional state, enabling a less burdensome test for the user and allowing for the provision of personalized feedback that takes emotions into account.

[0574] "Screen interface components" refer to various elements on the display screen that users interact with, including visually represented elements such as buttons, links, and text fields.

[0575] "Information data" refers to metadata generated based on the components of the screen interface, and is used for generating and conducting test items.

[0576] A "test item list" is a list of specific items for conducting a test, constructed based on identified components.

[0577] "User voice characteristics and facial expression data" refers to data on voice and facial expressions collected during interactions with the user, and is used to analyze the user's emotional state.

[0578] "Emotional state" refers to psychological states such as stress, relaxation, and interest, which are identified based on the analysis of the user's voice characteristics and facial expression data.

[0579] "Adjusting the priority of test items" means changing the order and importance of each item in the list of test items, taking into account the user's emotional state, in order to provide an appropriate test scenario.

[0580] The system for realizing this invention provides a new testing process that improves convenience by recognizing the user's emotional state and dynamically adjusting the testing process. It primarily uses the following hardware and software:

[0581] The server first automatically identifies the components of the screen interface and generates information data based on this. This information data is used to generate a list of test items, and test information is automatically prepared based on the list. Here, the user's voice characteristics and facial expression data are collected, and the emotional state is analyzed in real time using a generating AI model. For this, a standard camera and microphone are required as hardware. For emotion recognition software, an emotion analysis API, for example, is used.

[0582] The device automatically adjusts the priority of test items according to the user's emotional state, providing a test scenario optimized for the user's current psychological condition. If the analysis indicates that the user is stressed, the test is adjusted to reduce the burden; if the user is relaxed, the test can proceed with more detailed testing.

[0583] As a concrete example, suppose a user is beta testing a new application. By analyzing data obtained from the user's camera and microphone, if the system determines that the user is slightly excited, it considers the user to be highly focused and sets the test items to allow them to continue with complex tasks.

[0584] An example of a prompt message used is: "Using the user's facial expression data and voice patterns as input, determine if the user needs rest and generate appropriate relaxation suggestions."

[0585] In this way, the system provides a testing environment that takes user emotions into consideration, and realizes a personalized testing and feedback mechanism.

[0586] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0587] Step 1:

[0588] The server identifies the components of the screen interface and generates informational data. Specifically, it analyzes the UI elements displayed on the screen and collects metadata such as buttons and text fields. This process takes screen capture data as input and outputs informational data about each UI element.

[0589] Step 2:

[0590] The server generates a list of test items based on the generated informational data. It configures the test items from the metadata and determines the initial priority for each item. The input for this step is informational data, and the output is a list of test items.

[0591] Step 3:

[0592] The device collects voice characteristics and facial expression data from the user's camera and microphone. This data is sent to a server in real time, and a generative AI model is used to analyze the user's emotional state. In this step, video and audio data are the inputs, and the output is the classification result of the emotional state.

[0593] Step 4:

[0594] The server adjusts the priority of test items based on the acquired emotional state. If the user is stressed, it prioritizes simplified items to reduce the burden; if relaxed, it sets more complex items. The input is the emotional state and a list of test items, and the output is a list of adjusted test items.

[0595] Step 5:

[0596] The terminal performs tests according to a pre-configured list of test items. It notifies the user of the start and end of the tests and visually displays the progress. In this step, the pre-configured list of test items is the input, and the progress data is the output.

[0597] Step 6:

[0598] The server collects and analyzes test results and provides feedback that takes into account the user's emotional state. This feedback includes suggestions for improvement and messages tailored to the user's situation. The input for this step is test results and emotional data, while the output is personalized feedback.

[0599] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0600] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0601] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0602] [Fourth Embodiment]

[0603] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0604] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0605] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0606] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0607] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0608] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0609] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0610] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0611] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0612] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0613] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0614] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0615] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0616] This invention provides a method for automating and efficiently executing the system testing process. Here, the program's processing is explained in natural language, and the overall operation is illustrated with concrete examples.

[0617] In this invention, the server first automatically acquires the screen elements of the user interface of the target system. The server identifies elements such as input fields and buttons from the screen and generates metadata related to them. This allows the interface information necessary for the testing process to be accumulated.

[0618] Next, the user specifies the type of test, and the server generates a test item list based on the identified elements and the specified test content. This test item list includes the specific procedures and expected results for each test case.

[0619] Subsequently, the server automatically prepares the test data based on the test item list. This includes the actual input data and processes used, and optimized data is provided for each test case.

[0620] Once ready, the server autonomously performs the tests. The server accesses the system through a terminal, inputs data according to the test item list, and performs operations. It also checks the system's response to each operation and evaluates whether the tests were performed as planned.

[0621] After the test, the server analyzes the results and reports successful test cases and problems. The server analyzes the test results as feedback and notifies the user of details of areas for improvement and anomalies. The results displayed on the terminal allow the user to understand the test status and modify the system or test items as needed.

[0622] As a concrete example, consider testing an online money transfer system for a bank. The user selects internal integration testing and specifies testing the money transfer function. The server scans the UI and obtains elements such as "transfer amount" and "transfer button." Based on this, the server generates a test item list and sets up test cases that include verifying the correctness of the transfer amount and checking the operation of the transfer button. Next, the server prepares the transfer test data and conducts the test. After the server checks whether the transfer process is successful and confirms the system's response, it provides feedback to the user with the analyzed results.

[0623] This implementation is expected to improve the overall efficiency and accuracy of system testing.

[0624] The following describes the processing flow.

[0625] Step 1:

[0626] The server retrieves the UI screen of the target system and automatically identifies screen elements using image analysis and DOM analysis. The server extracts elements such as input fields, buttons, and dropdown menus, and generates metadata to record these elements.

[0627] Step 2:

[0628] The user selects the type of test and specifies the test objective (e.g., unit test, integration test). Based on this, the server automatically generates a test item list in conjunction with the screen element data it has analyzed. The test item list includes the specific procedure and expected results for each test case.

[0629] Step 3:

[0630] The server prepares the necessary test data for each case listed in the test item sheet. Using the system's database and existing datasets, it automatically generates datasets, including dummy data as needed, to provide the optimal data for each test case.

[0631] Step 4:

[0632] The server autonomously conducts tests using prepared test data. It accesses the system via a terminal and automatically performs operations according to the test item list. Specifically, it performs actions such as entering appropriate values ​​into screen elements and clicking designated buttons.

[0633] Step 5:

[0634] During the test, the server records the system's response to each operation and monitors the results in real time. It collects logs to compare the expected results with the actual responses and determine success or failure.

[0635] Step 6:

[0636] The server analyzes the test results and compiles the success rate and errors for all test cases. A detailed test report is provided to the terminal, allowing the user to review the test results. If a defect is found, details and improvement suggestions are included as feedback.

[0637] (Example 1)

[0638] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0639] Traditional system testing processes require manual verification of screen interfaces and setting of test items, which is time-consuming and labor-intensive, and has limitations in terms of test accuracy and efficiency. Furthermore, because the preparation and optimization of test data relied on manual processes, a system that could flexibly adapt to changes in test content was needed. To address these problems, it is necessary to automate the testing process while improving accuracy and efficiency.

[0640] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0641] In this invention, the server includes means for automatically identifying elements of a screen interface and generating metadata, means for generating a test item list based on the identified elements, and means for automatically preparing test data based on the test item list. This reduces manual work and enables automation and improved accuracy of the testing process.

[0642] A "screen interface" is a visual component of a computer program that a user can operate visually.

[0643] An "element" refers to an individual component or part within a screen interface that has a specific function, including input fields and buttons.

[0644] "Metadata" is data that describes information about an element, and may include the element's ID, class name, and location information.

[0645] A "test item list" is a table that lists the specific procedures required for the test and the expected results.

[0646] "Test data" refers to input information such as specific numerical values ​​and strings necessary to carry out the test items.

[0647] "Autonomously" refers to the ability to act automatically based on one's own judgment without waiting for external instructions.

[0648] "Test results" refers to the record of the outcomes and conditions obtained through the testing process.

[0649] "Feedback" refers to evaluations, notifications, or reports of areas for improvement provided based on test results.

[0650] "Optimization" refers to adjusting the resources of a system or data to utilize them in the most efficient way possible in order to achieve a specific objective.

[0651] "Access control" refers to the procedure of controlling access rights to a system or parts of it, allowing only authorized operations to be performed.

[0652] This invention provides a method for automating and efficiently executing the testing process of a system. The system consists of the interaction of a server, a terminal, and a user.

[0653] First, the server scans the target system's screen interface using web scraping or automation tools (e.g., Selenium). The server automatically identifies elements such as input fields and buttons from the scanned screen and generates metadata about them. This aggregates the interface information necessary for the testing process.

[0654] The user accesses the system through a terminal and specifies the type of test. For example, they can choose from UI tests, functional tests, stress tests, etc. Based on the user's specifications, the server generates a test item list according to the identified elements and specified test content. These test item lists include specific procedures and expected results.

[0655] Next, the server automatically prepares test data from the generated AI model and database. This involves leveraging AI technology for data generation and preparing a dataset optimized for the test case. The test data is organized according to the test item list and provided in an immediately usable state.

[0656] During testing, the server autonomously performs the tests via the terminal. Based on a pre-configured script, the server inputs test data to the target interface and verifies the operation of each operation. Subsequently, the server analyzes the obtained test results and evaluates and reports successful test cases and any problems that occurred.

[0657] As a concrete example, consider testing a bank's online money transfer system. The user selects internal integration testing and specifies testing the money transfer function. The server retrieves screen elements such as "transfer amount" and "transfer button" and generates a test item list based on them. The test includes test cases that verify the correctness of the transfer amount and the operation of the transfer button. The server then prepares transfer test data and conducts the test. The test results are analyzed and fed back to the user, improving the efficiency and accuracy of the test.

[0658] An example of a prompt message could be: "The user wants to perform an internal integration test of the online money transfer function. Please describe the process by which the server scans the UI elements of the target system and automatically generates a test item list."

[0659] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0660] Step 1:

[0661] The server scans the target system's screen interface. Using web scraping techniques, the server analyzes the page's HTML structure and identifies elements such as input fields and buttons. The input is the entire screen's HTML data, and the output is a list of identified elements and their associated metadata. Specifically, the server analyzes the HTML document and extracts the necessary UI elements.

[0662] Step 2:

[0663] The user specifies the type of test using the system's terminal. The user selects options such as UI testing or functional testing from the interface. The input is the type of test selected by the user, and the output is the transmission of that selection to the server. Specifically, this involves the user selecting from a menu and confirming their selection.

[0664] Step 3:

[0665] The server generates a test item list based on the specified test type. The input is the test type selected by the user and metadata of previously retrieved elements. The output is a test item list, containing a list of procedures and expected results for each test case. As a data processing step, the server determines the appropriate test cases by applying rules specific to the test type.

[0666] Step 4:

[0667] The server prepares test data using a generated AI model. The input is a test item list, and the output is the specific dataset required for the test. Specifically, the AI ​​model automatically generates the optimal data for each test case and filters the data based on pre-set conditions.

[0668] Step 5:

[0669] The server autonomously performs the tests via the terminal. The inputs are prepared test data and a test item list, and the output is the result of each test step. Specifically, the server follows a script, inputs test data into the target interface, and records the system's response at each step.

[0670] Step 6:

[0671] The server analyzes the test results and provides feedback to the user. The input is the test execution result, and the output is a report of the analyzed results. As a data calculation, the server aggregates the test results, classifies them into success cases and errors, and creates a report. Specifically, the server uses a particular analysis algorithm to generate feedback based on the obtained data.

[0672] (Application Example 1)

[0673] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0674] In recent years, there has been a growing demand for improved efficiency and accuracy in the complex operational testing of factory equipment. However, current testing processes rely heavily on manual labor, resulting in significant time and cost issues associated with operational verification. Furthermore, quickly identifying malfunctions and defects presents challenges. In this context, there is a need to provide a method for conducting more rapid and accurate operational testing of equipment.

[0675] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0676] In this invention, the server includes means for automatically detecting components of a screen display device and generating related information, means for creating a test procedure table based on the detected components, and means for autonomously performing verification using the prepared test information. This makes it possible to quickly and accurately perform operational tests on factory equipment and efficiently identify faulty parts.

[0677] A "screen display device" refers to equipment or devices used to present visual information. This includes interfaces designed for user operation and confirmation.

[0678] "Components" refer to basic units such as operable buttons, text input fields, and display labels placed on a screen display device. These form the user interface.

[0679] "Related information" refers to additional data generated about the detected components. This may include metadata such as location, size, and function.

[0680] A "test procedure sheet" refers to a document or dataset that details the series of steps required to perform operational tests on factory equipment. It is automatically generated based on the detected components.

[0681] "Test information" refers to the specific data and instructions prepared based on the test procedure sheet and used in the verification process. This information is customized according to the purpose of the test.

[0682] "Autonomously performing verification" refers to a server or system following pre-configured procedures and carrying out the testing process without human intervention.

[0683] "Factory equipment" refers to general equipment such as robots, machines, and systems used in manufacturing. It plays a role in automating various production processes.

[0684] A "defective point" refers to a point in a factory's equipment or system where it does not function as expected or generates an error. These are problems that should be identified through testing.

[0685] According to this invention, the server provides a system for autonomously performing operational tests on factory equipment. First, the server uses a camera mounted on a smartphone or tablet to capture images of the factory equipment's screen display. Using OpenCV as image recognition software, it identifies the elements on the screen and generates related information. This related information includes metadata about the location and function of each element.

[0686] Based on this metadata, the server automatically generates a test procedure table. This table details the steps required for operational testing, providing a foundation for efficiently verifying the operational processes of the relevant factory equipment.

[0687] Furthermore, the server prepares the necessary test information based on the test procedure sheet. At this stage, AWS Lambda is used as a cloud service to create and prepare the test scenario. The test information includes specific input data and control commands.

[0688] The server transmits the prepared test information to the factory equipment via Bluetooth or Wi-Fi and autonomously performs the verification. During testing, the equipment's response is monitored in real time, and the collected data is sent to the server for analysis.

[0689] Once verification is complete, the server analyzes the test results and provides feedback to the user. Important information is displayed on the terminal's screen, and a detailed report is provided, especially regarding any anomalies or malfunctions. This report is analyzed using a generative AI model to generate specific improvement suggestions.

[0690] A concrete example is the operational testing of screw tightening operations in a specific assembly process. The server automatically recognizes screw tightening buttons and confirmation indicators, and generates a test procedure table, significantly improving the efficiency of the actual test. In this way, improvements in both the accuracy and efficiency of the test are expected.

[0691] An example of a prompt statement could be, "Please tell me how to automate the test process and identify faults when a factory machine performs a series of complex operations." Using this prompt statement, a generative AI model can derive detailed test procedures and improvement suggestions.

[0692] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0693] Step 1:

[0694] The terminal uses a camera to capture images of the factory equipment's screen display. It collects image data as input and sends it to a server. It provides clear screen image data as output. This data is used for subsequent image recognition processing.

[0695] Step 2:

[0696] The server uses image recognition software (e.g., OpenCV) to detect screen elements from the input image data. Specifically, the server calculates the position and size of each element and labels them. The output generates a list of the detected elements and their corresponding metadata.

[0697] Step 3:

[0698] The server automatically generates a test procedure table based on the generated metadata. Here, it organizes the components based on a predefined test scenario and assembles the specific steps required for the test. The output is a detailed test procedure table.

[0699] Step 4:

[0700] The server uses cloud services such as AWS Lambda to prepare test information based on the test procedure table. It references the test procedure table as input and generates input data and control commands appropriate for the test target. The output includes the prepared test information.

[0701] Step 5:

[0702] The server transmits test information prepared for the factory equipment via Bluetooth or Wi-Fi. Here, control commands are sent to the equipment as input, initiating operational testing. The equipment's response data is returned to the server in real time as output.

[0703] Step 6:

[0704] The server analyzes the response data from the operational test. Specifically, it automatically checks whether the execution results meet the test conditions. It takes the response data as input and generates the analyzed test results as output.

[0705] Step 7:

[0706] The user receives feedback on the problems based on the analysis results. The server uses a generative AI model to create a detailed report of the test results, which is displayed on the terminal's screen and includes information such as the locations of anomalies and suggestions for improvement.

[0707] Through these steps, operational testing of factory equipment can be performed efficiently and accurately, enabling the rapid identification and correction of any malfunctions.

[0708] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0709] This invention achieves a more user-friendly testing process by incorporating an emotion engine that recognizes user emotions, in addition to a system that automates the testing process. This system can flexibly modify the testing process by utilizing emotion data to enhance the effectiveness of the tests.

[0710] The system first automatically retrieves elements from the UI screen and generates metadata. Next, the user selects the type of test, and the server automatically generates a test item list based on this. During the item list creation process, the emotion engine starts up. The emotion engine analyzes the user's voice patterns, facial expressions, and operation methods to identify their current emotional state.

[0711] The server automatically adjusts the preparation of test data and the prioritization of test items based on the emotions recognized by the emotion engine. For example, if the system determines that the user is stressed, it reduces the amount and complexity of the test data and selects a test scenario that is less burdensome for the user. Conversely, if the user is relaxed or interested, the system can change the settings to allow for more detailed testing.

[0712] Even during autonomous testing, the emotion engine monitors the user's emotions in real time and provides appropriate feedback. If the user shows signs of frustration or fatigue, the device supports them by displaying encouraging messages or brief explanations of their progress.

[0713] In analyzing test results, the server takes emotional data into account and personalizes feedback. By incorporating emotional elements, users can receive improvement suggestions that are sensitive to their own feelings. Even if a problem occurs, it will be reported in a way that aligns with the user's emotions, thus reducing stress.

[0714] In this way, an intelligent system that enhances user engagement can be realized. The present invention provides an efficient and highly accurate testing process while also being able to respond flexibly to user emotions.

[0715] The following describes the processing flow.

[0716] Step 1:

[0717] The server analyzes the UI screen of the target system and automatically retrieves screen elements. The server identifies interface components such as input fields and buttons, generates metadata related to them, and prepares the foundation for test preparation.

[0718] Step 2:

[0719] The user logs into the system and specifies the type and purpose of the test. For example, they can select unit testing or integration testing, and the scope of the test is determined based on the user's specifications.

[0720] Step 3:

[0721] The emotion engine analyzes the user's emotions in real time. The device captures the user's voice and facial expression data, which the server processes to determine the user's current emotional state.

[0722] Step 4:

[0723] The server automatically generates a test item list based on the emotional information obtained and the metadata of the screen elements. The test content is customized according to the user's emotions; for example, simpler items are prioritized when the user is stressed, while more detailed items are prioritized when the user is concentrating.

[0724] Step 5:

[0725] The server prepares test data based on the test item list. It selects and generates the necessary data while reflecting emotional information, and assembles a dataset for conducting the test.

[0726] Step 6:

[0727] An autonomous test is conducted. The server accesses the system via a terminal and sequentially executes the procedures listed in the checklist. During the operation, the emotion engine continuously monitors the user's emotions, and feedback appropriate to the user's state is displayed on the screen.

[0728] Step 7:

[0729] Once the test is complete, the server analyzes the test results and generates a detailed report that includes emotional information. The results are presented to the user on their device, and emotionally sensitive improvement suggestions are provided as feedback.

[0730] Step 8:

[0731] Based on the information provided, users can readjust the system and test settings. They can rerun the test as needed and continue to optimize the test process with support from the emotion engine.

[0732] (Example 2)

[0733] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0734] Traditional testing systems struggled to provide a flexible testing process that took into account the emotional state of test-takers, and lacked measures to mitigate stress during testing. Furthermore, feedback was typically based solely on test results, lacking a personalized approach that reflected the user's emotions. This resulted in reduced accuracy and efficiency of testing, and insufficient user engagement.

[0735] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0736] In this invention, the server includes means for automatically acquiring elements of the screen interface and generating information, means for generating a list of test items based on the acquired elements, and means for identifying the user's emotional state using emotion recognition technology when creating the list of test items. This makes it possible to provide a flexible testing process that responds to the user's emotional state, enabling highly accurate testing while reducing stress.

[0737] "Screen interface elements" refer to interactive components such as buttons and text fields that are displayed on a user interface.

[0738] "Means for generating information" refers to a mechanism for analyzing elements of the acquired screen interface and automatically generating metadata and other supplementary information.

[0739] "Means for generating a list of test items" refers to a process for automatically creating a list of tasks, questions, and other items for a test based on predefined criteria.

[0740] "Emotion recognition technology" is a technology that analyzes a user's voice, facial expressions, and actions to identify their emotional state.

[0741] "Means for identifying the user's emotional state" refers to a mechanism that uses emotion recognition technology to analyze and understand the user's psychological state and feedback.

[0742] "Means for dynamically adjusting test data" refers to a function that automatically changes the difficulty level and amount of test content based on the user's current emotional state.

[0743] A "means of providing feedback" is a system for conveying information, advice, and improvement measures to the other party, taking into account test results and the user's emotional state.

[0744] The system of this invention automates the testing process performed by the user and flexibly adjusts the process based on the user's emotional state. The system is implemented using the following hardware and software.

[0745] The core of the system consists of a server with powerful data processing and analysis capabilities. The server uses specific interface analysis software to automatically acquire elements of the screen interface. This software has the ability to scan the components on the screen and generate metadata.

[0746] When a user selects the type of test according to the purpose of the test, the server automatically generates a list of test items based on that information. This item generation incorporates a list creation algorithm that instantly generates appropriate items for the selected test content.

[0747] Based on this generated list of test items, the emotion recognition technology begins to operate. The device analyzes the user's voice, facial expressions, and operation patterns using an emotion engine to identify the user's emotional state in real time. For this purpose, speech recognition software and facial recognition algorithms are used.

[0748] The server has the ability to dynamically adjust test data according to the user's emotional state. Specifically, if the user's emotional state indicates stress, the test content is simplified; conversely, if they are relaxed, the test content is made more detailed. This procedure ensures that the test is conducted in a way that is less burdensome for the user.

[0749] Data obtained through real-time emotion recognition is also reflected in the feedback system. The device displays encouraging messages and progress reports to the user at appropriate times, reducing stress during the test.

[0750] For example, when a user tests a new software system, the server can scan UI elements and set up test items to start with simple operations. If the user shows signs of stress, the test content can be further simplified accordingly. Another example of a prompt might be, "Please tell me how to optimize test items and suggest stress reduction measures based on user emotion data."

[0751] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0752] Step 1:

[0753] The server retrieves elements of the screen interface used by the user. The user's interface screen is provided as input, and the server uses interface analysis software to scan UI elements such as buttons and text fields, generating metadata. This metadata is used in a later stage to generate test items.

[0754] Step 2:

[0755] The user selects the type of test. The user specifies the purpose and category of the test as input. The server receives this input and uses a list generation algorithm to automatically generate a list of test items according to the selected test type. The output is a list of test items.

[0756] Step 3:

[0757] The server initiates emotion recognition based on the generated list of test items. Inputs include the user's voice, facial expressions, and action patterns. The terminal analyzes this data through an emotion engine to identify the user's emotional state. As a result, information regarding the current emotional state is output.

[0758] Step 4:

[0759] The server adjusts the test data according to the identified emotional state. The input includes the user's emotional state and a list of test items. The server simplifies the test content for stressed users and elaborates on it for relaxed users. The output of this process is the adjusted test data.

[0760] Step 5:

[0761] The device continuously monitors the user's emotions in real time and provides feedback. Its input is updated emotion data from an emotion engine. The device supports the user by displaying encouraging messages and progress reports when signs of frustration or fatigue are detected. Its output is specific feedback messages for the user.

[0762] Step 6:

[0763] The server analyzes the test results and generates personalized feedback that takes emotional data into account. Test result data and emotional state data are used as input. The server analyzes this data to provide improvement suggestions and issue reports, and even provides positive feedback if the results are unfavorable. The output consists of improvement suggestions and feedback for the user.

[0764] (Application Example 2)

[0765] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0766] In automating the testing process, conventional systems followed a fixed process without considering the user's emotional state, resulting in a heavy burden on users and low satisfaction. Furthermore, in analyzing test results, the inability to provide feedback that reflected user emotions limited the effectiveness of improvement suggestions. To address these challenges, it is necessary to provide a flexible testing process that responds to the user's emotional state and personalized feedback.

[0767] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0768] In this invention, the server includes means for automatically identifying components of the screen interface and generating information data, means for generating a list of test items based on the identified components, and means for analyzing the user's voice characteristics and facial expression data to identify their emotional state and adjust the priority of the test items. This provides an adaptive testing process that responds to the user's emotional state, enabling a less burdensome test for the user and allowing for the provision of personalized feedback that takes emotions into account.

[0769] "Screen interface components" refer to various elements on the display screen that users interact with, including visually represented elements such as buttons, links, and text fields.

[0770] "Information data" refers to metadata generated based on the components of the screen interface, and is used for generating and conducting test items.

[0771] A "test item list" is a list of specific items for conducting a test, constructed based on identified components.

[0772] "User voice characteristics and facial expression data" refers to data on voice and facial expressions collected during interactions with the user, and is used to analyze the user's emotional state.

[0773] "Emotional state" refers to psychological states such as stress, relaxation, and interest, which are identified based on the analysis of the user's voice characteristics and facial expression data.

[0774] "Adjusting the priority of test items" means changing the order and importance of each item in the list of test items, taking into account the user's emotional state, in order to provide an appropriate test scenario.

[0775] The system for realizing this invention provides a new testing process that improves convenience by recognizing the user's emotional state and dynamically adjusting the testing process. It primarily uses the following hardware and software:

[0776] The server first automatically identifies the components of the screen interface and generates information data based on this. This information data is used to generate a list of test items, and test information is automatically prepared based on the list. Here, the user's voice characteristics and facial expression data are collected, and the emotional state is analyzed in real time using a generating AI model. For this, a standard camera and microphone are required as hardware. For emotion recognition software, an emotion analysis API, for example, is used.

[0777] The device automatically adjusts the priority of test items according to the user's emotional state, providing a test scenario optimized for the user's current psychological condition. If the analysis indicates that the user is stressed, the test is adjusted to reduce the burden; if the user is relaxed, the test can proceed with more detailed testing.

[0778] As a concrete example, suppose a user is beta testing a new application. By analyzing data obtained from the user's camera and microphone, if the system determines that the user is slightly excited, it considers the user to be highly focused and sets the test items to allow them to continue with complex tasks.

[0779] An example of a prompt message used is: "Using the user's facial expression data and voice patterns as input, determine if the user needs rest and generate appropriate relaxation suggestions."

[0780] In this way, the system provides a testing environment that takes user emotions into consideration, and realizes a personalized testing and feedback mechanism.

[0781] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0782] Step 1:

[0783] The server identifies the components of the screen interface and generates informational data. Specifically, it analyzes the UI elements displayed on the screen and collects metadata such as buttons and text fields. This process takes screen capture data as input and outputs informational data about each UI element.

[0784] Step 2:

[0785] The server generates a list of test items based on the generated informational data. It configures the test items from the metadata and determines the initial priority for each item. The input for this step is informational data, and the output is a list of test items.

[0786] Step 3:

[0787] The device collects voice characteristics and facial expression data from the user's camera and microphone. This data is sent to a server in real time, and a generative AI model is used to analyze the user's emotional state. In this step, video and audio data are the inputs, and the output is the classification result of the emotional state.

[0788] Step 4:

[0789] The server adjusts the priority of test items based on the acquired emotional state. If the user is stressed, it prioritizes simplified items to reduce the burden; if relaxed, it sets more complex items. The input is the emotional state and a list of test items, and the output is a list of adjusted test items.

[0790] Step 5:

[0791] The terminal performs tests according to a pre-configured list of test items. It notifies the user of the start and end of the tests and visually displays the progress. In this step, the pre-configured list of test items is the input, and the progress data is the output.

[0792] Step 6:

[0793] The server collects and analyzes test results and provides feedback that takes into account the user's emotional state. This feedback includes suggestions for improvement and messages tailored to the user's situation. The input for this step is test results and emotional data, while the output is personalized feedback.

[0794] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0795] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0796] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0797] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0798] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0799] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0800] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0801] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0802] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0803] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0804] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0805] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0806] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0807] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0808] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0809] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0810] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0811] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0812] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0813] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0814] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0815] The following is further disclosed regarding the embodiments described above.

[0816] (Claim 1)

[0817] A means for automatically identifying elements of a screen interface and generating metadata,

[0818] Means for generating a test item list based on identified elements,

[0819] A means of automatically preparing test data based on a test item list,

[0820] A means of autonomously conducting tests using prepared test data,

[0821] A means of automatically analyzing test results and providing feedback,

[0822] A system that includes this.

[0823] (Claim 2)

[0824] The system according to claim 1, which customizes the test item list based on the type of test specified.

[0825] (Claim 3)

[0826] The system according to claim 1, which automatically notifies details of anomalies based on test results and provides improvement suggestions.

[0827] "Example 1"

[0828] (Claim 1)

[0829] A means for automatically identifying elements of a screen interface and generating metadata,

[0830] Means for generating a test item list based on identified elements,

[0831] A means of automatically preparing test data based on a test item list,

[0832] A means of autonomously conducting tests using prepared test data,

[0833] A means of automatically analyzing test results and providing feedback,

[0834] A means of performing optimization on the generated test data,

[0835] A means of automatically managing access to the system through the testing process,

[0836] A system that includes this.

[0837] (Claim 2)

[0838] The system according to claim 1, which customizes the test item list based on the type of test specified.

[0839] (Claim 3)

[0840] The system according to claim 1, which automatically notifies details of anomalies based on test results and provides improvement suggestions.

[0841] "Application Example 1"

[0842] (Claim 1)

[0843] A means for automatically detecting the components of a screen display device and generating related information,

[0844] A means for creating a test procedure table based on the detected components,

[0845] A means for automatically preparing test information based on a test procedure table,

[0846] A means of autonomously performing verification using well-organized test information,

[0847] A means to automatically analyze verification results and provide feedback,

[0848] A means of recognizing the operation of a work device using a camera and supporting the operation test process,

[0849] A means of sending instructions to a device via communication technology and executing the verification process,

[0850] A system that includes this.

[0851] (Claim 2)

[0852] The system according to claim 1, which personalizes the test procedure table according to the specified type of verification.

[0853] (Claim 3)

[0854] The system according to claim 1, which automatically notifies details of anomalies based on verification results and provides improvement suggestions.

[0855] "Example 2 of combining an emotion engine"

[0856] (Claim 1)

[0857] A means of automatically acquiring elements of the screen interface and generating information,

[0858] A means for generating a list of test items based on acquired elements,

[0859] A means of identifying the user's emotional state using emotion recognition technology when creating a list of test items,

[0860] A means for dynamically adjusting test data based on identified emotional states,

[0861] A means of monitoring users' emotions in real time and providing appropriate feedback,

[0862] A means of analyzing test results and providing individualized feedback,

[0863] A system that includes this.

[0864] (Claim 2)

[0865] The system according to claim 1, which personalizes a list of test items based on a specified type of test.

[0866] (Claim 3)

[0867] The system according to claim 1, which automatically provides detailed notifications and improvement suggestions based on test results and acquired emotional data.

[0868] "Application example 2 when combining with an emotional engine"

[0869] (Claim 1)

[0870] A means for automatically identifying the components of a screen interface and generating information data,

[0871] Means for generating a list of test items based on identified components,

[0872] A means of automatically preparing test information based on a list of test items,

[0873] A means of independently conducting tests using prepared test information,

[0874] A means for analyzing the user's voice characteristics and facial expression data to identify their emotional state and adjust the priority of test items,

[0875] A means of automatically analyzing test results and providing feedback,

[0876] A system that includes this.

[0877] (Claim 2)

[0878] The system according to claim 1, which customizes the list of test items based on the specified test type and adaptively modifies the test scenario in consideration of the user's emotional state.

[0879] (Claim 3)

[0880] The system according to claim 1, which automatically notifies the user of details of anomalies based on test results, provides improvement suggestions, and reports information corresponding to the user's emotional state. [Explanation of Symbols]

[0881] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for automatically detecting the components of a screen display device and generating related information, A means for creating a test procedure table based on the detected components, A means for automatically preparing test information based on a test procedure table, A means of autonomously performing verification using well-organized test information, A means to automatically analyze verification results and provide feedback, A means of recognizing the operation of a work device using a camera and supporting the operation test process, A means of sending instructions to a device via communication technology and executing the verification process, A system that includes this.

2. The system according to claim 1, which personalizes the test procedure table according to the specified type of verification.

3. The system according to claim 1, which automatically notifies details of anomalies based on verification results and provides improvement suggestions.

Images