system

Abstract

The system according to this embodiment aims to automate UI testing of applications and websites. [Solution] The system according to the embodiment comprises a specification unit, a generation unit, an execution unit, an acquisition unit, and a management unit. The specification unit identifies UI components. The generation unit automatically generates test cases based on the UI components identified by the specification unit. The execution unit executes the test cases generated by the generation unit. The acquisition unit acquires the results obtained by the execution unit. The management unit displays and manages the results obtained by the acquisition unit in a list.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the prior art, UI tests of applications and websites are often performed manually, which has the problem of taking a lot of time and labor.

[0005] The system according to the embodiment aims to automate UI tests of applications and websites.

Means for Solving the Problems

[0006] The system according to the embodiment comprises a specification unit, a generation unit, an execution unit, an acquisition unit, and a management unit. The specification unit identifies UI components. The generation unit automatically generates test cases based on the UI components identified by the specification unit. The execution unit executes the test cases generated by the generation unit. The acquisition unit acquires the results obtained by the execution unit. The management unit displays and manages the results obtained by the acquisition unit in a list. [Effects of the Invention]

[0007] The system according to this embodiment can automate UI testing of applications and websites. [Brief explanation of the drawing]

[0008] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Modes for carrying out the invention]

[0009] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0010] First, let's explain the terminology used in the following explanation.

[0011] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), or TPU (Tensor Processing Unit).

[0012] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0013] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0014] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F manages communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0016] [First Embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. Also, the database 24 and the communication I / F 26 are connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 comprises a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The receiving device 38, output device 40, and camera 42 are also connected to the bus 52.

[0020] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, and accepts user input. The touch panel 38A accepts user input via touch by detecting contact with an object (e.g., a pen or finger). The microphone 38B accepts user input via voice by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 (see Figure 2) acquires the data indicating the user input.

[0021] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user by outputting the data in a form perceptible to the user (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0023] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0025] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0026] In the smart device 14, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart device 14 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0027] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device having the data generation model 58. The data processing device 12 may also be a server device or a terminal device owned by a user (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example of form 1) The UI test automation system according to an embodiment of the present invention is a system that automates UI testing of applications and websites using generative AI. This system is compatible with different devices and environments and responds immediately to changes and updates. The UI test automation system allows developers and quality managers to easily conduct UI testing and improve the quality of applications. For example, manual UI testing is time-consuming and labor-intensive, and has a high potential for human error. It also has the problem of being difficult to respond to continuous UI changes and updates. Specifically, UI testing includes the following steps: launching the application and executing processes, and confirming the results (assert), obtaining values ​​of UI components, executing UI operations, identifying target UI components, and displaying and managing a list of test cases and execution results. These steps are prone to failure if the UI components are not accurately identified. In particular, it is important to define the UI structure and names of UI components for testing. As a solution to this problem in the present invention, UI testing is automated using generative AI. The generative AI identifies UI components, operates them, obtains values, executes test cases, and manages the results. This allows for flexible response to changes and continuous and efficient verification of UI quality. This system dramatically simplifies the repetitive testing process, reducing the burden on developers and quality managers. Specifically, the generative AI works as follows: The user specifies the application or website to be tested. The generative AI analyzes the specified application or website and identifies UI components. The generative AI automatically generates test cases for the identified UI components. The generative AI executes the automatically generated test cases and retrieves the results. The results retrieved by the generative AI are displayed in a list and managed. In this way, by utilizing the generative AI, testing can be performed more efficiently and accurately compared to manual UI testing. Furthermore, it can flexibly respond to UI changes and updates, enabling continuous quality improvement. As a result, the UI test automation system can efficiently and accurately perform UI testing of applications and websites.

[0029] The UI test automation system according to the embodiment comprises an identification unit, a generation unit, an execution unit, an acquisition unit, and a management unit. The identification unit identifies UI components. The identification unit identifies UI components, for example, using a generation AI. The generation AI can identify UI components using screenshots or code analysis. The generation unit automatically generates test cases based on the UI components identified by the identification unit. The generation unit automatically generates test cases, for example, using a generation AI. The generation AI can automatically generate test cases based on past test data or user operation history. The execution unit executes the test cases generated by the generation unit. The execution unit can execute test cases, for example, using a generation AI. The execution unit can configure environment settings to support different devices and browsers. The acquisition unit acquires the results acquired by the execution unit. The acquisition unit can acquire the results, for example, using a generation AI. The management unit displays and manages the results acquired by the acquisition unit. The management unit can manage the results, for example, using a generation AI. The management unit includes functions for filtering and sorting the acquired results. The management unit includes a report generation function that visually displays the acquired results in graphs and charts. As a result, the UI test automation system according to this embodiment can automate UI testing, support different devices and environments, and respond immediately to changes and updates.

[0030] The identification unit identifies UI components. The identification unit identifies UI components using, for example, a generative AI. The generative AI can identify UI components using screenshots and code analysis. Specifically, the generative AI analyzes screenshots and uses image recognition technology to identify UI components such as buttons, text fields, and dropdown menus. Furthermore, by performing code analysis, it can extract the structure and attributes of UI components from code such as HTML, CSS, and JavaScript (registered trademark). The generative AI integrates this information to accurately identify the position, size, and attributes of the UI components. For example, by identifying the position of a button from a screenshot and obtaining its ID and class name from code analysis, the identification unit can uniquely identify the UI component. As a result, the identification unit can quickly and accurately identify UI components, providing a foundation for subsequent test case generation and execution.

[0031] The generation unit automatically generates test cases based on UI components identified by the specific unit. The generation unit can, for example, automatically generate test cases using a generation AI. The generation AI can automatically generate test cases based on past test data and user operation history. Specifically, the generation AI analyzes previously executed test cases and user operation logs to extract frequently used operation patterns and edge cases. This allows the generation AI to generate realistic test cases based on actual user operations. For example, a test case for a login screen can include not only cases where the correct username and password are entered, but also cases where the password is incorrect or not entered. Furthermore, the generation AI can consider the attributes and dependencies of UI components and generate complex test cases that link multiple UI components. This allows the generation unit to automatically generate comprehensive and efficient test cases, improving the comprehensiveness and quality of the tests.

[0032] The execution unit executes the test cases generated by the generation unit. The execution unit can execute test cases using, for example, a generation AI. The execution unit can configure environments to support different devices and browsers. Specifically, the execution unit automatically builds a virtual environment for executing test cases and simulates different combinations of devices and browsers. For example, it can run tests on different operating systems such as Windows®, macOS®, and Linux®, and different browsers such as Chrome®, Firefox®, and Safari®. The generation AI monitors the rendering and operation of the UI in each environment and collects the results of the test cases. Furthermore, the execution unit can automatically detect errors and exceptions that occur during the execution of test cases and generate detailed logs. This allows the execution unit to efficiently perform UI tests in different environments and detect potential problems early.

[0033] The acquisition unit acquires the results obtained by the execution unit. The acquisition unit can acquire results using, for example, a generating AI. Specifically, the acquisition unit collects the execution results of test cases in real time and analyzes the results using a generating AI. The generating AI automatically calculates metrics such as the success rate of test cases, the frequency of errors, and the execution time, and evaluates the results. Furthermore, the acquisition unit collects detailed execution results such as screenshots and log files, which can be used for subsequent analysis and debugging. For example, if a particular test case fails, the acquisition unit can collect screenshots and error logs from the time of the failure and analyze the cause of the error using a generating AI in order to identify the cause. In this way, the acquisition unit can acquire test results quickly and accurately, contributing to the improvement of the overall system quality.

[0034] The management department displays and manages the results obtained by the acquisition department. The management department can manage the results using, for example, a generating AI. The management department has functions to filter and sort the obtained results. Specifically, the management department centrally manages the execution results of test cases and automatically classifies and organizes the results using the generating AI. For example, it can display successful and failed test cases separately and sort them based on the type and frequency of errors. Furthermore, the management department has a report generation function that visually displays the obtained results in graphs and charts. The generating AI analyzes trends and patterns in the test results and generates reports in a visually easy-to-understand format. This allows the management department to quickly grasp the overall picture of the test results and support early detection of problems and the planning of countermeasures. In addition, the management department can save the history of test results and track fluctuations in system quality by comparing them with past results. This allows the management department to efficiently manage and analyze test results and contribute to improving the overall quality of the system.

[0035] The management unit has the function of filtering and sorting the acquired results. For example, the management unit can filter the acquired results and display them based on specific conditions. For example, the management unit can filter the results based on the type and frequency of errors. The management unit can also sort the acquired results and display them in a specific order. For example, the management unit can sort the results based on the severity and time of occurrence of errors. This allows for efficient management of the acquired results. Filtering is the process of selecting data based on specific conditions, for example, and sorting is the process of rearranging data based on specific criteria. Some or all of the above processing in the management unit may be performed using, for example, a generative AI, or without using a generative AI. For example, the management unit can input the acquired results into a generative AI and have the generative AI perform the filtering and sorting processes.

[0036] The management unit has a report generation function that visually displays the acquired results in graphs or charts. For example, the management unit can display the acquired results in graphs. For example, the management unit can display the frequency of error occurrences in a bar graph. The management unit can also display the acquired results in charts. For example, the management unit can display the types of errors in a pie chart. Furthermore, the management unit can also display the acquired results in line graphs. For example, the management unit can display the time it takes for an error to occur in a line graph. This makes it easier to visually understand the acquired results. A method of visual display is, for example, the process of displaying data in a visual format such as graphs or charts. Some or all of the above-described processes in the management unit may be performed using, for example, a generation AI, or without using a generation AI. For example, the management unit can input the acquired results into a generation AI and have the generation AI generate graphs or charts.

[0037] The execution unit can configure settings for different devices or browsers. For example, the execution unit can configure settings for different devices such as smartphones, tablets, and desktops. The execution unit can also configure settings for different browsers such as Chrome, Firefox, and Safari. Furthermore, the execution unit can configure settings for different device and browser combinations. For example, the execution unit can configure settings for combinations such as the Chrome browser on a smartphone and the Firefox browser on a tablet. This allows tests to be run on different devices and browsers. Configuration is the process of configuring settings for specific devices and browsers. Some or all of the above processing in the execution unit may be performed using, for example, a generative AI, or without a generative AI. For example, the execution unit can input settings for different devices and browsers into a generative AI and have the generative AI perform the configuration process.

[0038] The identification unit can identify UI components using screenshots or code analysis. For example, the identification unit can identify UI components using screenshots. For example, the identification unit can perform image analysis of screenshots to identify specific UI components. The identification unit can also identify UI components using code analysis. For example, the identification unit can analyze the source code of an application to identify specific UI components. Furthermore, the identification unit can also identify UI components by combining screenshots and code analysis. For example, the identification unit can identify UI components with higher accuracy by combining image analysis of screenshots and code analysis. This improves the accuracy of UI component identification. A screenshot is, for example, an image captured from the application screen, and code analysis is the process of analyzing source code to extract specific information. Some or all of the above-described processes in the identification unit may be performed using, for example, a generative AI, or without a generative AI. For example, the identification unit can input screenshots or code into a generative AI and have the generative AI perform the identification of UI components.

[0039] The generation unit can automatically generate test cases based on past test data or user operation history. For example, the generation unit can automatically generate test cases based on past test data. For example, the generation unit can analyze past test data, identify frequently failing patterns, and generate test cases based on them. The generation unit can also automatically generate test cases based on user operation history. For example, the generation unit can analyze user operation history and generate test cases based on specific operation sequences. Furthermore, the generation unit can automatically generate test cases by combining past test data and user operation history. For example, the generation unit can analyze past test data and user operation history to generate more accurate test cases. This improves the accuracy of test case generation. Past test data is, for example, the result data of previously executed tests, and user operation history is a record of when the user operated the application. Some or all of the above processing in the generation unit may be performed using, for example, a generation AI, or without using a generation AI. For example, the generation unit can input past test data and user operation history into a generation AI and have the generation AI perform the generation of test cases.

[0040] The identification unit can improve the accuracy of identification based on the color or shape of UI components when analyzing screenshots. For example, the identification unit can have the generating AI analyze the hue of the screenshot and identify UI components with a specific color. It can also have the generating AI analyze the shape of the screenshot and identify UI components with a specific shape. Furthermore, the identification unit can have the generating AI analyze combinations of color and shape in the screenshot and identify specific UI components. This improves the accuracy of screenshot analysis. "Color" refers to, for example, a specific hue or color range, and "shape" refers to a specific shape or pattern. Some or all of the above processing in the identification unit may be performed using, for example, the generating AI, or without the generating AI. For example, the identification unit can input a screenshot into the generating AI and have the generating AI perform the identification of UI components based on color and shape.

[0041] The identification unit can improve the accuracy of identification by specifically analyzing the attribute information of UI components during code analysis. For example, the identification unit can have the generating AI analyze the attribute information in the code and identify UI components that have specific attributes. The identification unit can also have the generating AI analyze the attribute information in the code in detail and identify UI components that have multiple attributes. Furthermore, the identification unit can have the generating AI analyze the attribute information in the code and identify UI components that have a specific combination of attributes. This improves the accuracy of code analysis. Attribute information refers to, for example, specific properties or metadata related to UI components. Some or all of the above processing in the identification unit may be performed using, for example, the generating AI, or without using the generating AI. For example, the identification unit can input the code into the generating AI and have the generating AI perform the identification of UI components based on attribute information.

[0042] The identification unit can apply identification methods that correspond to different resolutions and devices when analyzing screenshots. For example, the identification unit can use a generation AI to analyze screenshots of different resolutions and identify UI components. Furthermore, the identification unit can use a generation AI to analyze screenshots of different devices and identify UI components. In addition, the identification unit can use a generation AI to analyze combinations of different resolutions and devices and identify UI components. This makes it possible to identify UI components corresponding to different resolutions and devices. Resolution refers to, for example, the number of pixels or screen density of a screenshot, and device refers to different types of devices such as smartphones, tablets, and desktops. Some or all of the above processing in the identification unit may be performed using, for example, a generation AI, or without a generation AI. For example, the identification unit can input screenshots of different resolutions and devices into a generation AI and have the generation AI perform UI component identification.

[0043] The identification unit can apply specific algorithms corresponding to different programming languages ​​during code analysis. For example, the identification unit can use a generation AI to analyze code in a different programming language and identify UI components. Furthermore, the identification unit can use a generation AI to analyze attribute information in a different programming language and identify UI components. Additionally, the identification unit can use a generation AI to analyze the code structure of a different programming language and identify UI components. This makes it possible to identify UI components corresponding to different programming languages. Programming languages ​​refer to different languages ​​such as JavaScript, Python, and Java. Some or all of the above-described processes in the identification unit may be performed using a generation AI, or without one. For example, the identification unit can input code in a different programming language into a generation AI and have the generation AI perform the identification of UI components.

[0044] The generation unit can analyze past test data and generate test cases based on frequently failing patterns. For example, the generation unit can use the generation AI to analyze past test data and identify frequently failing UI components. The generation unit can also use the generation AI to analyze past test data and identify frequently failing operation sequences. Furthermore, the generation unit can use the generation AI to analyze past test data and identify frequently failing conditions. This makes it possible to generate test cases based on frequently failing patterns. Frequently failing patterns refer to, for example, the tendency for specific UI components or operation sequences to repeatedly fail. Some or all of the above processing in the generation unit may be performed using, for example, the generation AI, or without the generation AI. For example, the generation unit can input past test data into the generation AI and have the generation AI identify frequently failing patterns and generate test cases.

[0045] The generation unit can analyze the user's operation history in detail and generate test cases based on specific operation sequences. For example, the generation unit's generation AI can analyze the user's operation history and identify specific operation sequences. The generation unit can also have the generation AI analyze the user's operation history in detail and identify frequently performed operation sequences. Furthermore, the generation unit can have the generation AI analyze the user's operation history and generate test cases based on specific operation sequences. This makes it possible to generate test cases based on specific operation sequences. An operation sequence refers, for example, to a series of steps or actions when a user operates an application. Some or all of the above processing in the generation unit may be performed using, for example, the generation AI, or without the generation AI. For example, the generation unit can input the user's operation history into the generation AI and have the generation AI execute the generation of test cases based on operation sequences.

[0046] The generation unit can generate test cases that correspond to different devices and browsers based on past test data. For example, the generation unit can use a generation AI to analyze past test data and generate test cases that correspond to different devices. Furthermore, the generation unit can use a generation AI to analyze past test data and generate test cases that correspond to different browsers. In addition, the generation unit can use a generation AI to analyze past test data and generate test cases that correspond to different device and browser combinations. This makes it possible to generate test cases that correspond to different devices and browsers. "Devices" refer to different types of devices such as smartphones, tablets, and desktops, and "browsers" refer to different types of browsers such as Chrome, Firefox, and Safari. Some or all of the above-described processes in the generation unit may be performed using a generation AI, or they may be performed without a generation AI. For example, the generation unit can input past test data into a generation AI and have the generation AI generate test cases that correspond to different devices and browsers.

[0047] The generation unit can generate test cases corresponding to different user profiles based on the user's operation history. For example, the generation unit's generation AI can analyze the user's operation history and generate test cases corresponding to different user profiles. The generation unit can also have the generation AI analyze the user's operation history in detail and identify operation sequences corresponding to different user profiles. Furthermore, the generation unit can have the generation AI analyze the user's operation history and identify conditions corresponding to different user profiles. This makes it possible to generate test cases corresponding to different user profiles. A user profile refers to, for example, the operation patterns and settings of a particular user. Some or all of the above processing in the generation unit may be performed using, for example, the generation AI, or without the generation AI. For example, the generation unit can input the user's operation history into the generation AI and have the generation AI execute the generation of test cases corresponding to different user profiles.

[0048] The execution unit can improve execution accuracy by performing detailed environment settings for different devices and browsers. For example, the execution unit can improve execution accuracy by having the generation AI perform detailed environment settings for different devices. Furthermore, the execution unit can improve execution accuracy by having the generation AI perform detailed environment settings for different browsers. In addition, the execution unit can improve execution accuracy by having the generation AI perform detailed environment settings for different device and browser combinations. This improves the execution accuracy of test cases that correspond to different devices and browsers. A device refers to a different type of device, such as a smartphone, tablet, or desktop, and a browser refers to a different type of browser, such as Chrome, Firefox, or Safari. Some or all of the above processing in the execution unit may be performed using the generation AI, or without using the generation AI. For example, the execution unit can input the environment settings for different devices and browsers into the generation AI and have the generation AI perform the improvement of execution accuracy.

[0049] The execution unit can detect errors in real time during test case execution and immediately propose corrective actions. For example, the execution unit can have a generating AI detect errors in real time during test case execution and immediately propose corrective actions. Furthermore, the execution unit can have a generating AI analyze errors in detail in real time during test case execution and propose corrective actions. In addition, the execution unit can have a generating AI automatically detect errors in real time during test case execution and propose corrective actions. This improves test accuracy by detecting errors in real time and immediately proposing corrective actions. "Real time" refers to the process of immediately detecting errors and proposing corrective actions during test case execution. Some or all of the above-described processes in the execution unit may be performed using a generating AI, or without a generating AI. For example, the execution unit can use a generating AI to detect errors during test case execution and have the generating AI execute corrective action suggestions.

[0050] The execution unit can apply test case execution methods corresponding to different network environments. For example, the generation AI can apply test case execution methods corresponding to different network environments. The execution unit can also adjust the test case execution methods considering the characteristics of different network environments. Furthermore, the execution unit can automatically apply test case execution methods corresponding to different network environments using the generation AI. This makes it possible to execute test cases corresponding to different network environments. Network environments refer to different types of networks, such as wired networks, wireless networks, and mobile networks. Some or all of the above processing in the execution unit may be performed using the generation AI, for example, or without the generation AI. For example, the execution unit can input the settings of different network environments into the generation AI and have the generation AI apply the test case execution methods.

[0051] The execution unit can adjust the execution content of test cases based on different user privileges. For example, the generation AI can adjust the execution content of test cases based on different user privileges. The execution unit can also have the generation AI execute detailed test cases for users with administrator privileges. Furthermore, the execution unit can have the generation AI execute basic test cases for users with general user privileges. This makes it possible to execute test cases according to different user privileges. User privileges refer to different types of privileges, such as administrator privileges and general user privileges. Some or all of the above processing in the execution unit may be performed using the generation AI, or not. For example, the execution unit can input user privilege information into the generation AI and have the generation AI adjust the execution content of the test cases.

[0052] The acquisition unit can analyze the execution results of test cases in detail and identify specific error causes. For example, the acquisition unit can have the generation AI analyze the execution results of test cases and identify specific error causes. The acquisition unit can also have the generation AI analyze the execution results of test cases in detail and identify multiple error causes. Furthermore, the acquisition unit can have the generation AI analyze the execution results of test cases and identify error cause patterns. This allows for rapid problem resolution by analyzing specific error causes in detail. Error causes refer to, for example, the reasons why a particular UI component or operation sequence fails. Some or all of the above processing in the acquisition unit may be performed using, for example, the generation AI, or without the generation AI. For example, the acquisition unit can input the execution results of test cases to the generation AI and have the generation AI perform the error cause identification.

[0053] The acquisition unit can suggest improvements to the next test case based on the acquired results. For example, the acquisition unit can analyze the results acquired by the generation AI and suggest improvements to the next test case. Furthermore, the acquisition unit can analyze the results acquired by the generation AI in detail and suggest specific improvements. In addition, the acquisition unit can automatically suggest improvements to the next test case based on the results acquired by the generation AI. This improves the accuracy of the tests by suggesting improvements to the next test case. Improvements refer to, for example, methods for improving specific UI components or operation sequences. Some or all of the above processing in the acquisition unit may be performed using, for example, the generation AI, or without the generation AI. For example, the acquisition unit can input the acquired results into the generation AI and have the generation AI execute the improvement suggestions.

[0054] The acquisition unit can apply result acquisition methods that correspond to different devices and browsers. For example, the acquisition unit can apply result acquisition methods that correspond to different devices for the generating AI. Furthermore, the acquisition unit can apply result acquisition methods that correspond to different browsers for the generating AI. In addition, the acquisition unit can apply result acquisition methods that correspond to different combinations of devices and browsers for the generating AI. This makes it possible to acquire results that correspond to different devices and browsers. A device refers to a different type of device, such as a smartphone, tablet, or desktop, and a browser refers to a different type of browser, such as Chrome, Firefox, or Safari. Some or all of the above processing in the acquisition unit may be performed using the generating AI, for example, or without using the generating AI. For example, the acquisition unit can input the settings of different devices and browsers into the generating AI and have the generating AI execute the application of the result acquisition method.

[0055] The acquisition unit can provide feedback corresponding to different user profiles based on the acquired results. For example, the acquisition unit can analyze the results acquired by the generation AI and provide feedback corresponding to different user profiles. Furthermore, the acquisition unit can analyze the results acquired by the generation AI in detail and provide specific feedback corresponding to different user profiles. In addition, the acquisition unit can automatically provide feedback corresponding to different user profiles based on the results acquired by the generation AI. This allows for the provision of feedback tailored to different user profiles. A user profile refers, for example, to the operation patterns and settings of a specific user. Some or all of the above-described processing in the acquisition unit may be performed using, for example, the generation AI, or without using the generation AI. For example, the acquisition unit can input the acquired results into the generation AI and have the generation AI provide the feedback.

[0056] The management unit can configure detailed functions for filtering and sorting the acquired results to improve management accuracy. For example, the management unit can filter the results acquired by the generating AI and prioritize the display of important results. The management unit can also sort the results acquired by the generating AI and adjust the display order based on specific conditions. Furthermore, the management unit can filter and sort the results acquired by the generating AI to improve management accuracy. This improves the accuracy of managing the acquired results. Filtering is the process of selecting data based on specific conditions, for example, and sorting is the process of rearranging data based on specific criteria. Some or all of the above processing in the management unit may be performed using the generating AI, for example, or without using the generating AI. For example, the management unit can input the acquired results into the generating AI and have the generating AI perform the filtering and sorting processes.

[0057] The management department can enhance its report generation function by visually displaying acquired results in graphs and charts. For example, the management department can display the results acquired by the generation AI in graphs, making them easier to understand visually. The management department can also display the results acquired by the generation AI in charts, making them easier to understand visually. Furthermore, the management department can enhance its report generation function by visually displaying the results acquired by the generation AI in graphs and charts. This improves the visual understanding of the acquired results. A method of visual display is, for example, the process of displaying data in a visual format such as graphs and charts. Some or all of the above-described processes in the management department may be performed using, for example, the generation AI, or not using the generation AI. For example, the management department can input acquired results into the generation AI and have the generation AI generate graphs and charts.

[0058] The management unit can apply result display methods that correspond to different devices and browsers. For example, the management unit can have the generating AI apply result display methods that correspond to different devices. The management unit can also have the generating AI apply result display methods that correspond to different browsers. Furthermore, the management unit can have the generating AI apply result display methods that correspond to different combinations of devices and browsers. This makes it possible to display results that correspond to different devices and browsers. A device refers to a different type of device, such as a smartphone, tablet, or desktop, and a browser refers to a different type of browser, such as Chrome, Firefox, or Safari. Some or all of the above processing in the management unit may be performed using the generating AI, for example, or without using the generating AI. For example, the management unit can input the settings of different devices and browsers into the generating AI and have the generating AI execute the application of the result display method.

[0059] The management department can generate reports corresponding to different user profiles based on the acquired results. For example, the management department can analyze the acquired results using a generation AI and generate reports corresponding to different user profiles. Furthermore, the management department can analyze the acquired results in detail using a generation AI and generate specific reports corresponding to different user profiles. In addition, the management department can automatically generate reports corresponding to different user profiles based on the acquired results using a generation AI. This allows for the generation of reports tailored to different user profiles. A user profile refers, for example, to the operation patterns and settings of a specific user. Some or all of the above-described processes in the management department may be performed using a generation AI, or without using a generation AI. For example, the management department can input the acquired results into a generation AI and have the generation AI generate the report.

[0060] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0061] The UI test automation system can also have the functionality to apply test case execution methods to different network environments. For example, the generating AI can apply test case execution methods to different network environments. Furthermore, the generating AI can adjust the test case execution method considering the characteristics of different network environments. In addition, the generating AI can automatically apply test case execution methods to different network environments. This enables the execution of test cases that correspond to different network environments. Network environments refer to different types of networks, such as wired networks, wireless networks, and mobile networks. Some or all of the above-described processes in the execution unit may be performed using the generating AI, or without using the generating AI. For example, the execution unit can input the settings for different network environments into the generating AI and have the generating AI apply the test case execution method.

[0062] The UI test automation system can also include a function to adjust the execution of test cases based on different user privileges. For example, the generating AI can adjust the execution of test cases based on different user privileges. The generating AI can also execute detailed test cases for users with administrator privileges. Furthermore, the generating AI can execute basic test cases for users with general user privileges. This makes it possible to execute test cases according to different user privileges. User privileges refer to different types of privileges, such as administrator privileges and general user privileges. Some or all of the above processing in the execution unit may be performed using the generating AI, or it may be performed without using the generating AI. For example, the execution unit can input user privilege information into the generating AI and have the generating AI adjust the execution of test cases.

[0063] The UI test automation system can also have the functionality to apply result acquisition methods that correspond to different devices and browsers. For example, the generating AI can apply result acquisition methods that correspond to different devices. Furthermore, the generating AI can apply result acquisition methods that correspond to different browsers. In addition, the generating AI can apply result acquisition methods that correspond to different combinations of devices and browsers. This makes it possible to acquire results that correspond to different devices and browsers. "Device" refers to different types of devices such as smartphones, tablets, and desktops, and "browser" refers to different types of browsers such as Chrome, Firefox, and Safari. Some or all of the above processing in the acquisition unit may be performed using the generating AI, or without using the generating AI. For example, the acquisition unit can input the settings of different devices and browsers into the generating AI and have the generating AI execute the application of the result acquisition method.

[0064] The UI test automation system can also be equipped with the ability to apply specific algorithms corresponding to different programming languages. For example, the generating AI can analyze code in different programming languages ​​and identify UI components. Furthermore, the generating AI can analyze attribute information in different programming languages ​​and identify UI components. In addition, the generating AI can analyze the code structure of different programming languages ​​and identify UI components. This makes it possible to identify UI components corresponding to different programming languages. Programming languages ​​refer to different languages ​​such as JavaScript, Python, and Java. Some or all of the above-described processes in the identification unit may be performed using the generating AI, or without using the generating AI. For example, the identification unit can input code in a different programming language into the generating AI and have the generating AI perform the identification of UI components.

[0065] The UI test automation system can also be equipped with the ability to provide feedback corresponding to different user profiles. For example, it can analyze the results obtained by the generating AI and provide feedback corresponding to different user profiles. It can also analyze the results obtained by the generating AI in detail and provide specific feedback corresponding to different user profiles. Furthermore, it can automatically provide feedback corresponding to different user profiles based on the results obtained by the generating AI. This allows for the provision of feedback tailored to different user profiles. A user profile refers, for example, to the operation patterns and settings of a particular user. Some or all of the above processing in the acquisition unit may be performed using, for example, the generating AI, or without using the generating AI. For example, the acquisition unit can input the acquired results into the generating AI and have the generating AI perform the provision of feedback.

[0066] The following briefly describes the processing flow for example form 1.

[0067] Step 1: The identification unit identifies UI components. The identification unit uses a generation AI to analyze screenshots and code to identify UI components. Step 2: The generation unit automatically generates test cases based on the UI components identified by the identification unit. The generation unit uses generation AI to automatically generate test cases based on past test data and user operation history. Step 3: The execution unit executes the test cases generated by the generation unit. The execution unit uses the generation AI to execute the test cases and configures the environment to support different devices and browsers. Step 4: The acquisition unit acquires the results obtained by the execution unit. The acquisition unit acquires the results using the generation AI. Step 5: The management unit displays and manages the results acquired by the acquisition unit. The management unit has functions to manage, filter, and sort the results using generation AI. It also has a report generation function that visually displays the acquired results in graphs and charts.

[0068] (Example of form 2) The UI test automation system according to an embodiment of the present invention is a system that automates UI testing of applications and websites using generative AI. This system is compatible with different devices and environments and responds immediately to changes and updates. The UI test automation system allows developers and quality managers to easily conduct UI testing and improve the quality of applications. For example, manual UI testing is time-consuming and labor-intensive, and has a high potential for human error. It also has the problem of being difficult to respond to continuous UI changes and updates. Specifically, UI testing includes the following steps: launching the application and executing processes, and confirming the results (assert), obtaining values ​​of UI components, executing UI operations, identifying target UI components, and displaying and managing a list of test cases and execution results. These steps are prone to failure if the UI components are not accurately identified. In particular, it is important to define the UI structure and names of UI components for testing. As a solution to this problem in the present invention, UI testing is automated using generative AI. The generative AI identifies UI components, operates them, obtains values, executes test cases, and manages the results. This allows for flexible response to changes and continuous and efficient verification of UI quality. This system dramatically simplifies the repetitive testing process, reducing the burden on developers and quality managers. Specifically, the generative AI works as follows: The user specifies the application or website to be tested. The generative AI analyzes the specified application or website and identifies UI components. The generative AI automatically generates test cases for the identified UI components. The generative AI executes the automatically generated test cases and retrieves the results. The results retrieved by the generative AI are displayed in a list and managed. In this way, by utilizing the generative AI, testing can be performed more efficiently and accurately compared to manual UI testing. Furthermore, it can flexibly respond to UI changes and updates, enabling continuous quality improvement. As a result, the UI test automation system can efficiently and accurately perform UI testing of applications and websites.

[0069] The UI test automation system according to the embodiment comprises an identification unit, a generation unit, an execution unit, an acquisition unit, and a management unit. The identification unit identifies UI components. The identification unit identifies UI components, for example, using a generation AI. The generation AI can identify UI components using screenshots or code analysis. The generation unit automatically generates test cases based on the UI components identified by the identification unit. The generation unit automatically generates test cases, for example, using a generation AI. The generation AI can automatically generate test cases based on past test data or user operation history. The execution unit executes the test cases generated by the generation unit. The execution unit can execute test cases, for example, using a generation AI. The execution unit can configure environment settings to support different devices and browsers. The acquisition unit acquires the results acquired by the execution unit. The acquisition unit can acquire the results, for example, using a generation AI. The management unit displays and manages the results acquired by the acquisition unit. The management unit can manage the results, for example, using a generation AI. The management unit includes functions for filtering and sorting the acquired results. The management unit includes a report generation function that visually displays the acquired results in graphs and charts. As a result, the UI test automation system according to this embodiment can automate UI testing, support different devices and environments, and respond immediately to changes and updates.

[0070] The identification unit identifies UI components. The identification unit identifies UI components using, for example, a generative AI. The generative AI can identify UI components using screenshots and code analysis. Specifically, the generative AI analyzes screenshots and uses image recognition technology to identify UI components such as buttons, text fields, and dropdown menus. Furthermore, by performing code analysis, it can extract the structure and attributes of UI components from code such as HTML, CSS, and JavaScript. The generative AI integrates this information to accurately identify the position, size, and attributes of the UI components. For example, by identifying the position of a button from a screenshot and obtaining its ID and class name from code analysis, the identification unit can uniquely identify the UI component. This allows the identification unit to quickly and accurately identify UI components, providing a foundation for subsequent test case generation and execution.

[0071] The generation unit automatically generates test cases based on UI components identified by the specific unit. The generation unit can, for example, automatically generate test cases using a generation AI. The generation AI can automatically generate test cases based on past test data and user operation history. Specifically, the generation AI analyzes previously executed test cases and user operation logs to extract frequently used operation patterns and edge cases. This allows the generation AI to generate realistic test cases based on actual user operations. For example, a test case for a login screen can include not only cases where the correct username and password are entered, but also cases where the password is incorrect or not entered. Furthermore, the generation AI can consider the attributes and dependencies of UI components and generate complex test cases that link multiple UI components. This allows the generation unit to automatically generate comprehensive and efficient test cases, improving the comprehensiveness and quality of the tests.

[0072] The execution unit executes the test cases generated by the generation unit. The execution unit can execute test cases using, for example, a generation AI. The execution unit can configure environments to support different devices and browsers. Specifically, the execution unit automatically builds virtual environments for executing test cases and simulates different combinations of devices and browsers. For example, it can run tests on different operating systems such as Windows, macOS, and Linux, and different browsers such as Chrome, Firefox, and Safari. The generation AI monitors the rendering and behavior of the UI in each environment and collects the results of the test cases. Furthermore, the execution unit can automatically detect errors and exceptions that occur during the execution of test cases and generate detailed logs. This allows the execution unit to efficiently perform UI tests in different environments and detect potential problems early.

[0073] The acquisition unit acquires the results obtained by the execution unit. The acquisition unit can acquire results using, for example, a generating AI. Specifically, the acquisition unit collects the execution results of test cases in real time and analyzes the results using a generating AI. The generating AI automatically calculates metrics such as the success rate of test cases, the frequency of errors, and the execution time, and evaluates the results. Furthermore, the acquisition unit collects detailed execution results such as screenshots and log files, which can be used for subsequent analysis and debugging. For example, if a particular test case fails, the acquisition unit can collect screenshots and error logs from the time of the failure and analyze the cause of the error using a generating AI in order to identify the cause. In this way, the acquisition unit can acquire test results quickly and accurately, contributing to the improvement of the overall system quality.

[0074] The management department displays and manages the results obtained by the acquisition department. The management department can manage the results using, for example, a generating AI. The management department has functions to filter and sort the obtained results. Specifically, the management department centrally manages the execution results of test cases and automatically classifies and organizes the results using the generating AI. For example, it can display successful and failed test cases separately and sort them based on the type and frequency of errors. Furthermore, the management department has a report generation function that visually displays the obtained results in graphs and charts. The generating AI analyzes trends and patterns in the test results and generates reports in a visually easy-to-understand format. This allows the management department to quickly grasp the overall picture of the test results and support early detection of problems and the planning of countermeasures. In addition, the management department can save the history of test results and track fluctuations in system quality by comparing them with past results. This allows the management department to efficiently manage and analyze test results and contribute to improving the overall quality of the system.

[0075] The management unit has the function of filtering and sorting the acquired results. For example, the management unit can filter the acquired results and display them based on specific conditions. For example, the management unit can filter the results based on the type and frequency of errors. The management unit can also sort the acquired results and display them in a specific order. For example, the management unit can sort the results based on the severity and time of occurrence of errors. This allows for efficient management of the acquired results. Filtering is the process of selecting data based on specific conditions, for example, and sorting is the process of rearranging data based on specific criteria. Some or all of the above processing in the management unit may be performed using, for example, a generative AI, or without using a generative AI. For example, the management unit can input the acquired results into a generative AI and have the generative AI perform the filtering and sorting processes.

[0076] The management unit has a report generation function that visually displays the acquired results in graphs or charts. For example, the management unit can display the acquired results in graphs. For example, the management unit can display the frequency of error occurrences in a bar graph. The management unit can also display the acquired results in charts. For example, the management unit can display the types of errors in a pie chart. Furthermore, the management unit can also display the acquired results in line graphs. For example, the management unit can display the time it takes for an error to occur in a line graph. This makes it easier to visually understand the acquired results. A method of visual display is, for example, the process of displaying data in a visual format such as graphs or charts. Some or all of the above-described processes in the management unit may be performed using, for example, a generation AI, or without using a generation AI. For example, the management unit can input the acquired results into a generation AI and have the generation AI generate graphs or charts.

[0077] The execution unit can configure settings for different devices or browsers. For example, the execution unit can configure settings for different devices such as smartphones, tablets, and desktops. The execution unit can also configure settings for different browsers such as Chrome, Firefox, and Safari. Furthermore, the execution unit can configure settings for different device and browser combinations. For example, the execution unit can configure settings for combinations such as the Chrome browser on a smartphone and the Firefox browser on a tablet. This allows tests to be run on different devices and browsers. Configuration is the process of configuring settings for specific devices and browsers. Some or all of the above processing in the execution unit may be performed using, for example, a generative AI, or without a generative AI. For example, the execution unit can input settings for different devices and browsers into a generative AI and have the generative AI perform the configuration process.

[0078] The identification unit can identify UI components using screenshots or code analysis. For example, the identification unit can identify UI components using screenshots. For example, the identification unit can perform image analysis of screenshots to identify specific UI components. The identification unit can also identify UI components using code analysis. For example, the identification unit can analyze the source code of an application to identify specific UI components. Furthermore, the identification unit can also identify UI components by combining screenshots and code analysis. For example, the identification unit can identify UI components with higher accuracy by combining image analysis of screenshots and code analysis. This improves the accuracy of UI component identification. A screenshot is, for example, an image captured from the application screen, and code analysis is the process of analyzing source code to extract specific information. Some or all of the above-described processes in the identification unit may be performed using, for example, a generative AI, or without a generative AI. For example, the identification unit can input screenshots or code into a generative AI and have the generative AI perform the identification of UI components.

[0079] The generation unit can automatically generate test cases based on past test data or user operation history. For example, the generation unit can automatically generate test cases based on past test data. For example, the generation unit can analyze past test data, identify frequently failing patterns, and generate test cases based on them. The generation unit can also automatically generate test cases based on user operation history. For example, the generation unit can analyze user operation history and generate test cases based on specific operation sequences. Furthermore, the generation unit can automatically generate test cases by combining past test data and user operation history. For example, the generation unit can analyze past test data and user operation history to generate more accurate test cases. This improves the accuracy of test case generation. Past test data is, for example, the result data of previously executed tests, and user operation history is a record of when the user operated the application. Some or all of the above processing in the generation unit may be performed using, for example, a generation AI, or without using a generation AI. For example, the generation unit can input past test data and user operation history into a generation AI and have the generation AI perform the generation of test cases.

[0080] The identification unit can estimate the user's emotions and adjust the method of identifying UI components based on the estimated emotions. For example, if the user is stressed, the identification unit can have the generating AI select a simple method of identifying UI components for quick identification. If the user is relaxed, the identification unit can have the generating AI select a detailed method of identifying UI components, prioritizing accuracy. Furthermore, if the user is in a hurry, the identification unit can have the generating AI select a high-speed method of identifying UI components for quick identification. This enables the identification of UI components in accordance with the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or a generating AI. The generating AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the identification unit may be performed using a generating AI, or not using a generating AI. For example, the identification unit can input user emotion data into a generating AI and have the generating AI adjust the method of identifying UI components based on emotions.

[0081] The identification unit can improve the accuracy of identification based on the color or shape of UI components when analyzing screenshots. For example, the identification unit can have the generating AI analyze the hue of the screenshot and identify UI components with a specific color. It can also have the generating AI analyze the shape of the screenshot and identify UI components with a specific shape. Furthermore, the identification unit can have the generating AI analyze combinations of color and shape in the screenshot and identify specific UI components. This improves the accuracy of screenshot analysis. "Color" refers to, for example, a specific hue or color range, and "shape" refers to a specific shape or pattern. Some or all of the above processing in the identification unit may be performed using, for example, the generating AI, or without the generating AI. For example, the identification unit can input a screenshot into the generating AI and have the generating AI perform the identification of UI components based on color and shape.

[0082] The identification unit can improve the accuracy of identification by specifically analyzing the attribute information of UI components during code analysis. For example, the identification unit can have the generating AI analyze the attribute information in the code and identify UI components that have specific attributes. The identification unit can also have the generating AI analyze the attribute information in the code in detail and identify UI components that have multiple attributes. Furthermore, the identification unit can have the generating AI analyze the attribute information in the code and identify UI components that have a specific combination of attributes. This improves the accuracy of code analysis. Attribute information refers to, for example, specific properties or metadata related to UI components. Some or all of the above processing in the identification unit may be performed using, for example, the generating AI, or without using the generating AI. For example, the identification unit can input the code into the generating AI and have the generating AI perform the identification of UI components based on attribute information.

[0083] The identification unit can estimate the user's emotions and determine the priority of UI components to identify based on the estimated emotions. For example, if the user is stressed, the identification unit can have the generating AI prioritize identifying important UI components. If the user is relaxed, the identification unit can have the generating AI prioritize identifying detailed UI components. Furthermore, if the user is in a hurry, the identification unit can have the generating AI prioritize identifying UI components that can be identified quickly. This allows for the determination of UI component priorities according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or a generating AI. The generating AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the identification unit may be performed using a generating AI, or not using a generating AI. For example, the identification unit can input user emotion data into a generating AI and have the generating AI perform the determination of UI component priorities based on emotions.

[0084] The identification unit can apply identification methods that correspond to different resolutions and devices when analyzing screenshots. For example, the identification unit can use a generation AI to analyze screenshots of different resolutions and identify UI components. Furthermore, the identification unit can use a generation AI to analyze screenshots of different devices and identify UI components. In addition, the identification unit can use a generation AI to analyze combinations of different resolutions and devices and identify UI components. This makes it possible to identify UI components corresponding to different resolutions and devices. Resolution refers to, for example, the number of pixels or screen density of a screenshot, and device refers to different types of devices such as smartphones, tablets, and desktops. Some or all of the above processing in the identification unit may be performed using, for example, a generation AI, or without a generation AI. For example, the identification unit can input screenshots of different resolutions and devices into a generation AI and have the generation AI perform UI component identification.

[0085] The identification unit can apply specific algorithms corresponding to different programming languages ​​during code analysis. For example, the identification unit can use a generation AI to analyze code in a different programming language and identify UI components. Furthermore, the identification unit can use a generation AI to analyze attribute information in a different programming language and identify UI components. Additionally, the identification unit can use a generation AI to analyze the code structure of a different programming language and identify UI components. This makes it possible to identify UI components corresponding to different programming languages. Programming languages ​​refer to different languages ​​such as JavaScript, Python, and Java. Some or all of the above-described processes in the identification unit may be performed using a generation AI, or without one. For example, the identification unit can input code in a different programming language into a generation AI and have the generation AI perform the identification of UI components.

[0086] The generation unit can estimate the user's emotions and adjust the test case generation method based on the estimated emotions. For example, if the user is stressed, the generation unit can have the generation AI generate simple test cases. If the user is relaxed, the generation unit can have the generation AI generate detailed test cases. Furthermore, if the user is in a hurry, the generation unit can have the generation AI generate test cases that can be executed quickly. This makes it possible to generate test cases that are appropriate to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or a generation AI. The generation AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the generation unit may be performed using a generation AI, or not. For example, the generation unit can input user emotion data into a generation AI and have the generation AI adjust the test case generation method based on emotions.

[0087] The generation unit can analyze past test data and generate test cases based on frequently failing patterns. For example, the generation unit can use the generation AI to analyze past test data and identify frequently failing UI components. The generation unit can also use the generation AI to analyze past test data and identify frequently failing operation sequences. Furthermore, the generation unit can use the generation AI to analyze past test data and identify frequently failing conditions. This makes it possible to generate test cases based on frequently failing patterns. Frequently failing patterns refer to, for example, the tendency for specific UI components or operation sequences to repeatedly fail. Some or all of the above processing in the generation unit may be performed using, for example, the generation AI, or without the generation AI. For example, the generation unit can input past test data into the generation AI and have the generation AI identify frequently failing patterns and generate test cases.

[0088] The generation unit can analyze the user's operation history in detail and generate test cases based on specific operation sequences. For example, the generation unit's generation AI can analyze the user's operation history and identify specific operation sequences. The generation unit can also have the generation AI analyze the user's operation history in detail and identify frequently performed operation sequences. Furthermore, the generation unit can have the generation AI analyze the user's operation history and generate test cases based on specific operation sequences. This makes it possible to generate test cases based on specific operation sequences. An operation sequence refers, for example, to a series of steps or actions when a user operates an application. Some or all of the above processing in the generation unit may be performed using, for example, the generation AI, or without the generation AI. For example, the generation unit can input the user's operation history into the generation AI and have the generation AI execute the generation of test cases based on operation sequences.

[0089] The generation unit can estimate the user's emotions and determine the priority of test cases to generate based on the estimated emotions. For example, if the user is stressed, the generation unit can have the generation AI prioritize generating important test cases. If the user is relaxed, the generation unit can have the generation AI prioritize generating detailed test cases. Furthermore, if the user is in a hurry, the generation unit can have the generation AI prioritize generating test cases that can be executed quickly. This allows for the prioritization of test cases according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generation AI. The generation AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the generation unit may be performed using a generation AI, or not. For example, the generation unit can input user emotion data into a generation AI and have the generation AI determine the priority of test cases based on emotions.

[0090] The generation unit can generate test cases that correspond to different devices and browsers based on past test data. For example, the generation unit can use a generation AI to analyze past test data and generate test cases that correspond to different devices. Furthermore, the generation unit can use a generation AI to analyze past test data and generate test cases that correspond to different browsers. In addition, the generation unit can use a generation AI to analyze past test data and generate test cases that correspond to different device and browser combinations. This makes it possible to generate test cases that correspond to different devices and browsers. "Devices" refer to different types of devices such as smartphones, tablets, and desktops, and "browsers" refer to different types of browsers such as Chrome, Firefox, and Safari. Some or all of the above-described processes in the generation unit may be performed using a generation AI, or they may be performed without a generation AI. For example, the generation unit can input past test data into a generation AI and have the generation AI generate test cases that correspond to different devices and browsers.

[0091] The generation unit can generate test cases corresponding to different user profiles based on the user's operation history. For example, the generation unit's generation AI can analyze the user's operation history and generate test cases corresponding to different user profiles. The generation unit can also have the generation AI analyze the user's operation history in detail and identify operation sequences corresponding to different user profiles. Furthermore, the generation unit can have the generation AI analyze the user's operation history and identify conditions corresponding to different user profiles. This makes it possible to generate test cases corresponding to different user profiles. A user profile refers to, for example, the operation patterns and settings of a particular user. Some or all of the above processing in the generation unit may be performed using, for example, the generation AI, or without the generation AI. For example, the generation unit can input the user's operation history into the generation AI and have the generation AI execute the generation of test cases corresponding to different user profiles.

[0092] The execution unit can estimate the user's emotions and adjust the execution order of test cases based on the estimated emotions. For example, if the user is stressed, the execution unit can have the generative AI prioritize the execution of important test cases. If the user is relaxed, the execution unit can have the generative AI prioritize the execution of detailed test cases. Furthermore, if the user is in a hurry, the execution unit can have the generative AI prioritize the execution of test cases that can be executed quickly. This allows the execution order of test cases to be adjusted according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. The generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the execution unit may be performed using a generative AI, or not using a generative AI. For example, the execution unit can input user emotion data into a generative AI and have the generative AI adjust the execution order of test cases based on emotions.

[0093] The execution unit can improve execution accuracy by performing detailed environment settings for different devices and browsers. For example, the execution unit can improve execution accuracy by having the generation AI perform detailed environment settings for different devices. Furthermore, the execution unit can improve execution accuracy by having the generation AI perform detailed environment settings for different browsers. In addition, the execution unit can improve execution accuracy by having the generation AI perform detailed environment settings for different device and browser combinations. This improves the execution accuracy of test cases that correspond to different devices and browsers. A device refers to a different type of device, such as a smartphone, tablet, or desktop, and a browser refers to a different type of browser, such as Chrome, Firefox, or Safari. Some or all of the above processing in the execution unit may be performed using the generation AI, or without using the generation AI. For example, the execution unit can input the environment settings for different devices and browsers into the generation AI and have the generation AI perform the improvement of execution accuracy.

[0094] The execution unit can detect errors in real time during test case execution and immediately propose corrective actions. For example, the execution unit can have a generating AI detect errors in real time during test case execution and immediately propose corrective actions. Furthermore, the execution unit can have a generating AI analyze errors in detail in real time during test case execution and propose corrective actions. In addition, the execution unit can have a generating AI automatically detect errors in real time during test case execution and propose corrective actions. This improves test accuracy by detecting errors in real time and immediately proposing corrective actions. "Real time" refers to the process of immediately detecting errors and proposing corrective actions during test case execution. Some or all of the above-described processes in the execution unit may be performed using a generating AI, or without a generating AI. For example, the execution unit can use a generating AI to detect errors during test case execution and have the generating AI execute corrective action suggestions.

[0095] The execution unit can estimate the user's emotions and determine the priority of test cases to execute based on the estimated emotions. For example, if the user is stressed, the execution unit can have the generative AI prioritize executing important test cases. If the user is relaxed, the execution unit can have the generative AI prioritize executing detailed test cases. Furthermore, if the user is in a hurry, the execution unit can have the generative AI prioritize executing test cases that can be executed quickly. This allows for the prioritization of test cases according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. The generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the execution unit may be performed using a generative AI, or not. For example, the execution unit can input user emotion data into a generative AI and have the generative AI determine the priority of test cases based on emotions.

[0096] The execution unit can apply test case execution methods corresponding to different network environments. For example, the generation AI can apply test case execution methods corresponding to different network environments. The execution unit can also adjust the test case execution methods considering the characteristics of different network environments. Furthermore, the execution unit can automatically apply test case execution methods corresponding to different network environments using the generation AI. This makes it possible to execute test cases corresponding to different network environments. Network environments refer to different types of networks, such as wired networks, wireless networks, and mobile networks. Some or all of the above processing in the execution unit may be performed using the generation AI, for example, or without the generation AI. For example, the execution unit can input the settings of different network environments into the generation AI and have the generation AI apply the test case execution methods.

[0097] The execution unit can adjust the execution content of test cases based on different user privileges. For example, the generation AI can adjust the execution content of test cases based on different user privileges. The execution unit can also have the generation AI execute detailed test cases for users with administrator privileges. Furthermore, the execution unit can have the generation AI execute basic test cases for users with general user privileges. This makes it possible to execute test cases according to different user privileges. User privileges refer to different types of privileges, such as administrator privileges and general user privileges. Some or all of the above processing in the execution unit may be performed using the generation AI, or not. For example, the execution unit can input user privilege information into the generation AI and have the generation AI adjust the execution content of the test cases.

[0098] The acquisition unit can estimate the user's emotions and adjust the level of detail of the results obtained based on the estimated emotions. For example, if the user is stressed, the acquisition unit can have the generating AI obtain concise results. If the user is relaxed, the acquisition unit can have the generating AI obtain detailed results. Furthermore, if the user is in a hurry, the acquisition unit can prioritize results that can be obtained quickly by the generating AI. This allows the level of detail of the results to be adjusted according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or a generating AI. The generating AI is a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above processing in the acquisition unit may be performed using a generating AI, or not using a generating AI. For example, the acquisition unit can input user emotion data into a generating AI and have the generating AI perform the adjustment of the level of detail of the results based on the emotions.

[0099] The acquisition unit can analyze the execution results of test cases in detail and identify specific error causes. For example, the acquisition unit can have the generation AI analyze the execution results of test cases and identify specific error causes. The acquisition unit can also have the generation AI analyze the execution results of test cases in detail and identify multiple error causes. Furthermore, the acquisition unit can have the generation AI analyze the execution results of test cases and identify error cause patterns. This allows for rapid problem resolution by analyzing specific error causes in detail. Error causes refer to, for example, the reasons why a particular UI component or operation sequence fails. Some or all of the above processing in the acquisition unit may be performed using, for example, the generation AI, or without the generation AI. For example, the acquisition unit can input the execution results of test cases to the generation AI and have the generation AI perform the error cause identification.

[0100] The acquisition unit can suggest improvements to the next test case based on the acquired results. For example, the acquisition unit can analyze the results acquired by the generation AI and suggest improvements to the next test case. Furthermore, the acquisition unit can analyze the results acquired by the generation AI in detail and suggest specific improvements. In addition, the acquisition unit can automatically suggest improvements to the next test case based on the results acquired by the generation AI. This improves the accuracy of the tests by suggesting improvements to the next test case. Improvements refer to, for example, methods for improving specific UI components or operation sequences. Some or all of the above processing in the acquisition unit may be performed using, for example, the generation AI, or without the generation AI. For example, the acquisition unit can input the acquired results into the generation AI and have the generation AI execute the improvement suggestions.

[0101] The acquisition unit can estimate the user's emotions and determine the priority of results to acquire based on the estimated emotions. For example, if the user is stressed, the acquisition unit can have the generating AI prioritize acquiring important results. If the user is relaxed, the acquisition unit can have the generating AI prioritize acquiring detailed results. Furthermore, if the user is in a hurry, the acquisition unit can have the generating AI prioritize acquiring results that can be acquired quickly. This allows for the determination of results prioritization according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or a generating AI. The generating AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the acquisition unit may be performed using a generating AI, or not using a generating AI. For example, the acquisition unit can input user emotion data into a generating AI and have the generating AI perform the determination of result prioritization based on emotions.

[0102] The acquisition unit can apply result acquisition methods that correspond to different devices and browsers. For example, the acquisition unit can apply result acquisition methods that correspond to different devices for the generating AI. Furthermore, the acquisition unit can apply result acquisition methods that correspond to different browsers for the generating AI. In addition, the acquisition unit can apply result acquisition methods that correspond to different combinations of devices and browsers for the generating AI. This makes it possible to acquire results that correspond to different devices and browsers. A device refers to a different type of device, such as a smartphone, tablet, or desktop, and a browser refers to a different type of browser, such as Chrome, Firefox, or Safari. Some or all of the above processing in the acquisition unit may be performed using the generating AI, for example, or without using the generating AI. For example, the acquisition unit can input the settings of different devices and browsers into the generating AI and have the generating AI execute the application of the result acquisition method.

[0103] The acquisition unit can provide feedback corresponding to different user profiles based on the acquired results. For example, the acquisition unit can analyze the results acquired by the generation AI and provide feedback corresponding to different user profiles. Furthermore, the acquisition unit can analyze the results acquired by the generation AI in detail and provide specific feedback corresponding to different user profiles. In addition, the acquisition unit can automatically provide feedback corresponding to different user profiles based on the results acquired by the generation AI. This allows for the provision of feedback tailored to different user profiles. A user profile refers, for example, to the operation patterns and settings of a specific user. Some or all of the above-described processing in the acquisition unit may be performed using, for example, the generation AI, or without using the generation AI. For example, the acquisition unit can input the acquired results into the generation AI and have the generation AI provide the feedback.

[0104] The management unit can estimate the user's emotions and adjust the display method of the results based on the estimated emotions. For example, if the user is stressed, the management unit can have the generating AI provide a concise display method. If the user is relaxed, the management unit can have the generating AI provide a detailed display method. Furthermore, if the user is in a hurry, the management unit can have the generating AI provide a method that allows for quick display. This allows the display method of the results to be adjusted according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or a generating AI. The generating AI is a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above processing in the management unit may be performed using a generating AI, for example, or not using a generating AI. For example, the management unit can input user emotion data into a generating AI and have the generating AI adjust the display method of the results based on the emotions.

[0105] The management unit can configure detailed functions for filtering and sorting the acquired results to improve management accuracy. For example, the management unit can filter the results acquired by the generating AI and prioritize the display of important results. The management unit can also sort the results acquired by the generating AI and adjust the display order based on specific conditions. Furthermore, the management unit can filter and sort the results acquired by the generating AI to improve management accuracy. This improves the accuracy of managing the acquired results. Filtering is the process of selecting data based on specific conditions, for example, and sorting is the process of rearranging data based on specific criteria. Some or all of the above processing in the management unit may be performed using the generating AI, for example, or without using the generating AI. For example, the management unit can input the acquired results into the generating AI and have the generating AI perform the filtering and sorting processes.

[0106] The management department can enhance its report generation function by visually displaying acquired results in graphs and charts. For example, the management department can display the results acquired by the generation AI in graphs, making them easier to understand visually. The management department can also display the results acquired by the generation AI in charts, making them easier to understand visually. Furthermore, the management department can enhance its report generation function by visually displaying the results acquired by the generation AI in graphs and charts. This improves the visual understanding of the acquired results. A method of visual display is, for example, the process of displaying data in a visual format such as graphs and charts. Some or all of the above-described processes in the management department may be performed using, for example, the generation AI, or not using the generation AI. For example, the management department can input acquired results into the generation AI and have the generation AI generate graphs and charts.

[0107] The management unit can estimate the user's emotions and adjust the display order of results based on the estimated emotions. For example, if the user is stressed, the management unit can have the generating AI prioritize displaying important results. If the user is relaxed, the management unit can have the generating AI prioritize displaying detailed results. Furthermore, if the user is in a hurry, the management unit can have the generating AI prioritize displaying results that can be displayed quickly. This allows the display order of results to be adjusted according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or a generating AI. The generating AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the management unit may be performed using a generating AI, or not using a generating AI. For example, the management unit can input user emotion data into a generating AI and have the generating AI adjust the display order of results based on emotions.

[0108] The management unit can apply result display methods that correspond to different devices and browsers. For example, the management unit can have the generating AI apply result display methods that correspond to different devices. The management unit can also have the generating AI apply result display methods that correspond to different browsers. Furthermore, the management unit can have the generating AI apply result display methods that correspond to different combinations of devices and browsers. This makes it possible to display results that correspond to different devices and browsers. A device refers to a different type of device, such as a smartphone, tablet, or desktop, and a browser refers to a different type of browser, such as Chrome, Firefox, or Safari. Some or all of the above processing in the management unit may be performed using the generating AI, for example, or without using the generating AI. For example, the management unit can input the settings of different devices and browsers into the generating AI and have the generating AI execute the application of the result display method.

[0109] The management department can generate reports corresponding to different user profiles based on the acquired results. For example, the management department can analyze the acquired results using a generation AI and generate reports corresponding to different user profiles. Furthermore, the management department can analyze the acquired results in detail using a generation AI and generate specific reports corresponding to different user profiles. In addition, the management department can automatically generate reports corresponding to different user profiles based on the acquired results using a generation AI. This allows for the generation of reports tailored to different user profiles. A user profile refers, for example, to the operation patterns and settings of a specific user. Some or all of the above-described processes in the management department may be performed using a generation AI, or without using a generation AI. For example, the management department can input the acquired results into a generation AI and have the generation AI generate the report.

[0110] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0111] The UI test automation system can further include the ability to estimate the user's emotions and adjust the execution order of test cases based on the estimated emotions. For example, if the user is stressed, the generative AI can prioritize the execution of important test cases. If the user is relaxed, the generative AI can prioritize the execution of detailed test cases. Furthermore, if the user is in a hurry, the generative AI can prioritize the execution of test cases that can be executed quickly. This allows the execution order of test cases to be adjusted according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. The generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the execution unit may be performed using a generative AI, or not using a generative AI. For example, the execution unit can input user emotion data into a generative AI and have the generative AI adjust the execution order of test cases based on emotions.

[0112] The UI test automation system can also have the functionality to apply test case execution methods to different network environments. For example, the generating AI can apply test case execution methods to different network environments. Furthermore, the generating AI can adjust the test case execution method considering the characteristics of different network environments. In addition, the generating AI can automatically apply test case execution methods to different network environments. This enables the execution of test cases that correspond to different network environments. Network environments refer to different types of networks, such as wired networks, wireless networks, and mobile networks. Some or all of the above-described processes in the execution unit may be performed using the generating AI, or without using the generating AI. For example, the execution unit can input the settings for different network environments into the generating AI and have the generating AI apply the test case execution method.

[0113] The UI test automation system can further include a function to estimate the user's emotions and adjust the level of detail of the results obtained based on the estimated emotions. For example, if the user is stressed, the generating AI can obtain a concise result. If the user is relaxed, the generating AI can obtain a detailed result. Furthermore, if the user is in a hurry, the generating AI can prioritize results that can be obtained quickly. This allows the level of detail of the results to be adjusted according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or a generating AI. The generating AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the acquisition unit may be performed using a generating AI, for example, or not using a generating AI. For example, the acquisition unit can input user emotion data into a generating AI and have the generating AI perform an adjustment of the level of detail of the results based on the emotions.

[0114] The UI test automation system can also include a function to adjust the execution of test cases based on different user privileges. For example, the generating AI can adjust the execution of test cases based on different user privileges. The generating AI can also execute detailed test cases for users with administrator privileges. Furthermore, the generating AI can execute basic test cases for users with general user privileges. This makes it possible to execute test cases according to different user privileges. User privileges refer to different types of privileges, such as administrator privileges and general user privileges. Some or all of the above processing in the execution unit may be performed using the generating AI, or it may be performed without using the generating AI. For example, the execution unit can input user privilege information into the generating AI and have the generating AI adjust the execution of test cases.

[0115] The UI test automation system can further include the ability to estimate the user's emotions and adjust the display method of the results based on the estimated emotions. For example, if the user is stressed, the generating AI can provide a concise display method. If the user is relaxed, the generating AI can provide a detailed display method. Furthermore, if the user is in a hurry, the generating AI can provide a method that allows for quick display. This allows the display method of the results to be adjusted according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or a generating AI. The generating AI is a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above processing in the management unit may be performed using a generating AI, for example, or not using a generating AI. For example, the management unit can input user emotion data into a generating AI and have the generating AI adjust the display method of the results based on the emotions.

[0116] The UI test automation system can also have the functionality to apply result acquisition methods that correspond to different devices and browsers. For example, the generating AI can apply result acquisition methods that correspond to different devices. Furthermore, the generating AI can apply result acquisition methods that correspond to different browsers. In addition, the generating AI can apply result acquisition methods that correspond to different combinations of devices and browsers. This makes it possible to acquire results that correspond to different devices and browsers. "Device" refers to different types of devices such as smartphones, tablets, and desktops, and "browser" refers to different types of browsers such as Chrome, Firefox, and Safari. Some or all of the above processing in the acquisition unit may be performed using the generating AI, or without using the generating AI. For example, the acquisition unit can input the settings of different devices and browsers into the generating AI and have the generating AI execute the application of the result acquisition method.

[0117] The UI test automation system can further include the ability to estimate the user's emotions and determine the priority of test cases to generate based on the estimated emotions. For example, if the user is stressed, the generating AI can prioritize generating important test cases. If the user is relaxed, the generating AI can prioritize generating detailed test cases. Furthermore, if the user is in a hurry, the generating AI can prioritize generating test cases that can be executed quickly. This allows for the prioritization of test cases according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or a generating AI. The generating AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the generation unit may be performed using a generating AI, or not. For example, the generation unit can input user emotion data into the generating AI and have the generating AI determine the priority of test cases based on emotions.

[0118] The UI test automation system can also be equipped with the ability to apply specific algorithms corresponding to different programming languages. For example, the generating AI can analyze code in different programming languages ​​and identify UI components. Furthermore, the generating AI can analyze attribute information in different programming languages ​​and identify UI components. In addition, the generating AI can analyze the code structure of different programming languages ​​and identify UI components. This makes it possible to identify UI components corresponding to different programming languages. Programming languages ​​refer to different languages ​​such as JavaScript, Python, and Java. Some or all of the above-described processes in the identification unit may be performed using the generating AI, or without using the generating AI. For example, the identification unit can input code in a different programming language into the generating AI and have the generating AI perform the identification of UI components.

[0119] The UI test automation system can further include the ability to estimate the user's emotions and determine the priority of UI components to identify based on the estimated emotions. For example, if the user is stressed, the generative AI can prioritize identifying important UI components. If the user is relaxed, the generative AI can prioritize identifying detailed UI components. Furthermore, if the user is in a hurry, the generative AI can prioritize identifying UI components that can be identified quickly. This allows for the determination of UI component priorities according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. The generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the identification unit may be performed using a generative AI, or not using a generative AI. For example, the identification unit can input user emotion data into the generative AI and have the generative AI perform the determination of UI component priorities based on emotions.

[0120] The UI test automation system can also be equipped with the ability to provide feedback corresponding to different user profiles. For example, it can analyze the results obtained by the generating AI and provide feedback corresponding to different user profiles. It can also analyze the results obtained by the generating AI in detail and provide specific feedback corresponding to different user profiles. Furthermore, it can automatically provide feedback corresponding to different user profiles based on the results obtained by the generating AI. This allows for the provision of feedback tailored to different user profiles. A user profile refers, for example, to the operation patterns and settings of a particular user. Some or all of the above processing in the acquisition unit may be performed using, for example, the generating AI, or without using the generating AI. For example, the acquisition unit can input the acquired results into the generating AI and have the generating AI perform the provision of feedback.

[0121] The following briefly describes the processing flow for example form 2.

[0122] Step 1: The identification unit identifies UI components. The identification unit uses a generation AI to analyze screenshots and code to identify UI components. Step 2: The generation unit automatically generates test cases based on the UI components identified by the identification unit. The generation unit uses generation AI to automatically generate test cases based on past test data and user operation history. Step 3: The execution unit executes the test cases generated by the generation unit. The execution unit uses the generation AI to execute the test cases and configures the environment to support different devices and browsers. Step 4: The acquisition unit acquires the results obtained by the execution unit. The acquisition unit acquires the results using the generation AI. Step 5: The management unit displays and manages the results acquired by the acquisition unit. The management unit has functions to manage, filter, and sort the results using generation AI. It also has a report generation function that visually displays the acquired results in graphs and charts.

[0123] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0124] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include text generation AI, image generation AI, and multimodal generation AI. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats from audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each of the above parts is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example.Furthermore, processing performed by AI, including generative AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI, including generative AI.

[0125] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0126] For example, the identification unit can identify UI components using the camera 42 and microphone 38B of the smart device 14. The generation unit can automatically generate test cases using the identification processing unit 290 of the data processing device 12. The execution unit can execute the generated test cases using the control unit 46A of the smart device 14. The acquisition unit can acquire the execution results using the identification processing unit 290 of the data processing device 12. The management unit can manage the results acquired by the control unit 46A of the smart device 14. The correspondence between each unit and the devices and control units is not limited to the example described above and can be modified in various ways.

[0127] [Second Embodiment] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0128] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0129] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0130] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0131] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0132] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0133] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0134] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing by the processor 28. The storage 32 stores the specific processing program 56.

[0135] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0136] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0137] In the smart glasses 214, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0138] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0139] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0140] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0141] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0142] For example, the identification unit can identify UI components using the camera 42 and microphone 238 of the smart glasses 214. The generation unit can automatically generate test cases using the identification processing unit 290 of the data processing device 12. The execution unit can execute the test cases generated by the control unit 46A of the smart glasses 214. The acquisition unit can acquire the execution results using the identification processing unit 290 of the data processing device 12. The management unit can manage the results acquired by the control unit 46A of the smart glasses 214. The correspondence between each unit and the devices and control units is not limited to the example described above and can be modified in various ways.

[0143] [Third Embodiment] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0144] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0145] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0146] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0147] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0148] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0149] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0150] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0151] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0152] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0153] In the headset terminal 314, specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads the specific program 60 from the storage 50 and executes the read specific program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific program 60 executed on the RAM 48. The headset terminal 314 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0154] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0155] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0156] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0157] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset terminal 314, but may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the headset terminal 314 or an external device, and the headset terminal 314 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0158] For example, the identification unit can identify UI components using the camera 42 and microphone 238 of the headset terminal 314. The generation unit can automatically generate test cases using the identification processing unit 290 of the data processing device 12. The execution unit can execute the generated test cases using the control unit 46A of the headset terminal 314. The acquisition unit can acquire the execution results using the identification processing unit 290 of the data processing device 12. The management unit can manage the results acquired by the control unit 46A of the headset terminal 314. The correspondence between each unit and the devices and control units is not limited to the example described above and can be modified in various ways.

[0159] [Fourth Embodiment] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0160] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0161] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0162] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0163] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0164] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS image sensor or CCD image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0165] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0166] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. The robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0167] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0168] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0169] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0170] In robot 414, specific processing is performed by processor 46. A specific program 60 is stored in storage 50. Processor 46 reads the specific program 60 from storage 50 and executes it on RAM 48. The specific processing is achieved by processor 46 acting as a control unit 46A according to the specific program 60 executed on RAM 48. Robot 414 also has data generation model 58 and emotion identification model 59, similar to those of the robot, and can perform processing similar to that of the specific processing unit 290 using these models.

[0171] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0172] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0173] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0174] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the robot 414 or an external device, and the robot 414 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0175] For example, the identification unit can identify UI components using the camera 42 and microphone 238 of the robot 414. The generation unit can automatically generate test cases using the identification processing unit 290 of the data processing device 12. The execution unit can execute the generated test cases using the control unit 46A of the robot 414. The acquisition unit can acquire the execution results using the identification processing unit 290 of the data processing device 12. The management unit can manage the results acquired by the control unit 46A of the robot 414. The correspondence between each unit and the devices and control units is not limited to the example described above and can be modified in various ways.

[0176] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0177] Figure 9 shows the emotion map 400, in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0178] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0179] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0180] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, and motorcycles, emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0181] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0182] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0183] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing method for the specific process may be used, which includes computer 22 and multiple other computers.

[0184] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0185] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0186] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0187] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0188] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0189] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0190] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0191] Furthermore, although the above-described examples were divided into four embodiments, some or all of these embodiments may be combined. Also, the smart device 14, smart glasses 214, headset terminal 314, and robot 414 are just examples, and they may be combined, or other devices may be used. Also, although the above-described examples were divided into two embodiments, Embodiment 1 and Embodiment 2, these may be combined.

[0192] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and other things that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0193] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0194] (Note 1) A part that identifies UI components, A unit that automatically generates test cases based on the UI components identified by the aforementioned specific unit, An execution unit that executes the test cases generated by the generation unit, An acquisition unit that acquires the results obtained by the execution unit, The system includes a unit that displays and manages a list of results obtained by the acquisition unit. A system characterized by the following features. (Note 2) The aforementioned management department, It has the ability to filter and sort the acquired results. The system described in Appendix 1, characterized by the features described herein. (Note 3) The aforementioned management department, It has a report generation function that visually displays the acquired results in graphs or charts. The system described in Appendix 1, characterized by the features described herein. (Note 4) The execution unit is, Configure settings to support different devices or browsers. The system described in Appendix 1, characterized by the features described herein. (Note 5) The specified part is, Identify UI components using screenshots or code analysis. The system described in Appendix 1, characterized by the features described herein. (Note 6) The generating unit is Automatically generate test cases based on past test data or user operation history. The system described in Appendix 1, characterized by the features described herein. (Note 7) The specified part is, It estimates the user's emotions and adjusts how UI components are identified based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 8) The specified part is, When analyzing screenshots, improve the accuracy of identification based on the color or shape of UI components. The system described in Appendix 1, characterized by the features described herein. (Note 9) The specified part is, During code analysis, the attribute information of UI components is specifically analyzed to improve identification accuracy. The system described in Appendix 1, characterized by the features described herein. (Note 10) The specified part is, It estimates the user's emotions and determines the priority of UI components based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 11) The specified part is, When analyzing screenshots, specific methods are applied to accommodate different resolutions and devices. The system described in Appendix 1, characterized by the features described herein. (Note 12) The specified part is, During code analysis, apply specific algorithms that correspond to different programming languages. The system described in Appendix 1, characterized by the features described herein. (Note 13) The generating unit is We estimate the user's emotions and adjust the test case generation method based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 14) The generating unit is Analyze past test data and generate test cases based on frequently failing patterns. The system described in Appendix 1, characterized by the features described herein. (Note 15) The generating unit is Detailed analysis of user operation history and generation of test cases based on specific operation sequences. The system described in Appendix 1, characterized by the features described herein. (Note 16) The generating unit is Estimate user emotions and prioritize test cases based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 17) The generating unit is Based on past test data, generate test cases that support different devices and browsers. The system described in Appendix 1, characterized by the features described herein. (Note 18) The generating unit is Based on the user's operation history, generate test cases corresponding to different user profiles. The system described in Appendix 1, characterized by the features described herein. (Note 19) The execution unit is, The system estimates the user's emotions and adjusts the execution order of test cases based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 20) The execution unit is, Detailed environment settings are configured to support different devices and browsers, improving execution accuracy. The system described in Appendix 1, characterized by the features described herein. (Note 21) The execution unit is, During test case execution, errors are detected in real time, and corrective suggestions are provided immediately. The system described in Appendix 1, characterized by the features described herein. (Note 22) The execution unit is, Estimate the user's emotions and determine the priority of test cases to execute based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 23) The execution unit is, Applying test case execution methods to different network environments The system described in Appendix 1, characterized by the features described herein. (Note 24) The execution unit is, When running test cases, adjust the execution based on different user permissions. The system described in Appendix 1, characterized by the features described herein. (Note 25) The acquisition unit is, It estimates the user's emotions and adjusts the level of detail in the results obtained based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 26) The acquisition unit is, The execution results of the test cases are analyzed in detail to identify the cause of specific errors. The system described in Appendix 1, characterized by the features described herein. (Note 27) The acquisition unit is, Based on the results obtained, we will propose improvements to the next test case. The system described in Appendix 1, characterized by the features described herein. (Note 28) The acquisition unit is, It estimates the user's emotions and determines the priority of the results to retrieve based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 29) The acquisition unit is, Apply a result retrieval method that is compatible with different devices and browsers. The system described in Appendix 1, characterized by the features described herein. (Note 30) The acquisition unit is, Based on the results obtained, provide feedback that corresponds to different user profiles. The system described in Appendix 1, characterized by the features described herein. (Note 31) The aforementioned management department, It estimates the user's emotions and adjusts how the results are displayed based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 32) The aforementioned management department, Detailed settings for filtering and sorting the retrieved results improve management accuracy. The system described in Appendix 1, characterized by the features described herein. (Note 33) The aforementioned management department, Enhance the report generation function to visually display acquired results in graphs and charts. The system described in Appendix 1, characterized by the features described herein. (Note 34) The aforementioned management department, It estimates the user's emotions and adjusts the display order of results based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 35) The aforementioned management department, Apply a results display method that is compatible with different devices and browsers. The system described in Appendix 1, characterized by the features described herein. (Note 36) The aforementioned management department, Based on the acquired results, generate reports that correspond to different user profiles. The system described in Appendix 1, characterized by the features described herein. [Explanation of Symbols]

[0195] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots

Claims

1. A part that identifies UI components, A generation unit that automatically generates test cases based on the UI components identified by the aforementioned specific unit, An execution unit that executes the test cases generated by the generation unit, An acquisition unit that acquires the results obtained by the execution unit, The system includes a management unit that displays and manages a list of results obtained by the acquisition unit. A system characterized by the following features.

2. The aforementioned management department, It has the ability to filter and sort the acquired results. The system according to feature 1.

3. The aforementioned management department, It has a report generation function that visually displays the acquired results in graphs or charts. The system according to feature 1.

4. The execution unit is, Configure settings to support different devices or browsers. The system according to feature 1.

5. The specified part is, Identify UI components using screenshots or code analysis. The system according to feature 1.

6. The generating unit is Automatically generate test cases based on past test data or user operation history. The system according to feature 1.

7. The specified part is, It estimates the user's emotions and adjusts how UI components are identified based on those estimated emotions. The system according to feature 1.

8. The specified part is, When analyzing screenshots, improve the accuracy of identification based on the color or shape of UI components. The system according to feature 1.

9. The specified part is, During code analysis, the attribute information of UI components is specifically analyzed to improve identification accuracy. The system according to feature 1.

Images