system

Abstract

A system is provided.SOLUTION: A system comprising: means for a user to upload a URL or file of an application or web page under test; means for analyzing a screenshot of the test object using a generative AI model to identify UI elements; means for automatically generating test cases based on the identified UI elements and storing the test cases in a database; means for terminals to receive the test cases and automatically perform operations; and means for analyzing test results sent from the terminals and generating reports to provide to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Manual UI testing consumes a significant amount of time and effort, and there is a high risk of human error. It is also extremely difficult to keep up with continuous changes and updates to applications and websites. These issues have a negative impact on the efficiency of the entire development process and the quality of the product. Therefore, there is a need to solve these issues and realize automation and efficiency in UI testing. [Means for solving the problem]

[0005] The present invention provides a UI test automation system that utilizes a generative AI model. This system solves these problems by including a means for a user to upload the URL or file of an application or web page to be tested, a means for analyzing screenshots of the test target using a generative AI model to identify UI elements, a means for automatically generating test cases based on the identified UI elements and saving them in a database, a means for a terminal to receive the test cases and automatically execute operations, and a means for analyzing test results sent from the terminal, generating reports, and providing them to the user. Furthermore, the system achieves more efficient and flexible UI test automation by providing a function for the generative AI model to adapt test cases to different devices and environments, and a means for the user to select configuration options and configure the device or environment to be tested based on those options.

[0006] "Test target" refers to the application or web page that the user specifies for UI testing.

[0007] A "generative AI model" is an artificial intelligence model that uses machine learning technology to automate the recognition of UI elements and the generation of test cases.

[0008] A "screenshot" is a capture of the screen contents of a specified application or web page in image format.

[0009] A "UI element" is a user interface element on an application or web page, including buttons, text fields, links, etc.

[0010] A "test case" is a scenario that is automatically generated in UI testing, and defines what operations should be performed on which UI elements.

[0011] A "database" is a system for storing and managing systematically structured information, and is used to store test cases and analysis results.

[0012] A "terminal" is an execution environment for executing test cases, and refers to a computer or device that automatically performs operations based on user instructions.

[0013] A "report" is a document that analyzes test results and includes information such as success / failure, error logs, and improvement suggestions.

[0014] A "device" is a hardware device used to display and operate the application or web page being tested, including PCs, smartphones, tablets, etc.

[0015] "Environment" refers to the software and hardware configuration conditions under which the test is run, including the operating system, browser version, screen resolution, etc. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The system of the present invention automates UI testing of applications and websites using generative AI models. The system includes three main components: the user, the server, and the terminal.

[0038] First, the user uploads the URL of the application or website or application file to be tested to the server, clearly specifying what should be tested. The user can then further specify specific devices and environments (e.g. different browsers or resolutions).

[0039] The server then receives the specified URL or uploaded file and uses a browser emulator to capture a screenshot, which is then analyzed by the generative AI model. As a result of the analysis, UI elements (buttons, input fields, links, etc.) are identified and stored in a database along with their location information.

[0040] The server then automatically generates test cases based on the detected UI elements. For example, if a login screen is detected, it generates a scenario that says "Enter username and password and click login button." The generated test cases are saved in a database.

[0041] The terminal receives these test cases from the server and executes them on the device and environment specified by the user. The terminal automatically performs operations according to the test cases, taking screenshots and error logs as operation results, and sending the results to the server.

[0042] The server analyzes the received test results and determines whether they passed or failed. It also generates a detailed report of any errors or the impact of UI changes, which is provided to the user. This report includes a summary of success or failure, an error log, and suggestions for improvement. Based on this, the user can make necessary corrections and plan retesting.

[0043] As a concrete example, consider the case where a user wants to test a login screen. The user enters the URL of the login screen into the server. The server retrieves a screenshot from the URL and analyzes it with a generative AI model to identify the username field, password field, and login button. Based on this, the server generates a test case that says, "Enter username and password and click the login button."

[0044] The terminal executes this test case and performs operations on the specified device and environment. The operation results are sent to the server, which determines whether the operation was successful or not and generates a detailed report. The user can check this report to confirm that the login operation was successful.

[0045] The above is a specific embodiment for carrying out the present invention. This system makes it possible to automate UI tests and execute them efficiently and accurately.

[0046] The processing flow will be explained below.

[0047] Step 1:

[0048] The user uploads the URL or file of the application or web page to be tested to the server, and also selects the device and environment configuration options required for the test execution.

[0049] Step 2:

[0050] The server stores the URL or file received from the user in a database. The server configures the browser emulator based on the specified device and environment information.

[0051] Step 3:

[0052] The server uses a browser emulator to open the specified URL, captures a screenshot, and saves it in the appropriate resolution.

[0053] Step 4:

[0054] The server uses the generative AI model to analyze the captured screenshot and identify UI elements (buttons, input fields, links, etc.) The server then stores the UI elements identified as a result of the analysis and their location information in a database.

[0055] Step 5:

[0056] The server automatically generates test cases based on detected UI elements. For example, if a login screen is detected, it generates a scenario that says "Enter your username and password and click the login button." The generated test cases are saved in a database.

[0057] Step 6:

[0058] The terminal receives the test cases from the server and prepares them for execution on the device and environment specified by the user. The terminal prepares the environment according to the specified setting conditions.

[0059] Step 7:

[0060] The device automatically performs UI operations according to the received test cases, such as entering text into text fields and clicking buttons, and records any screen changes or errors that occur during the operation.

[0061] Step 8:

[0062] The device sends screenshots and error logs containing the test results to the server, providing information on which operations were successful and which failed, along with the results of the operations.

[0063] Step 9:

[0064] The server analyzes the test results sent from the device, determines whether the test succeeded or failed, and analyzes the cause of errors and the impact of UI changes.

[0065] Step 10:

[0066] The server generates a detailed report based on the analysis results, including a summary of successes / failures, error logs, and suggestions for improvement, and uploads it to the user's dashboard.

[0067] Step 11:

[0068] Users can access a dashboard to view detailed reports of tests that have been performed, allowing them to plan any necessary corrections or retests.

[0069] Example 1

[0070] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0071] Traditional methods for testing the user interface (UI) of applications and websites require a significant amount of manual work, which is time-consuming and costly. It is also difficult to perform consistent testing across different devices and environments, making it difficult to ensure the reliability and efficiency of testing. This often results in extended release times.

[0072] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0073] In this invention, the server includes a means for a user to upload a URL or file of an application or webpage to be tested, a means for analyzing screenshots of the test target using a generative AI model and identifying user interface elements, a means for automatically generating test cases based on the identified user interface elements and saving them in a database, a means for a terminal to receive the test cases and automatically execute operations, a means for analyzing test results sent from the terminal, generating a report and providing it to the user, and a means for executing tests on different devices and environments. This enables efficient and accurate automation of UI testing of applications and websites.

[0074] "User" means the person or entity who uploads URLs and files to the server, selects configuration options, and receives reports to run tests on an application or website.

[0075] "Server" is a computer system that receives URLs and files submitted by users, analyzes screenshots using a generative AI model to identify user interface elements, generates test cases, stores them in a database, analyzes test results, and generates reports.

[0076] A "generative AI model" is an artificial intelligence model used to analyze captured screenshots and identify user interface elements.

[0077] A "terminal" is a device that receives test cases sent from a server, automatically executes the operations, and sends the results to the server.

[0078] A "test case" is a specific scenario or procedure that is generated based on identified user interface elements and that is used to perform automated testing.

[0079] "User interface elements" are interactive elements such as buttons, input fields, and links that appear on the screen of an application or website.

[0080] A "screenshot" is an image that captures the display state of a specified URL or uploaded file.

[0081] A "device" is a physical or virtual hardware environment for running test cases.

[0082] An "environment" is the settings and conditions for running a test (e.g., browser type, screen resolution, etc.).

[0083] A "report" is a document generated after analyzing test results, which includes a summary of successes / failures, an error log, and suggestions for improvement.

[0084] The system of the present invention automates user interface (UI) testing of applications and websites using generative AI models. In this system, a user specifies a test target, a server and a terminal work together to execute the test, analyze the results, and provide a report. An embodiment of this system is described in detail below.

[0085] First, the user uploads the URL or file of the application or website to be tested to the server. This operation clearly specifies the target to be tested. The upload is performed by accessing the server interface using a standard web browser and operating the "Enter URL" or "Upload file" button. The user can also specify specific devices and environments (e.g., different browsers or resolutions). This sets up the test environment.

[0086] Next, the server receives the specified URL or uploaded application file and takes a screenshot using a browser emulator (e.g., Selenium). Selenium is a browser automation tool that opens the specified URL and captures a screenshot of the screen. The server takes this screenshot and passes it to the generative AI model.

[0087] The server uses a generative AI model to analyze the captured screenshot and identify user interface elements (buttons, input fields, links, etc.). The AI ​​model analyzes the image data of the screenshot and identifies the location and attributes of the UI elements. The analysis results are stored in a database.

[0088] The server then automatically generates test cases based on the identified user interface elements. For example, if a login screen is detected, it generates a test case such as "Enter username and password and click login button." This generated test case is stored in a database and later sent to the device.

[0089] The terminal executes the test cases received from the server on the device and environment specified by the user. Using a testing framework (e.g., Appium), it launches an emulator for the specified device and automatically executes operations according to the test case scenario. The terminal captures screenshots and error logs as operation results and sends them to the server.

[0090] The server analyzes the test results sent from the device and determines whether the test succeeded or failed. Specifically, it compares the expected UI state with the actual operation result, and records any discrepancies as an error. A detailed report is generated based on the analysis results and provided to the user. This report includes a summary of success or failure, an error log, and suggestions for improvement. Based on this report, the user can make any necessary corrections and plan retesting.

[0091] As a concrete example, consider the case where a user wants to test a login screen. The user enters the URL of the login screen into the server and specifies the browser and resolution. The server uses Selenium to obtain a screenshot from the specified URL, and analyzes it with a generative AI model to identify the username field, password field, and login button. Based on this, a test case is generated that says "Enter username and password and click the login button." The device executes this test case and sends the results to the server. The server analyzes the results, generates a report, and provides it to the user.

[0092] An example of a prompt statement might be something like, "Enter the URL of the login screen into the server, perform the operations according to the generated test case, and generate a prompt to evaluate the results."

[0093] By implementing the invention in this manner, it is possible to automate UI testing of applications and websites efficiently and accurately.

[0094] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0095] Step 1:

[0096] The user specifies the test target

[0097] Input: The URL or application file to test.

[0098] Output: The URL or file received by the server.

[0099] Specific operation: A user accesses the server interface using a web browser and operates the "Enter URL" or "Upload file" button. The user enters or uploads the target URL or file and presses the send button. The server receives the sent URL or file.

[0100] Step 2:

[0101] The user specifies the test environment

[0102] Input: Preferences such as device, browser, and resolution.

[0103] Output: The configuration information received by the server.

[0104] What happens: The user selects the appropriate option from the "Device Selection" or "Browser Selection" drop-down list on the server's interface. The user's selection is sent to the server, which receives this preference information.

[0105] Step 3:

[0106] Server takes screenshot

[0107] Input: URLs, application files, and configuration information submitted by the user.

[0108] Output: The captured screenshot image.

[0109] What happens: The server launches a browser emulator, such as Selenium, to open the specified URL or file. The browser emulator opens the URL with the specified configuration (e.g., browser type and resolution) and captures a screenshot of the screen. The server saves the screenshot.

[0110] Step 4:

[0111] The server analyzes the UI elements

[0112] Input: The captured screenshot image.

[0113] Output: Information about the identified UI element (position and attributes).

[0114] How it works: The server passes a screenshot as input to the generative AI model. The generative AI model analyzes the image, identifies UI elements (e.g., buttons, input fields, links, etc.), and extracts their locations and attributes. The analysis results are stored in a database.

[0115] Step 5:

[0116] The server generates the test cases

[0117] Input: Information about the identified UI element.

[0118] Output: Generated test cases.

[0119] Specific operation: The server automatically generates test cases based on UI element information retrieved from the database. For example, if a login screen is detected, a test scenario will be created that says "Enter username and password and click login button." The generated test cases are saved in the database.

[0120] Step 6:

[0121] The device runs the test case

[0122] Input: Test cases received from the server.

[0123] Output: Operation results (screenshots and error logs).

[0124] Specific operation: The terminal executes the test cases received from the server on the device and environment specified by the user. The terminal uses a testing framework (e.g., Appium) to launch the emulator for the specified device and automatically perform operations according to the test case scenario. The terminal captures screenshots and error logs as operation results and sends them to the server.

[0125] Step 7:

[0126] The server analyzes the test results

[0127] Input: Operation results sent from the device (screenshots and error logs).

[0128] Output: Analysis result (success / failure).

[0129] Specific operation: The server analyzes the operation results sent from the device. It compares the expected UI state (for example, whether a button is clickable or a form is submitted) with the actual operation result. This comparison determines whether the test is successful or failed.

[0130] Step 8:

[0131] The server generates the report

[0132] Input: Analysis data of test results.

[0133] Output: Detailed report (success / failure summary, error log, improvement suggestions).

[0134] Specific operation: The server generates a detailed report based on the analysis results, including a summary of the test success / failure, a log of errors that occurred, and suggestions for improvement. The generated report is then provided to the user.

[0135] (Application example 1)

[0136] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0137] Product picking work at logistics centers requires a great deal of time and effort, and is prone to human error. In particular, it is difficult to quickly and accurately pick large quantities of products in a large warehouse, so there is a need to improve work efficiency and reduce errors.

[0138] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0139] In this invention, the server includes a means for a user to upload a URL or file of an application or webpage to be tested, a means for analyzing screenshots of the test target using a generative AI model and identifying UI elements, a means for automatically generating test cases based on the identified UI elements and saving them in a database, a means for a terminal to receive the test cases and automatically execute operations, a means for analyzing test results sent from the terminal, generating a report and providing it to the user, and a means for supporting product picking using smart glasses at a logistics center, thereby enabling improved efficiency and accuracy of picking work at logistics centers.

[0140] "User" refers to the person who uploads the URL or file of the application or web page to be tested and reviews the generated test cases and reports.

[0141] "Generative AI Model" refers to an artificial intelligence model used to analyze a given screenshot and identify UI elements.

[0142] A "test case" refers to a scenario that is automatically generated based on identified UI elements to verify the operation and behavior of the system.

[0143] "Terminal" refers to a device or equipment that receives the generated test cases and automatically executes the operations.

[0144] "Smart glasses" refer to devices worn by workers at logistics centers that display information in real time to assist with picking operations.

[0145] "Picking" refers to the task of removing specified products from shelves at the logistics center and preparing them for shipment.

[0146] "Report" refers to a detailed report provided to the User after analyzing the test results sent from the Terminal.

[0147] A "logistics center" refers to a facility that stores, manages, and ships goods.

[0148] This invention is a system that utilizes a generative AI model and smart glasses to improve the efficiency and accuracy of product picking work in logistics centers. Specifically, the system uses a generative AI model to identify the location of products and shelves, and displays instructions to workers through the smart glasses.

[0149] First, the user uploads the URL or file of the application or web page to be tested to the server, which then recognizes the test target using the generative AI model.

[0150] The server then receives the specified URL or file and captures screenshots from a browser emulator or a real-world distribution center's surveillance cameras. These screenshots are then analyzed by a generative AI model to identify UI elements and shelf and product locations. This data is then stored in a database, and test cases are automatically generated based on the identified UI elements.

[0151] The generated test cases are received by a device (such as smart glasses) and automatically performed in the set environment. The smart glasses are worn by workers and display real-time information on shelves and product locations, as well as picking instructions. This display allows workers to accurately locate products in a short amount of time and perform picking tasks.

[0152] The results of operations performed by the terminal (e.g., the progress of a picking operation) are sent to the server, which analyzes them and generates a report that includes a success / failure summary, an error log, and suggestions for improvement, which the user can review and correct or retest.

[0153] The specific technologies used are OpenCV for image processing and machine learning models for generative AI models. The smart glasses used are commercially available (e.g., Vuzix, Google Glass Enterprise Edition). The entire system is built by combining a browser emulator, a database management system, and a communication protocol.

[0154] An example of a prompt is:

[0155] "To support product picking in the warehouse, please recognize the product and its location from the captured image. Specifically, please analyze the label or barcode and indicate the corresponding product."

[0156] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0157] Step 1:

[0158] The user uploads the URL or file of the application or web page to be tested to the server. This input includes the URL or file path, based on which the server identifies the target for testing.

[0159] Step 2:

[0160] The server receives the specified URL or file and captures a screenshot from a browser emulator or the actual surveillance camera at the distribution center. The input is the specified URL or file, and the captured screenshot is generated as the output. Specifically, the specified URL is opened using a browser emulator and the screen is captured.

[0161] Step 3:

[0162] The server uses a generative AI model to analyze the captured screenshot and identify UI elements, shelves, and product locations. The input is a screenshot, and the output is the identified UI elements, shelves, and product locations. This analysis uses an image recognition method using a machine learning model.

[0163] Step 4:

[0164] The server automatically generates test cases based on the identified UI elements and shelf / product locations and saves them in a database. The input is data on UI elements and shelf / product locations, and the generated test cases are obtained as output. Specifically, a test case is created that specifies which product to pick from which shelf based on the product's barcode and label information.

[0165] Step 5:

[0166] The terminal (smart glasses) receives the test cases generated from the server and automatically executes the operations. The test cases are input and the operation results are output. The smart glasses display shelf and product location information and picking instructions in real time.

[0167] Step 6:

[0168] The terminal sends the results of the operation it performed to the server. The input is the operation result of the smart glasses, and the output is the operation result data sent to the server. This operation result includes the progress of the picking work and an error log.

[0169] Step 7:

[0170] The server analyzes the received operation results, generates a report, and provides it to the user. The operation result data is input, and a report is generated as output. This report includes a summary of success / failure, an error log, and suggestions for improvement. The user can make corrections or retest based on this report.

[0171] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0172] Our system not only automates UI testing using generative AI models, but also combines an emotion engine to recognize user emotions and dynamically adjust the testing process accordingly. This system consists of four main components: the user, the server, the device, and the emotion engine.

[0173] First, the user uploads the URL or file of the application or web page to be tested to the server. The user also selects the device and environment configuration options required for the test execution. The emotion engine then analyzes the user's facial expressions and tone of voice to recognize the user's emotional state.

[0174] The server then receives the specified URL or uploaded file and uses a browser emulator to capture a screenshot, which is then analyzed by the generative AI model. As a result of the analysis, UI elements (buttons, input fields, links, etc.) are identified and stored in a database along with their location information.

[0175] The server then automatically generates test cases based on the detected UI elements. For example, if a login screen is detected, it generates a scenario that says "Enter username and password and click login button." The generated test cases are saved in a database.

[0176] The device receives these test cases from the server and prepares to run them on the device and environment specified by the user. The device automatically performs operations according to the test cases, taking screenshots and error logs as operation results. The obtained results are sent to the server.

[0177] The server analyzes the received test results, determines whether they passed or failed, and generates a detailed report of any errors or the impact of UI changes, which is then provided to the user. This report includes a summary of success or failure, an error log, and suggestions for improvement.

[0178] The emotion engine analyzes the user's emotional state in real time and dynamically changes the priority of test cases based on that information. For example, if the user is feeling stressed, the system will adjust the priority of test cases so that more important tests are executed first. The user's emotional state is also reflected in the test result report and is used for analysis and improvement suggestions.

[0179] As a concrete example, consider the case where a user wants to test a login screen. The user enters the URL of the login screen into the server. The server retrieves a screenshot from the URL and analyzes it with a generative AI model to identify the username field, password field, and login button. Based on this, the server generates a test case that says, "Enter username and password and click the login button."

[0180] The device runs the test cases and performs operations on the specified device and environment. The results of the operations are sent to the server, which determines whether the operation was successful or not and generates a detailed report. The emotion engine analyzes the user's facial expressions and tone of voice, and if it determines that the user is feeling stressed, it adds that information to the report and makes suggestions for improvement.

[0181] The above is a specific embodiment of the present invention, which not only automates UI testing and executes it efficiently and accurately, but also dynamically adjusts the testing process by taking into account the user's emotional state.

[0182] The processing flow will be explained below.

[0183] Step 1:

[0184] The user uploads the URL or file of the application or web page to be tested to the server, and then selects the device and environment configuration options required for the test execution.

[0185] Step 2:

[0186] The emotion engine analyzes the user's facial expressions and tone of voice to recognize the user's emotional state, and stores this information in a database within the system.

[0187] Step 3:

[0188] The server stores the URL or file received from the user in a database. The server configures the browser emulator based on the specified device and environment information.

[0189] Step 4:

[0190] The server uses a browser emulator to open the specified URL, captures a screenshot, and saves it in the appropriate resolution.

[0191] Step 5:

[0192] The server uses the generative AI model to analyze the captured screenshot and identify UI elements (buttons, input fields, links, etc.) The server then stores the UI elements identified as a result of the analysis and their location information in a database.

[0193] Step 6:

[0194] The server automatically generates test cases based on detected UI elements. For example, if a login screen is detected, it generates a scenario that says "Enter your username and password and click the login button." The generated test cases are saved in a database.

[0195] Step 7:

[0196] The terminal receives the test cases from the server and prepares them for execution on the device and environment specified by the user. The terminal prepares the environment according to the specified setting conditions.

[0197] Step 8:

[0198] The device automatically performs UI operations according to the received test cases, such as entering text into text fields and clicking buttons, and records any screen changes or errors that occur during the operation.

[0199] Step 9:

[0200] The device sends screenshots and error logs containing the test results to the server, providing information on which operations were successful and which failed, along with the results of the operations.

[0201] Step 10:

[0202] The server analyzes the test results sent from the device. The server determines whether the test was successful or failed, and analyzes the cause of any errors and the impact of UI changes. The emotion engine also references the user's emotional state and reflects that information in the test result report.

[0203] Step 11:

[0204] The server generates a detailed report based on the analysis results. The report includes a summary of success / failure, an error log, and suggestions for improvement. The report also incorporates data from the emotion engine, and includes the emotions felt by the user during the test. This report is then uploaded to the user's dashboard.

[0205] Step 12:

[0206] Users can access the dashboard and view detailed reports of the tests they have performed. Based on the reports, users can make necessary corrections and plan retests. The reports also reflect the user's emotional state, allowing for more accurate improvements.

[0207] Example 2

[0208] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0209] Conventional UI test automation systems have limited adaptability to multiple devices and environments, and have faced challenges in analyzing test results and generating reports efficiently. Furthermore, they do not dynamically adjust the test process to take into account the user's emotional state. This often leads to user frustration, and improvements to the UX are needed.

[0210] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0211] In this invention, the server includes: a means for a user to upload a URL or file of an application or webpage to be tested; a means for analyzing screenshots of the test target using a generative AI model to identify UI elements; a means for automatically generating test cases based on the identified UI elements and saving them in a database; a means for a terminal to receive the test cases and automatically execute operations; a means for analyzing test results sent from the terminal, generating a report and providing it to the user; and a means for recognizing the user's emotional state in real time and dynamically adjusting the test process based on the result. This improves the efficiency and accuracy of UI testing and enables dynamic adjustment of the test process taking the user's emotional state into account.

[0212] "User" means a person or entity that utilizes the system to upload the URL or file of an application or web page to be tested and select the configuration options required to run the test.

[0213] "Test subject" refers to the URL or file of an application or web page that a user uploads to the system, and refers to the entire software asset that is the subject of testing.

[0214] A "generative AI model" is an artificial intelligence model that analyzes screenshots to identify UI elements and automatically generate test cases.

[0215] A "screenshot" is a screen image of the application or web page under test that the system captures using a browser emulator.

[0216] A "UI element" is a user interface component (e.g., button, input field, link, etc.) identified from a screenshot by a generative AI model.

[0217] A "test case" is a set of operational scenarios that the system executes, automatically generated based on identified UI elements.

[0218] A "database" is an information storage means for saving information such as identified UI elements and generated test cases.

[0219] A "terminal" is an electronic device that executes test cases received from a server on a specified device or in a specified environment.

[0220] A "report" is a detailed report that analyzes the test results sent from the device and includes a success / failure determination, error log, and improvement suggestions.

[0221] "Emotional state" refers to the user's psychological state, recognized in real time by analyzing the user's facial expressions and tone of voice.

[0222] An "emotion engine" is an analysis device or software for analyzing a user's emotional state in real time and dynamically adjusting the testing process based on that analysis.

[0223] MODE FOR CARRYING OUT THE INVENTION

[0224] Our system automates user interface (UI) testing using generative AI models and combines them with an emotion engine to recognize user emotions and dynamically adjust the testing process accordingly. The system consists of four main components: the user, the server, the device, and the emotion engine.

[0225] server

[0226] The server receives the URL or file of the application or web page to be tested uploaded by the user. Specifically, the server uses a browser emulator (e.g., Selenium or Puppeteer) to take a screenshot of the specified URL or file. This screenshot is analyzed by a generative AI model (e.g., OpenAI's GPT-3) to identify UI elements (e.g., buttons, input fields, links, etc.). The location information of the UI elements obtained as a result of the analysis is stored in a database. The server also automatically generates test cases based on the identified UI elements, and the generated test cases are also stored in the database. The server then analyzes the test results sent from the device and determines whether they succeeded or failed. A detailed report is generated about errors and the impact of UI changes and provided to the user.

[0227] Terminal

[0228] The terminal executes the test cases received from the server on the device and environment specified by the user. Specifically, the terminal automatically performs operations according to the test cases and captures the operation results such as screenshots and error logs. The captured results are then sent to the server.

[0229] User

[0230] The user uploads the URL or file of the application or web page to be tested to the server, and then selects the device and environment configuration options required for the test execution. In addition, the emotion engine monitors the user's emotional state in real time. This is achieved by analyzing the user's facial expressions and tone of voice.

[0231] Emotion Engine

[0232] An emotion engine (e.g., Microsoft Azure Cognitive Services) analyzes the user's emotional state in real time and dynamically changes the priority of test cases. For example, if the user is feeling stressed, the system will change the priority of test cases and adjust the system so that more important tests are executed first. The user's emotional state is also reflected in the test result report and is used for analysis and improvement suggestions.

[0233] Specific examples

[0234] For example, if a user wants to test a login screen, they enter the URL of the login screen into the server. The server receives the URL and uses a browser emulator to take a screenshot of the login screen. This screenshot is analyzed by a generative AI model to identify the UI elements: username field, password field, and login button. Based on this, the server automatically generates a test case that says, "Enter username and password and click the login button."

[0235] The device runs this test case, performing operations on the specified device and environment. Screenshots and error logs obtained as a result of the operations are sent to the server, which determines whether the operation was successful or failed and generates a detailed report. The emotion engine analyzes the user's facial expressions and tone of voice, and if it determines that the user is feeling stressed, it adds that information to the report and makes suggestions for improvement.

[0236] Prompt Sentence Examples

[0237] "Write a program like the one below. A user enters the URL of a login screen into the server. The server receives the URL and takes a screenshot of the login screen using a browser emulator. The server analyzes this screenshot with a generative AI model and generates a test case (enter username and password and click the login button) based on the identified UI elements (username field, password field, login button). The generated test case is then sent to the terminal. The terminal executes the test case on the specified device and environment and sends the obtained operation results to the server. The server determines success / failure and generates a detailed report. The emotion engine analyzes the user's emotional state in real time and adjusts the testing process."

[0238] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0239] Step 1: User uploads URL or file to test

[0240] Specific behavior:

[0241] The user uploads the URL or file of the application or web page to be tested to the server, and also selects the device and environment configuration options required for the test execution.

[0242] input:

[0243] Test target URL or file, device and environment configuration options

[0244] output:

[0245] URLs or files and configuration data stored on the server

[0246] Step 2: The server receives the data and takes a screenshot

[0247] Specific behavior:

[0248] The server receives the URL or file provided by the user, then uses a browser emulator (e.g., Selenium, Puppeteer) to take a screenshot of the specified URL or file.

[0249] input:

[0250] URL or file from the user

[0251] output:

[0252] Captured screenshots

[0253] Step 3: Generate screenshots and analyze them with an AI model to identify UI elements

[0254] Specific behavior:

[0255] The server sends the captured screenshot to a generative AI model (e.g., OpenAI's GPT-3) to identify UI elements (buttons, input fields, links, etc.).

[0256] input:

[0257] Captured screenshots

[0258] output:

[0259] The location of the identified UI element

[0260] Step 4: The server generates the test cases and saves them in the database

[0261] Specific behavior:

[0262] The server automatically generates test cases based on the identified UI elements and saves them in a database. For example, if a login screen is detected, a scenario such as "Enter username and password and click login button" is generated.

[0263] input:

[0264] Location information of UI elements

[0265] output:

[0266] Generated test cases, saved test cases

[0267] Step 5: The device receives the test case and prepares to execute it

[0268] Specific behavior:

[0269] The terminal analyzes the test cases received from the server and prepares to execute the test cases on the device and in the environment specified by the user.

[0270] input:

[0271] Test cases received from the server

[0272] output:

[0273] Test cases ready to run

[0274] Step 6: The device performs the operation according to the test case

[0275] Specific behavior:

[0276] The device automatically performs operations according to the test cases, and captures screenshots and error logs as a result of the operations.

[0277] input:

[0278] Test cases ready to run

[0279] output:

[0280] Obtained operation results (screenshots, error logs)

[0281] Step 7: The server analyzes the test results and generates a report

[0282] Specific behavior:

[0283] The server analyzes the test results sent from the device and determines whether the test succeeded or failed. It also generates a detailed report on the impact of errors and UI changes and provides it to the user.

[0284] input:

[0285] Operation results sent from the device

[0286] output:

[0287] Success / failure results and detailed reports

[0288] Step 8: The emotion engine analyzes the user's emotional state and adjusts the process

[0289] Specific behavior:

[0290] The emotion engine analyzes the user's facial expressions and tone of voice to recognize the user's emotional state in real time, and dynamically reprioritizes test cases and adjusts the testing process accordingly.

[0291] input:

[0292] The user's facial expression and tone of voice

[0293] output:

[0294] Analyzed emotional states, adjusted testing process

[0295] (Application example 2)

[0296] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0297] Current UI testing automation systems do not consider the user's emotional state during the testing process, which can lead to a poor user experience. Furthermore, the testing process may not proceed effectively when the user is under stress. Furthermore, because electronic payment services often involve situations where users are prone to experiencing high levels of stress, a more advanced testing system incorporating emotion recognition is needed.

[0298] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to upload a URL or file of an application or webpage to be tested; means for analyzing a screenshot of the test target using a generative AI model and identifying UI elements; means for automatically generating test cases based on the identified UI elements and saving them in a database; means for a terminal to receive the test cases and automatically execute operations; means for analyzing test results sent from the terminal, generating a report, and providing it to the user; and means for recognizing the user's emotional state using an emotion engine that analyzes the user's facial expressions and tone of voice and dynamically changing the priority of test cases according to the emotional state. This enables advanced UI testing automation that takes the user's emotional state into consideration, making it possible to provide a high-quality user experience even in electronic payment services.

[0299] "User" is the person or system that uploads the URL or file of the application or web page under test and initiates the testing process.

[0300] "Generative AI model" refers to an artificial intelligence model used to analyze screenshots and identify UI elements.

[0301] A "screenshot" is an image captured from the screen of the application or web page being tested.

[0302] A "UI element" is an operable component in a user interface, such as a button, input field, or link.

[0303] A "test case" is a scenario of operations that is executed to specifically test a system.

[0304] A "database" is a digital storage for storing information such as test cases and test results.

[0305] A "terminal" is a device that receives test cases and automatically executes the operations.

[0306] An "emotion engine" is a system that analyzes a user's facial expressions and tone of voice to recognize the user's emotional state in real time.

[0307] An "emotional state" is an emotional state that a user is feeling, such as stress, joy, or impatience.

[0308] "Priority" is a concept that indicates the importance and order in which test cases are executed.

[0309] "Dynamic change" means changing the order and importance of test cases in real time according to the user's emotional state.

[0310] A "report" is a document or file that summarizes the test results and the analysis of the user's emotional state.

[0311] The system of the present invention not only automates UI testing using a generative AI model, but also combines it with an emotion engine to recognize user emotions and dynamically adjust the testing process accordingly. Specific embodiments are described below.

[0312] The system consists of four main components: the user, the server, the device, and the emotion engine. First, the user uploads the URL or file of the application or web page to be tested to the server. The user also selects the device and environment configuration options required for the test execution. The emotion engine then analyzes the user's facial expressions and tone of voice to recognize the user's emotional state.

[0313] The server receives the specified URL or uploaded file and takes a screenshot using a browser emulator. This screenshot is then analyzed by a generative AI model. As a result of the analysis, UI elements (buttons, input fields, links, etc.) are identified and stored in a database along with their location information. The server then automatically generates test cases based on the detected UI elements. For example, if a login screen is detected, a scenario such as "Enter your username and password and click the login button" is generated. The generated test cases are then stored in a database.

[0314] The device receives these test cases from the server and prepares to run them on the device and environment specified by the user. The device automatically performs operations according to the test cases, capturing screenshots and error logs as operation results. The captured results are sent to the server. The server analyzes the received test results and determines whether the test succeeded or failed. It also generates a detailed report on the impact of errors and UI changes and provides it to the user. This report includes a summary of success or failure, error logs, and improvement suggestions.

[0315] The emotion engine analyzes the user's emotional state in real time and dynamically changes the priority of test cases based on that information. For example, if the user is feeling stressed, the system will adjust the priority of test cases so that more important tests are executed first. The user's emotional state is also reflected in the test result report and is used for analysis and improvement suggestions.

[0316] As a concrete example, consider the case where a user wants to perform UI testing for an electronic payment app. The user enters the URL of the login screen into the server. The server retrieves a screenshot from the URL and analyzes it with a generative AI model to identify the username field, password field, and login button. Based on this, the server generates a test case that says, "Enter username and password and click login button." The terminal executes this test case, performing the operation in the specified device and environment. The operation results are sent to the server, which determines whether the operation was successful or failed and generates a detailed report. The emotion engine analyzes the user's facial expressions and tone of voice, and if it determines that the user is stressed, it adds that information to the report and makes suggestions for improvement.

[0317] An example of a prompt sentence is, "Identify the login button in this screenshot, enter your username and password, and generate a test case in which you click the button. Also, if the user seems stressed, please run this case first."

[0318] This will improve the quality of electronic payment services and the user experience.

[0319] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0320] Step 1:

[0321] The user uploads the URL or file of the application or web page to be tested to the server. At the same time, the user selects the device and environment configuration options required for test execution. The input is the URL or file of the application or web page and the environment settings, and the output is the URL or file and configuration information saved on the server.

[0322] Step 2:

[0323] The server receives the specified URL or uploaded file and takes a screenshot using a browser emulator. The input is the URL or file uploaded in step 1, and the output is the screenshot taken.

[0324] Step 3:

[0325] Using a generative AI model, the server analyzes the screenshot and identifies UI elements. The input is the screenshot, and the output is the location information of the identified UI elements. The generative AI model uses image recognition techniques to identify UI elements such as buttons and input fields in the screenshot.

[0326] Step 4:

[0327] The server automatically generates test cases based on the identified UI elements and saves the data in a database. The input is the location information of the UI elements, and the output is the generated test case. The generated test case might be a scenario such as "Enter your username and password and click the login button."

[0328] Step 5:

[0329] The terminal receives the generated test case from the server and prepares to execute it on the device and environment specified by the user. The input is the test case and environment setting information, and the output is the status of the test execution preparation. The terminal prepares to execute the test case based on the appropriate environment setting.

[0330] Step 6:

[0331] The terminal automatically performs operations according to the test cases and captures screenshots and error logs as the results of the operations. The input is the test cases, and the output is screenshots and error logs as the results of the operations. The terminal performs operations according to the set scenario and records the results.

[0332] Step 7:

[0333] The server receives the test results sent from the device, analyzes them, and determines whether the test was successful or not. The input is a screenshot of the operation results and an error log, and the output is the success / failure result. The server determines whether the test was successful or not based on the analysis results.

[0334] Step 8:

[0335] The server generates a detailed report of errors and the impact of UI changes and provides it to the user. The input is the analysis results and the output is a detailed report. The report includes a success / failure summary, error log, and improvement suggestions.

[0336] Step 9:

[0337] The emotion engine analyzes the user's facial expressions and tone of voice in real time to recognize the user's emotional state. The input is the user's facial expressions and tone of voice, and the output is the recognized emotional state. The emotion engine uses image analysis and voice analysis technology to determine the user's emotions.

[0338] Step 10:

[0339] The server dynamically changes the priority of test cases according to the user's emotional state based on the output of the emotion engine. The input is the recognized emotional state, and the output is the updated test case priority. If the user is feeling stressed, the system adjusts to prioritize important test cases.

[0340] As a concrete example, if a user wants to test a login screen, the user enters the URL into the server, the server takes a screenshot, analyzes it using a generative AI model, identifies UI elements, generates a test case, the device executes the case, sends the operation results to the server, the server determines success / failure, generates a report, analyzes the user's emotional state using an emotion engine, and changes priorities.

[0341] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0342] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0343] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0344] [Second embodiment]

[0345] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0346] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0347] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0348] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0349] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0350] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0351] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0352] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0353] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0354] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0355] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0356] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0357] The system of the present invention automates UI testing of applications and websites using generative AI models. The system includes three main components: the user, the server, and the terminal.

[0358] First, the user uploads the URL of the application or website or application file to be tested to the server, clearly specifying what should be tested. The user can then further specify specific devices and environments (e.g. different browsers or resolutions).

[0359] The server then receives the specified URL or uploaded file and uses a browser emulator to capture a screenshot, which is then analyzed by the generative AI model. As a result of the analysis, UI elements (buttons, input fields, links, etc.) are identified and stored in a database along with their location information.

[0360] The server then automatically generates test cases based on the detected UI elements. For example, if a login screen is detected, it generates a scenario that says "Enter username and password and click login button." The generated test cases are saved in a database.

[0361] The terminal receives these test cases from the server and executes them on the device and environment specified by the user. The terminal automatically performs operations according to the test cases, taking screenshots and error logs as operation results, and sending the results to the server.

[0362] The server analyzes the received test results and determines whether they passed or failed. It also generates a detailed report of any errors or the impact of UI changes, which is provided to the user. This report includes a summary of success or failure, an error log, and suggestions for improvement. Based on this, the user can make necessary corrections and plan retesting.

[0363] As a concrete example, consider the case where a user wants to test a login screen. The user enters the URL of the login screen into the server. The server retrieves a screenshot from the URL and analyzes it with a generative AI model to identify the username field, password field, and login button. Based on this, the server generates a test case that says, "Enter username and password and click the login button."

[0364] The terminal executes this test case and performs operations on the specified device and environment. The operation results are sent to the server, which determines whether the operation was successful or not and generates a detailed report. The user can check this report to confirm that the login operation was successful.

[0365] The above is a specific embodiment for carrying out the present invention. This system makes it possible to automate UI tests and execute them efficiently and accurately.

[0366] The processing flow will be explained below.

[0367] Step 1:

[0368] The user uploads the URL or file of the application or web page to be tested to the server, and also selects the device and environment configuration options required for the test execution.

[0369] Step 2:

[0370] The server stores the URL or file received from the user in a database. The server configures the browser emulator based on the specified device and environment information.

[0371] Step 3:

[0372] The server uses a browser emulator to open the specified URL, captures a screenshot, and saves it in the appropriate resolution.

[0373] Step 4:

[0374] The server uses the generative AI model to analyze the captured screenshot and identify UI elements (buttons, input fields, links, etc.) The server then stores the UI elements identified as a result of the analysis and their location information in a database.

[0375] Step 5:

[0376] The server automatically generates test cases based on detected UI elements. For example, if a login screen is detected, it generates a scenario that says "Enter your username and password and click the login button." The generated test cases are saved in a database.

[0377] Step 6:

[0378] The terminal receives the test cases from the server and prepares them for execution on the device and environment specified by the user. The terminal prepares the environment according to the specified setting conditions.

[0379] Step 7:

[0380] The device automatically performs UI operations according to the received test cases, such as entering text into text fields and clicking buttons, and records any screen changes or errors that occur during the operation.

[0381] Step 8:

[0382] The device sends screenshots and error logs containing the test results to the server, providing information on which operations were successful and which failed, along with the results of the operations.

[0383] Step 9:

[0384] The server analyzes the test results sent from the device, determines whether the test succeeded or failed, and analyzes the cause of errors and the impact of UI changes.

[0385] Step 10:

[0386] The server generates a detailed report based on the analysis results, including a summary of successes / failures, error logs, and suggestions for improvement, and uploads it to the user's dashboard.

[0387] Step 11:

[0388] Users can access a dashboard to view detailed reports of tests that have been performed, allowing them to plan any necessary corrections or retests.

[0389] Example 1

[0390] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0391] Traditional methods for testing the user interface (UI) of applications and websites require a significant amount of manual work, which is time-consuming and costly. It is also difficult to perform consistent testing across different devices and environments, making it difficult to ensure the reliability and efficiency of testing. This often results in extended release times.

[0392] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0393] In this invention, the server includes a means for a user to upload a URL or file of an application or webpage to be tested, a means for analyzing screenshots of the test target using a generative AI model and identifying user interface elements, a means for automatically generating test cases based on the identified user interface elements and saving them in a database, a means for a terminal to receive the test cases and automatically execute operations, a means for analyzing test results sent from the terminal, generating a report and providing it to the user, and a means for executing tests on different devices and environments. This enables efficient and accurate automation of UI testing of applications and websites.

[0394] "User" means the person or entity who uploads URLs and files to the server, selects configuration options, and receives reports to run tests on an application or website.

[0395] "Server" is a computer system that receives URLs and files submitted by users, analyzes screenshots using a generative AI model to identify user interface elements, generates test cases, stores them in a database, analyzes test results, and generates reports.

[0396] A "generative AI model" is an artificial intelligence model used to analyze captured screenshots and identify user interface elements.

[0397] A "terminal" is a device that receives test cases sent from a server, automatically executes the operations, and sends the results to the server.

[0398] A "test case" is a specific scenario or procedure that is generated based on identified user interface elements and that is used to perform automated testing.

[0399] "User interface elements" are interactive elements such as buttons, input fields, and links that appear on the screen of an application or website.

[0400] A "screenshot" is an image that captures the display state of a specified URL or uploaded file.

[0401] A "device" is a physical or virtual hardware environment for running test cases.

[0402] An "environment" is the settings and conditions for running a test (e.g., browser type, screen resolution, etc.).

[0403] A "report" is a document generated after analyzing test results, which includes a summary of successes / failures, an error log, and suggestions for improvement.

[0404] The system of the present invention automates user interface (UI) testing of applications and websites using generative AI models. In this system, a user specifies a test target, a server and a terminal work together to execute the test, analyze the results, and provide a report. An embodiment of this system is described in detail below.

[0405] First, the user uploads the URL or file of the application or website to be tested to the server. This operation clearly specifies the target to be tested. The upload is performed by accessing the server interface using a standard web browser and operating the "Enter URL" or "Upload file" button. The user can also specify specific devices and environments (e.g., different browsers or resolutions). This sets up the test environment.

[0406] Next, the server receives the specified URL or uploaded application file and takes a screenshot using a browser emulator (e.g., Selenium). Selenium is a browser automation tool that opens the specified URL and captures a screenshot of the screen. The server takes this screenshot and passes it to the generative AI model.

[0407] The server uses a generative AI model to analyze the captured screenshot and identify user interface elements (buttons, input fields, links, etc.). The AI ​​model analyzes the image data of the screenshot and identifies the location and attributes of the UI elements. The analysis results are stored in a database.

[0408] The server then automatically generates test cases based on the identified user interface elements. For example, if a login screen is detected, it generates a test case such as "Enter username and password and click login button." This generated test case is stored in a database and later sent to the device.

[0409] The terminal executes the test cases received from the server on the device and environment specified by the user. Using a testing framework (e.g., Appium), it launches an emulator for the specified device and automatically executes operations according to the test case scenario. The terminal captures screenshots and error logs as operation results and sends them to the server.

[0410] The server analyzes the test results sent from the device and determines whether the test succeeded or failed. Specifically, it compares the expected UI state with the actual operation result, and records any discrepancies as an error. A detailed report is generated based on the analysis results and provided to the user. This report includes a summary of success or failure, an error log, and suggestions for improvement. Based on this report, the user can make any necessary corrections and plan retesting.

[0411] As a concrete example, consider the case where a user wants to test a login screen. The user enters the URL of the login screen into the server and specifies the browser and resolution. The server uses Selenium to obtain a screenshot from the specified URL, and analyzes it with a generative AI model to identify the username field, password field, and login button. Based on this, a test case is generated that says "Enter username and password and click the login button." The device executes this test case and sends the results to the server. The server analyzes the results, generates a report, and provides it to the user.

[0412] An example of a prompt statement might be something like, "Enter the URL of the login screen into the server, perform the operations according to the generated test case, and generate a prompt to evaluate the results."

[0413] By implementing the invention in this manner, it is possible to automate UI testing of applications and websites efficiently and accurately.

[0414] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0415] Step 1:

[0416] The user specifies the test target

[0417] Input: The URL or application file to test.

[0418] Output: The URL or file received by the server.

[0419] Specific operation: A user accesses the server interface using a web browser and operates the "Enter URL" or "Upload file" button. The user enters or uploads the target URL or file and presses the send button. The server receives the sent URL or file.

[0420] Step 2:

[0421] The user specifies the test environment

[0422] Input: Preferences such as device, browser, and resolution.

[0423] Output: The configuration information received by the server.

[0424] What happens: The user selects the appropriate option from the "Device Selection" or "Browser Selection" drop-down list on the server's interface. The user's selection is sent to the server, which receives this preference information.

[0425] Step 3:

[0426] Server takes screenshot

[0427] Input: URLs, application files, and configuration information submitted by the user.

[0428] Output: The captured screenshot image.

[0429] What happens: The server launches a browser emulator, such as Selenium, to open the specified URL or file. The browser emulator opens the URL with the specified configuration (e.g., browser type and resolution) and captures a screenshot of the screen. The server saves the screenshot.

[0430] Step 4:

[0431] The server analyzes the UI elements

[0432] Input: The captured screenshot image.

[0433] Output: Information about the identified UI element (position and attributes).

[0434] How it works: The server passes a screenshot as input to the generative AI model. The generative AI model analyzes the image, identifies UI elements (e.g., buttons, input fields, links, etc.), and extracts their locations and attributes. The analysis results are stored in a database.

[0435] Step 5:

[0436] The server generates the test cases

[0437] Input: Information about the identified UI element.

[0438] Output: Generated test cases.

[0439] Specific operation: The server automatically generates test cases based on UI element information retrieved from the database. For example, if a login screen is detected, a test scenario will be created that says "Enter username and password and click login button." The generated test cases are saved in the database.

[0440] Step 6:

[0441] The device runs the test case

[0442] Input: Test cases received from the server.

[0443] Output: Operation results (screenshots and error logs).

[0444] Specific operation: The terminal executes the test cases received from the server on the device and environment specified by the user. The terminal uses a testing framework (e.g., Appium) to launch the emulator for the specified device and automatically perform operations according to the test case scenario. The terminal captures screenshots and error logs as operation results and sends them to the server.

[0445] Step 7:

[0446] The server analyzes the test results

[0447] Input: Operation results sent from the device (screenshots and error logs).

[0448] Output: Analysis result (success / failure).

[0449] Specific operation: The server analyzes the operation results sent from the device. It compares the expected UI state (for example, whether a button is clickable or a form is submitted) with the actual operation result. This comparison determines whether the test is successful or failed.

[0450] Step 8:

[0451] The server generates the report

[0452] Input: Analysis data of test results.

[0453] Output: Detailed report (success / failure summary, error log, improvement suggestions).

[0454] Specific operation: The server generates a detailed report based on the analysis results, including a summary of the test success / failure, a log of errors that occurred, and suggestions for improvement. The generated report is then provided to the user.

[0455] (Application example 1)

[0456] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0457] Product picking work at logistics centers requires a great deal of time and effort, and is prone to human error. In particular, it is difficult to quickly and accurately pick large quantities of products in a large warehouse, so there is a need to improve work efficiency and reduce errors.

[0458] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0459] In this invention, the server includes a means for a user to upload a URL or file of an application or webpage to be tested, a means for analyzing screenshots of the test target using a generative AI model and identifying UI elements, a means for automatically generating test cases based on the identified UI elements and saving them in a database, a means for a terminal to receive the test cases and automatically execute operations, a means for analyzing test results sent from the terminal, generating a report and providing it to the user, and a means for supporting product picking using smart glasses at a logistics center, thereby enabling improved efficiency and accuracy of picking work at logistics centers.

[0460] "User" refers to the person who uploads the URL or file of the application or web page to be tested and reviews the generated test cases and reports.

[0461] "Generative AI Model" refers to an artificial intelligence model used to analyze a given screenshot and identify UI elements.

[0462] A "test case" refers to a scenario that is automatically generated based on identified UI elements to verify the operation and behavior of the system.

[0463] "Terminal" refers to a device or equipment that receives the generated test cases and automatically executes the operations.

[0464] "Smart glasses" refer to devices worn by workers at logistics centers that display information in real time to assist with picking operations.

[0465] "Picking" refers to the task of removing specified products from shelves at the logistics center and preparing them for shipment.

[0466] "Report" refers to a detailed report provided to the User after analyzing the test results sent from the Terminal.

[0467] A "logistics center" refers to a facility that stores, manages, and ships goods.

[0468] This invention is a system that utilizes a generative AI model and smart glasses to improve the efficiency and accuracy of product picking work in logistics centers. Specifically, the system uses a generative AI model to identify the location of products and shelves, and displays instructions to workers through the smart glasses.

[0469] First, the user uploads the URL or file of the application or web page to be tested to the server, which then recognizes the test target using the generative AI model.

[0470] The server then receives the specified URL or file and captures screenshots from a browser emulator or a real-world distribution center's surveillance cameras. These screenshots are then analyzed by a generative AI model to identify UI elements and shelf and product locations. This data is then stored in a database, and test cases are automatically generated based on the identified UI elements.

[0471] The generated test cases are received by a device (such as smart glasses) and automatically performed in the set environment. The smart glasses are worn by workers and display real-time information on shelves and product locations, as well as picking instructions. This display allows workers to accurately locate products in a short amount of time and perform picking tasks.

[0472] The results of operations performed by the terminal (e.g., the progress of a picking operation) are sent to the server, which analyzes them and generates a report that includes a success / failure summary, an error log, and suggestions for improvement, which the user can review and correct or retest.

[0473] The specific technologies used are OpenCV for image processing and machine learning models for generative AI models. The smart glasses used are commercially available (e.g., Vuzix, Google Glass Enterprise Edition). The entire system is built by combining a browser emulator, a database management system, and a communication protocol.

[0474] An example of a prompt is:

[0475] "To support product picking in the warehouse, please recognize the product and its location from the captured image. Specifically, please analyze the label or barcode and indicate the corresponding product."

[0476] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0477] Step 1:

[0478] The user uploads the URL or file of the application or web page to be tested to the server. This input includes the URL or file path, based on which the server identifies the target for testing.

[0479] Step 2:

[0480] The server receives the specified URL or file and captures a screenshot from a browser emulator or the actual surveillance camera at the distribution center. The input is the specified URL or file, and the captured screenshot is generated as the output. Specifically, the specified URL is opened using a browser emulator and the screen is captured.

[0481] Step 3:

[0482] The server uses a generative AI model to analyze the captured screenshot and identify UI elements, shelves, and product locations. The input is a screenshot, and the output is the identified UI elements, shelves, and product locations. This analysis uses an image recognition method using a machine learning model.

[0483] Step 4:

[0484] The server automatically generates test cases based on the identified UI elements and shelf / product locations and saves them in a database. The input is data on UI elements and shelf / product locations, and the generated test cases are obtained as output. Specifically, a test case is created that specifies which product to pick from which shelf based on the product's barcode and label information.

[0485] Step 5:

[0486] The terminal (smart glasses) receives the test cases generated from the server and automatically executes the operations. The test cases are input and the operation results are output. The smart glasses display shelf and product location information and picking instructions in real time.

[0487] Step 6:

[0488] The terminal sends the results of the operation it performed to the server. The input is the operation result of the smart glasses, and the output is the operation result data sent to the server. This operation result includes the progress of the picking work and an error log.

[0489] Step 7:

[0490] The server analyzes the received operation results, generates a report, and provides it to the user. The operation result data is input, and a report is generated as output. This report includes a summary of success / failure, an error log, and suggestions for improvement. The user can make corrections or retest based on this report.

[0491] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0492] Our system not only automates UI testing using generative AI models, but also combines an emotion engine to recognize user emotions and dynamically adjust the testing process accordingly. This system consists of four main components: the user, the server, the device, and the emotion engine.

[0493] First, the user uploads the URL or file of the application or web page to be tested to the server. The user also selects the device and environment configuration options required for the test execution. The emotion engine then analyzes the user's facial expressions and tone of voice to recognize the user's emotional state.

[0494] The server then receives the specified URL or uploaded file and uses a browser emulator to capture a screenshot, which is then analyzed by the generative AI model. As a result of the analysis, UI elements (buttons, input fields, links, etc.) are identified and stored in a database along with their location information.

[0495] The server then automatically generates test cases based on the detected UI elements. For example, if a login screen is detected, it generates a scenario that says "Enter username and password and click login button." The generated test cases are saved in a database.

[0496] The device receives these test cases from the server and prepares to run them on the device and environment specified by the user. The device automatically performs operations according to the test cases, taking screenshots and error logs as operation results. The obtained results are sent to the server.

[0497] The server analyzes the received test results, determines whether they passed or failed, and generates a detailed report of any errors or the impact of UI changes, which is then provided to the user. This report includes a summary of success or failure, an error log, and suggestions for improvement.

[0498] The emotion engine analyzes the user's emotional state in real time and dynamically changes the priority of test cases based on that information. For example, if the user is feeling stressed, the system will adjust the priority of test cases so that more important tests are executed first. The user's emotional state is also reflected in the test result report and is used for analysis and improvement suggestions.

[0499] As a concrete example, consider the case where a user wants to test a login screen. The user enters the URL of the login screen into the server. The server retrieves a screenshot from the URL and analyzes it with a generative AI model to identify the username field, password field, and login button. Based on this, the server generates a test case that says, "Enter username and password and click the login button."

[0500] The device runs the test cases and performs operations on the specified device and environment. The results of the operations are sent to the server, which determines whether the operation was successful or not and generates a detailed report. The emotion engine analyzes the user's facial expressions and tone of voice, and if it determines that the user is feeling stressed, it adds that information to the report and makes suggestions for improvement.

[0501] The above is a specific embodiment of the present invention, which not only automates UI testing and executes it efficiently and accurately, but also dynamically adjusts the testing process by taking into account the user's emotional state.

[0502] The processing flow will be explained below.

[0503] Step 1:

[0504] The user uploads the URL or file of the application or web page to be tested to the server, and then selects the device and environment configuration options required for the test execution.

[0505] Step 2:

[0506] The emotion engine analyzes the user's facial expressions and tone of voice to recognize the user's emotional state, and stores this information in a database within the system.

[0507] Step 3:

[0508] The server stores the URL or file received from the user in a database. The server configures the browser emulator based on the specified device and environment information.

[0509] Step 4:

[0510] The server uses a browser emulator to open the specified URL, captures a screenshot, and saves it in the appropriate resolution.

[0511] Step 5:

[0512] The server uses the generative AI model to analyze the captured screenshot and identify UI elements (buttons, input fields, links, etc.) The server then stores the UI elements identified as a result of the analysis and their location information in a database.

[0513] Step 6:

[0514] The server automatically generates test cases based on detected UI elements. For example, if a login screen is detected, it generates a scenario that says "Enter your username and password and click the login button." The generated test cases are saved in a database.

[0515] Step 7:

[0516] The terminal receives the test cases from the server and prepares them for execution on the device and environment specified by the user. The terminal prepares the environment according to the specified setting conditions.

[0517] Step 8:

[0518] The device automatically performs UI operations according to the received test cases, such as entering text into text fields and clicking buttons, and records any screen changes or errors that occur during the operation.

[0519] Step 9:

[0520] The device sends screenshots and error logs containing the test results to the server, providing information on which operations were successful and which failed, along with the results of the operations.

[0521] Step 10:

[0522] The server analyzes the test results sent from the device. The server determines whether the test was successful or failed, and analyzes the cause of any errors and the impact of UI changes. The emotion engine also references the user's emotional state and reflects that information in the test result report.

[0523] Step 11:

[0524] The server generates a detailed report based on the analysis results. The report includes a summary of success / failure, an error log, and suggestions for improvement. The report also incorporates data from the emotion engine, and includes the emotions felt by the user during the test. This report is then uploaded to the user's dashboard.

[0525] Step 12:

[0526] Users can access the dashboard and view detailed reports of the tests they have performed. Based on the reports, users can make necessary corrections and plan retests. The reports also reflect the user's emotional state, allowing for more accurate improvements.

[0527] Example 2

[0528] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0529] Conventional UI test automation systems have limited adaptability to multiple devices and environments, and have faced challenges in analyzing test results and generating reports efficiently. Furthermore, they do not dynamically adjust the test process to take into account the user's emotional state. This often leads to user frustration, and improvements to the UX are needed.

[0530] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0531] In this invention, the server includes: a means for a user to upload a URL or file of an application or webpage to be tested; a means for analyzing screenshots of the test target using a generative AI model to identify UI elements; a means for automatically generating test cases based on the identified UI elements and saving them in a database; a means for a terminal to receive the test cases and automatically execute operations; a means for analyzing test results sent from the terminal, generating a report and providing it to the user; and a means for recognizing the user's emotional state in real time and dynamically adjusting the test process based on the result. This improves the efficiency and accuracy of UI testing and enables dynamic adjustment of the test process taking the user's emotional state into account.

[0532] "User" means a person or entity that utilizes the system to upload the URL or file of an application or web page to be tested and select the configuration options required to run the test.

[0533] "Test subject" refers to the URL or file of an application or web page that a user uploads to the system, and refers to the entire software asset that is the subject of testing.

[0534] A "generative AI model" is an artificial intelligence model that analyzes screenshots to identify UI elements and automatically generate test cases.

[0535] A "screenshot" is a screen image of the application or web page under test that the system captures using a browser emulator.

[0536] A "UI element" is a user interface component (e.g., button, input field, link, etc.) identified from a screenshot by a generative AI model.

[0537] A "test case" is a set of operational scenarios that the system executes, automatically generated based on identified UI elements.

[0538] A "database" is an information storage means for saving information such as identified UI elements and generated test cases.

[0539] A "terminal" is an electronic device that executes test cases received from a server on a specified device or in a specified environment.

[0540] A "report" is a detailed report that analyzes the test results sent from the device and includes a success / failure determination, error log, and improvement suggestions.

[0541] "Emotional state" refers to the user's psychological state, recognized in real time by analyzing the user's facial expressions and tone of voice.

[0542] An "emotion engine" is an analysis device or software for analyzing a user's emotional state in real time and dynamically adjusting the testing process based on that analysis.

[0543] MODE FOR CARRYING OUT THE INVENTION

[0544] Our system automates user interface (UI) testing using generative AI models and combines them with an emotion engine to recognize user emotions and dynamically adjust the testing process accordingly. The system consists of four main components: the user, the server, the device, and the emotion engine.

[0545] server

[0546] The server receives the URL or file of the application or web page to be tested uploaded by the user. Specifically, the server uses a browser emulator (e.g., Selenium or Puppeteer) to take a screenshot of the specified URL or file. This screenshot is analyzed by a generative AI model (e.g., OpenAI's GPT-3) to identify UI elements (e.g., buttons, input fields, links, etc.). The location information of the UI elements obtained as a result of the analysis is stored in a database. The server also automatically generates test cases based on the identified UI elements, and the generated test cases are also stored in the database. The server then analyzes the test results sent from the device and determines whether they succeeded or failed. A detailed report is generated about errors and the impact of UI changes and provided to the user.

[0547] Terminal

[0548] The terminal executes the test cases received from the server on the device and environment specified by the user. Specifically, the terminal automatically performs operations according to the test cases and captures the operation results such as screenshots and error logs. The captured results are then sent to the server.

[0549] User

[0550] The user uploads the URL or file of the application or web page to be tested to the server, and then selects the device and environment configuration options required for the test execution. In addition, the emotion engine monitors the user's emotional state in real time. This is achieved by analyzing the user's facial expressions and tone of voice.

[0551] Emotion Engine

[0552] An emotion engine (e.g., Microsoft Azure Cognitive Services) analyzes the user's emotional state in real time and dynamically changes the priority of test cases. For example, if the user is feeling stressed, the system will change the priority of test cases and adjust the system so that more important tests are executed first. The user's emotional state is also reflected in the test result report and is used for analysis and improvement suggestions.

[0553] Specific examples

[0554] For example, if a user wants to test a login screen, they enter the URL of the login screen into the server. The server receives the URL and uses a browser emulator to take a screenshot of the login screen. This screenshot is analyzed by a generative AI model to identify the UI elements: username field, password field, and login button. Based on this, the server automatically generates a test case that says, "Enter username and password and click the login button."

[0555] The device runs this test case, performing operations on the specified device and environment. Screenshots and error logs obtained as a result of the operations are sent to the server, which determines whether the operation was successful or failed and generates a detailed report. The emotion engine analyzes the user's facial expressions and tone of voice, and if it determines that the user is feeling stressed, it adds that information to the report and makes suggestions for improvement.

[0556] Prompt Sentence Examples

[0557] "Write a program like the one below. A user enters the URL of a login screen into the server. The server receives the URL and takes a screenshot of the login screen using a browser emulator. The server analyzes this screenshot with a generative AI model and generates a test case (enter username and password and click the login button) based on the identified UI elements (username field, password field, login button). The generated test case is then sent to the terminal. The terminal executes the test case on the specified device and environment and sends the obtained operation results to the server. The server determines success / failure and generates a detailed report. The emotion engine analyzes the user's emotional state in real time and adjusts the testing process."

[0558] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0559] Step 1: User uploads URL or file to test

[0560] Specific behavior:

[0561] The user uploads the URL or file of the application or web page to be tested to the server, and also selects the device and environment configuration options required for the test execution.

[0562] input:

[0563] Test target URL or file, device and environment configuration options

[0564] output:

[0565] URLs or files and configuration data stored on the server

[0566] Step 2: The server receives the data and takes a screenshot

[0567] Specific behavior:

[0568] The server receives the URL or file provided by the user, then uses a browser emulator (e.g., Selenium, Puppeteer) to take a screenshot of the specified URL or file.

[0569] input:

[0570] URL or file from the user

[0571] output:

[0572] Captured screenshots

[0573] Step 3: Generate screenshots and analyze them with an AI model to identify UI elements

[0574] Specific behavior:

[0575] The server sends the captured screenshot to a generative AI model (e.g., OpenAI's GPT-3) to identify UI elements (buttons, input fields, links, etc.).

[0576] input:

[0577] Captured screenshots

[0578] output:

[0579] The location of the identified UI element

[0580] Step 4: The server generates the test cases and saves them in the database

[0581] Specific behavior:

[0582] The server automatically generates test cases based on the identified UI elements and saves them in a database. For example, if a login screen is detected, a scenario such as "Enter username and password and click login button" is generated.

[0583] input:

[0584] Location information of UI elements

[0585] output:

[0586] Generated test cases, saved test cases

[0587] Step 5: The device receives the test case and prepares to execute it

[0588] Specific behavior:

[0589] The terminal analyzes the test cases received from the server and prepares to execute the test cases on the device and in the environment specified by the user.

[0590] input:

[0591] Test cases received from the server

[0592] output:

[0593] Test cases ready to run

[0594] Step 6: The device performs the operation according to the test case

[0595] Specific behavior:

[0596] The device automatically performs operations according to the test cases, and captures screenshots and error logs as a result of the operations.

[0597] input:

[0598] Test cases ready to run

[0599] output:

[0600] Obtained operation results (screenshots, error logs)

[0601] Step 7: The server analyzes the test results and generates a report

[0602] Specific behavior:

[0603] The server analyzes the test results sent from the device and determines whether the test succeeded or failed. It also generates a detailed report on the impact of errors and UI changes and provides it to the user.

[0604] input:

[0605] Operation results sent from the device

[0606] output:

[0607] Success / failure results and detailed reports

[0608] Step 8: The emotion engine analyzes the user's emotional state and adjusts the process

[0609] Specific behavior:

[0610] The emotion engine analyzes the user's facial expressions and tone of voice to recognize the user's emotional state in real time, and dynamically reprioritizes test cases and adjusts the testing process accordingly.

[0611] input:

[0612] The user's facial expression and tone of voice

[0613] output:

[0614] Analyzed emotional states, adjusted testing process

[0615] (Application example 2)

[0616] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0617] Current UI testing automation systems do not consider the user's emotional state during the testing process, which can lead to a poor user experience. Furthermore, the testing process may not proceed effectively when the user is under stress. Furthermore, because electronic payment services often involve situations where users are prone to experiencing high levels of stress, a more advanced testing system incorporating emotion recognition is needed.

[0618] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to upload a URL or file of an application or webpage to be tested; means for analyzing a screenshot of the test target using a generative AI model and identifying UI elements; means for automatically generating test cases based on the identified UI elements and saving them in a database; means for a terminal to receive the test cases and automatically execute operations; means for analyzing test results sent from the terminal, generating a report, and providing it to the user; and means for recognizing the user's emotional state using an emotion engine that analyzes the user's facial expressions and tone of voice and dynamically changing the priority of test cases according to the emotional state. This enables advanced UI testing automation that takes the user's emotional state into consideration, making it possible to provide a high-quality user experience even in electronic payment services.

[0619] "User" is the person or system that uploads the URL or file of the application or web page under test and initiates the testing process.

[0620] "Generative AI model" refers to an artificial intelligence model used to analyze screenshots and identify UI elements.

[0621] A "screenshot" is an image captured from the screen of the application or web page being tested.

[0622] A "UI element" is an operable component in a user interface, such as a button, input field, or link.

[0623] A "test case" is a scenario of operations that is executed to specifically test a system.

[0624] A "database" is a digital storage for storing information such as test cases and test results.

[0625] A "terminal" is a device that receives test cases and automatically executes the operations.

[0626] An "emotion engine" is a system that analyzes a user's facial expressions and tone of voice to recognize the user's emotional state in real time.

[0627] An "emotional state" is an emotional state that a user is feeling, such as stress, joy, or impatience.

[0628] "Priority" is a concept that indicates the importance and order in which test cases are executed.

[0629] "Dynamic change" means changing the order and importance of test cases in real time according to the user's emotional state.

[0630] A "report" is a document or file that summarizes the test results and the analysis of the user's emotional state.

[0631] The system of the present invention not only automates UI testing using a generative AI model, but also combines it with an emotion engine to recognize user emotions and dynamically adjust the testing process accordingly. Specific embodiments are described below.

[0632] The system consists of four main components: the user, the server, the device, and the emotion engine. First, the user uploads the URL or file of the application or web page to be tested to the server. The user also selects the device and environment configuration options required for the test execution. The emotion engine then analyzes the user's facial expressions and tone of voice to recognize the user's emotional state.

[0633] The server receives the specified URL or uploaded file and takes a screenshot using a browser emulator. This screenshot is then analyzed by a generative AI model. As a result of the analysis, UI elements (buttons, input fields, links, etc.) are identified and stored in a database along with their location information. The server then automatically generates test cases based on the detected UI elements. For example, if a login screen is detected, a scenario such as "Enter your username and password and click the login button" is generated. The generated test cases are then stored in a database.

[0634] The device receives these test cases from the server and prepares to run them on the device and environment specified by the user. The device automatically performs operations according to the test cases, capturing screenshots and error logs as operation results. The captured results are sent to the server. The server analyzes the received test results and determines whether the test succeeded or failed. It also generates a detailed report on the impact of errors and UI changes and provides it to the user. This report includes a summary of success or failure, error logs, and improvement suggestions.

[0635] The emotion engine analyzes the user's emotional state in real time and dynamically changes the priority of test cases based on that information. For example, if the user is feeling stressed, the system will adjust the priority of test cases so that more important tests are executed first. The user's emotional state is also reflected in the test result report and is used for analysis and improvement suggestions.

[0636] As a concrete example, consider the case where a user wants to perform UI testing for an electronic payment app. The user enters the URL of the login screen into the server. The server retrieves a screenshot from the URL and analyzes it with a generative AI model to identify the username field, password field, and login button. Based on this, the server generates a test case that says, "Enter username and password and click login button." The terminal executes this test case, performing the operation in the specified device and environment. The operation results are sent to the server, which determines whether the operation was successful or failed and generates a detailed report. The emotion engine analyzes the user's facial expressions and tone of voice, and if it determines that the user is stressed, it adds that information to the report and makes suggestions for improvement.

[0637] An example of a prompt sentence is, "Identify the login button in this screenshot, enter your username and password, and generate a test case in which you click the button. Also, if the user seems stressed, please run this case first."

[0638] This will improve the quality of electronic payment services and the user experience.

[0639] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0640] Step 1:

[0641] The user uploads the URL or file of the application or web page to be tested to the server. At the same time, the user selects the device and environment configuration options required for test execution. The input is the URL or file of the application or web page and the environment settings, and the output is the URL or file and configuration information saved on the server.

[0642] Step 2:

[0643] The server receives the specified URL or uploaded file and takes a screenshot using a browser emulator. The input is the URL or file uploaded in step 1, and the output is the screenshot taken.

[0644] Step 3:

[0645] Using a generative AI model, the server analyzes the screenshot and identifies UI elements. The input is the screenshot, and the output is the location information of the identified UI elements. The generative AI model uses image recognition techniques to identify UI elements such as buttons and input fields in the screenshot.

[0646] Step 4:

[0647] The server automatically generates test cases based on the identified UI elements and saves the data in a database. The input is the location information of the UI elements, and the output is the generated test case. The generated test case might be a scenario such as "Enter your username and password and click the login button."

[0648] Step 5:

[0649] The terminal receives the generated test case from the server and prepares to execute it on the device and environment specified by the user. The input is the test case and environment setting information, and the output is the status of the test execution preparation. The terminal prepares to execute the test case based on the appropriate environment setting.

[0650] Step 6:

[0651] The terminal automatically performs operations according to the test cases and captures screenshots and error logs as the results of the operations. The input is the test cases, and the output is screenshots and error logs as the results of the operations. The terminal performs operations according to the set scenario and records the results.

[0652] Step 7:

[0653] The server receives the test results sent from the device, analyzes them, and determines whether the test was successful or not. The input is a screenshot of the operation results and an error log, and the output is the success / failure result. The server determines whether the test was successful or not based on the analysis results.

[0654] Step 8:

[0655] The server generates a detailed report of errors and the impact of UI changes and provides it to the user. The input is the analysis results and the output is a detailed report. The report includes a success / failure summary, error log, and improvement suggestions.

[0656] Step 9:

[0657] The emotion engine analyzes the user's facial expressions and tone of voice in real time to recognize the user's emotional state. The input is the user's facial expressions and tone of voice, and the output is the recognized emotional state. The emotion engine uses image analysis and voice analysis technology to determine the user's emotions.

[0658] Step 10:

[0659] The server dynamically changes the priority of test cases according to the user's emotional state based on the output of the emotion engine. The input is the recognized emotional state, and the output is the updated test case priority. If the user is feeling stressed, the system adjusts to prioritize important test cases.

[0660] As a concrete example, if a user wants to test a login screen, the user enters the URL into the server, the server takes a screenshot, analyzes it using a generative AI model, identifies UI elements, generates a test case, the device executes the case, sends the operation results to the server, the server determines success / failure, generates a report, analyzes the user's emotional state using an emotion engine, and changes priorities.

[0661] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0662] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0663] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0664] [Third embodiment]

[0665] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0666] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0667] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0668] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0669] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0670] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0671] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0672] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0673] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0674] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0675] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0676] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0677] The system of the present invention automates UI testing of applications and websites using generative AI models. The system includes three main components: the user, the server, and the terminal.

[0678] First, the user uploads the URL of the application or website or application file to be tested to the server, clearly specifying what should be tested. The user can then further specify specific devices and environments (e.g. different browsers or resolutions).

[0679] The server then receives the specified URL or uploaded file and uses a browser emulator to capture a screenshot, which is then analyzed by the generative AI model. As a result of the analysis, UI elements (buttons, input fields, links, etc.) are identified and stored in a database along with their location information.

[0680] The server then automatically generates test cases based on the detected UI elements. For example, if a login screen is detected, it generates a scenario that says "Enter username and password and click login button." The generated test cases are saved in a database.

[0681] The terminal receives these test cases from the server and executes them on the device and environment specified by the user. The terminal automatically performs operations according to the test cases, taking screenshots and error logs as operation results, and sending the results to the server.

[0682] The server analyzes the received test results and determines whether they passed or failed. It also generates a detailed report of any errors or the impact of UI changes, which is provided to the user. This report includes a summary of success or failure, an error log, and suggestions for improvement. Based on this, the user can make necessary corrections and plan retesting.

[0683] As a concrete example, consider the case where a user wants to test a login screen. The user enters the URL of the login screen into the server. The server retrieves a screenshot from the URL and analyzes it with a generative AI model to identify the username field, password field, and login button. Based on this, the server generates a test case that says, "Enter username and password and click the login button."

[0684] The terminal executes this test case and performs operations on the specified device and environment. The operation results are sent to the server, which determines whether the operation was successful or not and generates a detailed report. The user can check this report to confirm that the login operation was successful.

[0685] The above is a specific embodiment for carrying out the present invention. This system makes it possible to automate UI tests and execute them efficiently and accurately.

[0686] The processing flow will be explained below.

[0687] Step 1:

[0688] The user uploads the URL or file of the application or web page to be tested to the server, and also selects the device and environment configuration options required for the test execution.

[0689] Step 2:

[0690] The server stores the URL or file received from the user in a database. The server configures the browser emulator based on the specified device and environment information.

[0691] Step 3:

[0692] The server uses a browser emulator to open the specified URL, captures a screenshot, and saves it in the appropriate resolution.

[0693] Step 4:

[0694] The server uses the generative AI model to analyze the captured screenshot and identify UI elements (buttons, input fields, links, etc.) The server then stores the UI elements identified as a result of the analysis and their location information in a database.

[0695] Step 5:

[0696] The server automatically generates test cases based on detected UI elements. For example, if a login screen is detected, it generates a scenario that says "Enter your username and password and click the login button." The generated test cases are saved in a database.

[0697] Step 6:

[0698] The terminal receives the test cases from the server and prepares them for execution on the device and environment specified by the user. The terminal prepares the environment according to the specified setting conditions.

[0699] Step 7:

[0700] The device automatically performs UI operations according to the received test cases, such as entering text into text fields and clicking buttons, and records any screen changes or errors that occur during the operation.

[0701] Step 8:

[0702] The device sends screenshots and error logs containing the test results to the server, providing information on which operations were successful and which failed, along with the results of the operations.

[0703] Step 9:

[0704] The server analyzes the test results sent from the device, determines whether the test succeeded or failed, and analyzes the cause of errors and the impact of UI changes.

[0705] Step 10:

[0706] The server generates a detailed report based on the analysis results, including a summary of successes / failures, error logs, and suggestions for improvement, and uploads it to the user's dashboard.

[0707] Step 11:

[0708] Users can access a dashboard to view detailed reports of tests that have been performed, allowing them to plan any necessary corrections or retests.

[0709] Example 1

[0710] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0711] Traditional methods for testing the user interface (UI) of applications and websites require a significant amount of manual work, which is time-consuming and costly. It is also difficult to perform consistent testing across different devices and environments, making it difficult to ensure the reliability and efficiency of testing. This often results in extended release times.

[0712] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0713] In this invention, the server includes a means for a user to upload a URL or file of an application or webpage to be tested, a means for analyzing screenshots of the test target using a generative AI model and identifying user interface elements, a means for automatically generating test cases based on the identified user interface elements and saving them in a database, a means for a terminal to receive the test cases and automatically execute operations, a means for analyzing test results sent from the terminal, generating a report and providing it to the user, and a means for executing tests on different devices and environments. This enables efficient and accurate automation of UI testing of applications and websites.

[0714] "User" means the person or entity who uploads URLs and files to the server, selects configuration options, and receives reports to run tests on an application or website.

[0715] "Server" is a computer system that receives URLs and files submitted by users, analyzes screenshots using a generative AI model to identify user interface elements, generates test cases, stores them in a database, analyzes test results, and generates reports.

[0716] A "generative AI model" is an artificial intelligence model used to analyze captured screenshots and identify user interface elements.

[0717] A "terminal" is a device that receives test cases sent from a server, automatically executes the operations, and sends the results to the server.

[0718] A "test case" is a specific scenario or procedure that is generated based on identified user interface elements and that is used to perform automated testing.

[0719] "User interface elements" are interactive elements such as buttons, input fields, and links that appear on the screen of an application or website.

[0720] A "screenshot" is an image that captures the display state of a specified URL or uploaded file.

[0721] A "device" is a physical or virtual hardware environment for running test cases.

[0722] An "environment" is the settings and conditions for running a test (e.g., browser type, screen resolution, etc.).

[0723] A "report" is a document generated after analyzing test results, which includes a summary of successes / failures, an error log, and suggestions for improvement.

[0724] The system of the present invention automates user interface (UI) testing of applications and websites using generative AI models. In this system, a user specifies a test target, a server and a terminal work together to execute the test, analyze the results, and provide a report. An embodiment of this system is described in detail below.

[0725] First, the user uploads the URL or file of the application or website to be tested to the server. This operation clearly specifies the target to be tested. The upload is performed by accessing the server interface using a standard web browser and operating the "Enter URL" or "Upload file" button. The user can also specify specific devices and environments (e.g., different browsers or resolutions). This sets up the test environment.

[0726] Next, the server receives the specified URL or uploaded application file and takes a screenshot using a browser emulator (e.g., Selenium). Selenium is a browser automation tool that opens the specified URL and captures a screenshot of the screen. The server takes this screenshot and passes it to the generative AI model.

[0727] The server uses a generative AI model to analyze the captured screenshot and identify user interface elements (buttons, input fields, links, etc.). The AI ​​model analyzes the image data of the screenshot and identifies the location and attributes of the UI elements. The analysis results are stored in a database.

[0728] The server then automatically generates test cases based on the identified user interface elements. For example, if a login screen is detected, it generates a test case such as "Enter username and password and click login button." This generated test case is stored in a database and later sent to the device.

[0729] The terminal executes the test cases received from the server on the device and environment specified by the user. Using a testing framework (e.g., Appium), it launches an emulator for the specified device and automatically executes operations according to the test case scenario. The terminal captures screenshots and error logs as operation results and sends them to the server.

[0730] The server analyzes the test results sent from the device and determines whether the test succeeded or failed. Specifically, it compares the expected UI state with the actual operation result, and records any discrepancies as an error. A detailed report is generated based on the analysis results and provided to the user. This report includes a summary of success or failure, an error log, and suggestions for improvement. Based on this report, the user can make any necessary corrections and plan retesting.

[0731] As a concrete example, consider the case where a user wants to test a login screen. The user enters the URL of the login screen into the server and specifies the browser and resolution. The server uses Selenium to obtain a screenshot from the specified URL, and analyzes it with a generative AI model to identify the username field, password field, and login button. Based on this, a test case is generated that says "Enter username and password and click the login button." The device executes this test case and sends the results to the server. The server analyzes the results, generates a report, and provides it to the user.

[0732] An example of a prompt statement might be something like, "Enter the URL of the login screen into the server, perform the operations according to the generated test case, and generate a prompt to evaluate the results."

[0733] By implementing the invention in this manner, it is possible to automate UI testing of applications and websites efficiently and accurately.

[0734] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0735] Step 1:

[0736] The user specifies the test target

[0737] Input: The URL or application file to test.

[0738] Output: The URL or file received by the server.

[0739] Specific operation: A user accesses the server interface using a web browser and operates the "Enter URL" or "Upload file" button. The user enters or uploads the target URL or file and presses the send button. The server receives the sent URL or file.

[0740] Step 2:

[0741] The user specifies the test environment

[0742] Input: Preferences such as device, browser, and resolution.

[0743] Output: The configuration information received by the server.

[0744] What happens: The user selects the appropriate option from the "Device Selection" or "Browser Selection" drop-down list on the server's interface. The user's selection is sent to the server, which receives this preference information.

[0745] Step 3:

[0746] Server takes screenshot

[0747] Input: URLs, application files, and configuration information submitted by the user.

[0748] Output: The captured screenshot image.

[0749] What happens: The server launches a browser emulator, such as Selenium, to open the specified URL or file. The browser emulator opens the URL with the specified configuration (e.g., browser type and resolution) and captures a screenshot of the screen. The server saves the screenshot.

[0750] Step 4:

[0751] The server analyzes the UI elements

[0752] Input: The captured screenshot image.

[0753] Output: Information about the identified UI element (position and attributes).

[0754] How it works: The server passes a screenshot as input to the generative AI model. The generative AI model analyzes the image, identifies UI elements (e.g., buttons, input fields, links, etc.), and extracts their locations and attributes. The analysis results are stored in a database.

[0755] Step 5:

[0756] The server generates the test cases

[0757] Input: Information about the identified UI element.

[0758] Output: Generated test cases.

[0759] Specific operation: The server automatically generates test cases based on UI element information retrieved from the database. For example, if a login screen is detected, a test scenario will be created that says "Enter username and password and click login button." The generated test cases are saved in the database.

[0760] Step 6:

[0761] The device runs the test case

[0762] Input: Test cases received from the server.

[0763] Output: Operation results (screenshots and error logs).

[0764] Specific operation: The terminal executes the test cases received from the server on the device and environment specified by the user. The terminal uses a testing framework (e.g., Appium) to launch the emulator for the specified device and automatically perform operations according to the test case scenario. The terminal captures screenshots and error logs as operation results and sends them to the server.

[0765] Step 7:

[0766] The server analyzes the test results

[0767] Input: Operation results sent from the device (screenshots and error logs).

[0768] Output: Analysis result (success / failure).

[0769] Specific operation: The server analyzes the operation results sent from the device. It compares the expected UI state (for example, whether a button is clickable or a form is submitted) with the actual operation result. This comparison determines whether the test is successful or failed.

[0770] Step 8:

[0771] The server generates the report

[0772] Input: Analysis data of test results.

[0773] Output: Detailed report (success / failure summary, error log, improvement suggestions).

[0774] Specific operation: The server generates a detailed report based on the analysis results, including a summary of the test success / failure, a log of errors that occurred, and suggestions for improvement. The generated report is then provided to the user.

[0775] (Application example 1)

[0776] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0777] Product picking work at logistics centers requires a great deal of time and effort, and is prone to human error. In particular, it is difficult to quickly and accurately pick large quantities of products in a large warehouse, so there is a need to improve work efficiency and reduce errors.

[0778] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0779] In this invention, the server includes a means for a user to upload a URL or file of an application or webpage to be tested, a means for analyzing screenshots of the test target using a generative AI model and identifying UI elements, a means for automatically generating test cases based on the identified UI elements and saving them in a database, a means for a terminal to receive the test cases and automatically execute operations, a means for analyzing test results sent from the terminal, generating a report and providing it to the user, and a means for supporting product picking using smart glasses at a logistics center, thereby enabling improved efficiency and accuracy of picking work at logistics centers.

[0780] "User" refers to the person who uploads the URL or file of the application or web page to be tested and reviews the generated test cases and reports.

[0781] "Generative AI Model" refers to an artificial intelligence model used to analyze a given screenshot and identify UI elements.

[0782] A "test case" refers to a scenario that is automatically generated based on identified UI elements to verify the operation and behavior of the system.

[0783] "Terminal" refers to a device or equipment that receives the generated test cases and automatically executes the operations.

[0784] "Smart glasses" refer to devices worn by workers at logistics centers that display information in real time to assist with picking operations.

[0785] "Picking" refers to the task of removing specified products from shelves at the logistics center and preparing them for shipment.

[0786] "Report" refers to a detailed report provided to the User after analyzing the test results sent from the Terminal.

[0787] A "logistics center" refers to a facility that stores, manages, and ships goods.

[0788] This invention is a system that utilizes a generative AI model and smart glasses to improve the efficiency and accuracy of product picking work in logistics centers. Specifically, the system uses a generative AI model to identify the location of products and shelves, and displays instructions to workers through the smart glasses.

[0789] First, the user uploads the URL or file of the application or web page to be tested to the server, which then recognizes the test target using the generative AI model.

[0790] The server then receives the specified URL or file and captures screenshots from a browser emulator or a real-world distribution center's surveillance cameras. These screenshots are then analyzed by a generative AI model to identify UI elements and shelf and product locations. This data is then stored in a database, and test cases are automatically generated based on the identified UI elements.

[0791] The generated test cases are received by a device (such as smart glasses) and automatically performed in the set environment. The smart glasses are worn by workers and display real-time information on shelves and product locations, as well as picking instructions. This display allows workers to accurately locate products in a short amount of time and perform picking tasks.

[0792] The results of operations performed by the terminal (e.g., the progress of a picking operation) are sent to the server, which analyzes them and generates a report that includes a success / failure summary, an error log, and suggestions for improvement, which the user can review and correct or retest.

[0793] The specific technologies used are OpenCV for image processing and machine learning models for generative AI models. The smart glasses used are commercially available (e.g., Vuzix, Google Glass Enterprise Edition). The entire system is built by combining a browser emulator, a database management system, and a communication protocol.

[0794] An example of a prompt is:

[0795] "To support product picking in the warehouse, please recognize the product and its location from the captured image. Specifically, please analyze the label or barcode and indicate the corresponding product."

[0796] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0797] Step 1:

[0798] The user uploads the URL or file of the application or web page to be tested to the server. This input includes the URL or file path, based on which the server identifies the target for testing.

[0799] Step 2:

[0800] The server receives the specified URL or file and captures a screenshot from a browser emulator or the actual surveillance camera at the distribution center. The input is the specified URL or file, and the captured screenshot is generated as the output. Specifically, the specified URL is opened using a browser emulator and the screen is captured.

[0801] Step 3:

[0802] The server uses a generative AI model to analyze the captured screenshot and identify UI elements, shelves, and product locations. The input is a screenshot, and the output is the identified UI elements, shelves, and product locations. This analysis uses an image recognition method using a machine learning model.

[0803] Step 4:

[0804] The server automatically generates test cases based on the identified UI elements and shelf / product locations and saves them in a database. The input is data on UI elements and shelf / product locations, and the generated test cases are obtained as output. Specifically, a test case is created that specifies which product to pick from which shelf based on the product's barcode and label information.

[0805] Step 5:

[0806] The terminal (smart glasses) receives the test cases generated from the server and automatically executes the operations. The test cases are input and the operation results are output. The smart glasses display shelf and product location information and picking instructions in real time.

[0807] Step 6:

[0808] The terminal sends the results of the operation it performed to the server. The input is the operation result of the smart glasses, and the output is the operation result data sent to the server. This operation result includes the progress of the picking work and an error log.

[0809] Step 7:

[0810] The server analyzes the received operation results, generates a report, and provides it to the user. The operation result data is input, and a report is generated as output. This report includes a summary of success / failure, an error log, and suggestions for improvement. The user can make corrections or retest based on this report.

[0811] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0812] Our system not only automates UI testing using generative AI models, but also combines an emotion engine to recognize user emotions and dynamically adjust the testing process accordingly. This system consists of four main components: the user, the server, the device, and the emotion engine.

[0813] First, the user uploads the URL or file of the application or web page to be tested to the server. The user also selects the device and environment configuration options required for the test execution. The emotion engine then analyzes the user's facial expressions and tone of voice to recognize the user's emotional state.

[0814] The server then receives the specified URL or uploaded file and uses a browser emulator to capture a screenshot, which is then analyzed by the generative AI model. As a result of the analysis, UI elements (buttons, input fields, links, etc.) are identified and stored in a database along with their location information.

[0815] The server then automatically generates test cases based on the detected UI elements. For example, if a login screen is detected, it generates a scenario that says "Enter username and password and click login button." The generated test cases are saved in a database.

[0816] The device receives these test cases from the server and prepares to run them on the device and environment specified by the user. The device automatically performs operations according to the test cases, taking screenshots and error logs as operation results. The obtained results are sent to the server.

[0817] The server analyzes the received test results, determines whether they passed or failed, and generates a detailed report of any errors or the impact of UI changes, which is then provided to the user. This report includes a summary of success or failure, an error log, and suggestions for improvement.

[0818] The emotion engine analyzes the user's emotional state in real time and dynamically changes the priority of test cases based on that information. For example, if the user is feeling stressed, the system will adjust the priority of test cases so that more important tests are executed first. The user's emotional state is also reflected in the test result report and is used for analysis and improvement suggestions.

[0819] As a concrete example, consider the case where a user wants to test a login screen. The user enters the URL of the login screen into the server. The server retrieves a screenshot from the URL and analyzes it with a generative AI model to identify the username field, password field, and login button. Based on this, the server generates a test case that says, "Enter username and password and click the login button."

[0820] The device runs the test cases and performs operations on the specified device and environment. The results of the operations are sent to the server, which determines whether the operation was successful or not and generates a detailed report. The emotion engine analyzes the user's facial expressions and tone of voice, and if it determines that the user is feeling stressed, it adds that information to the report and makes suggestions for improvement.

[0821] The above is a specific embodiment of the present invention, which not only automates UI testing and executes it efficiently and accurately, but also dynamically adjusts the testing process by taking into account the user's emotional state.

[0822] The processing flow will be explained below.

[0823] Step 1:

[0824] The user uploads the URL or file of the application or web page to be tested to the server, and then selects the device and environment configuration options required for the test execution.

[0825] Step 2:

[0826] The emotion engine analyzes the user's facial expressions and tone of voice to recognize the user's emotional state, and stores this information in a database within the system.

[0827] Step 3:

[0828] The server stores the URL or file received from the user in a database. The server configures the browser emulator based on the specified device and environment information.

[0829] Step 4:

[0830] The server uses a browser emulator to open the specified URL, captures a screenshot, and saves it in the appropriate resolution.

[0831] Step 5:

[0832] The server uses the generative AI model to analyze the captured screenshot and identify UI elements (buttons, input fields, links, etc.) The server then stores the UI elements identified as a result of the analysis and their location information in a database.

[0833] Step 6:

[0834] The server automatically generates test cases based on detected UI elements. For example, if a login screen is detected, it generates a scenario that says "Enter your username and password and click the login button." The generated test cases are saved in a database.

[0835] Step 7:

[0836] The terminal receives the test cases from the server and prepares them for execution on the device and environment specified by the user. The terminal prepares the environment according to the specified setting conditions.

[0837] Step 8:

[0838] The device automatically performs UI operations according to the received test cases, such as entering text into text fields and clicking buttons, and records any screen changes or errors that occur during the operation.

[0839] Step 9:

[0840] The device sends screenshots and error logs containing the test results to the server, providing information on which operations were successful and which failed, along with the results of the operations.

[0841] Step 10:

[0842] The server analyzes the test results sent from the device. The server determines whether the test was successful or failed, and analyzes the cause of any errors and the impact of UI changes. The emotion engine also references the user's emotional state and reflects that information in the test result report.

[0843] Step 11:

[0844] The server generates a detailed report based on the analysis results. The report includes a summary of success / failure, an error log, and suggestions for improvement. The report also incorporates data from the emotion engine, and includes the emotions felt by the user during the test. This report is then uploaded to the user's dashboard.

[0845] Step 12:

[0846] Users can access the dashboard and view detailed reports of the tests they have performed. Based on the reports, users can make necessary corrections and plan retests. The reports also reflect the user's emotional state, allowing for more accurate improvements.

[0847] Example 2

[0848] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0849] Conventional UI test automation systems have limited adaptability to multiple devices and environments, and have faced challenges in analyzing test results and generating reports efficiently. Furthermore, they do not dynamically adjust the test process to take into account the user's emotional state. This often leads to user frustration, and improvements to the UX are needed.

[0850] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0851] In this invention, the server includes: a means for a user to upload a URL or file of an application or webpage to be tested; a means for analyzing screenshots of the test target using a generative AI model to identify UI elements; a means for automatically generating test cases based on the identified UI elements and saving them in a database; a means for a terminal to receive the test cases and automatically execute operations; a means for analyzing test results sent from the terminal, generating a report and providing it to the user; and a means for recognizing the user's emotional state in real time and dynamically adjusting the test process based on the result. This improves the efficiency and accuracy of UI testing and enables dynamic adjustment of the test process taking the user's emotional state into account.

[0852] "User" means a person or entity that utilizes the system to upload the URL or file of an application or web page to be tested and select the configuration options required to run the test.

[0853] "Test subject" refers to the URL or file of an application or web page that a user uploads to the system, and refers to the entire software asset that is the subject of testing.

[0854] A "generative AI model" is an artificial intelligence model that analyzes screenshots to identify UI elements and automatically generate test cases.

[0855] A "screenshot" is a screen image of the application or web page under test that the system captures using a browser emulator.

[0856] A "UI element" is a user interface component (e.g., button, input field, link, etc.) identified from a screenshot by a generative AI model.

[0857] A "test case" is a set of operational scenarios that the system executes, automatically generated based on identified UI elements.

[0858] A "database" is an information storage means for saving information such as identified UI elements and generated test cases.

[0859] A "terminal" is an electronic device that executes test cases received from a server on a specified device or in a specified environment.

[0860] A "report" is a detailed report that analyzes the test results sent from the device and includes a success / failure determination, error log, and improvement suggestions.

[0861] "Emotional state" refers to the user's psychological state, recognized in real time by analyzing the user's facial expressions and tone of voice.

[0862] An "emotion engine" is an analysis device or software for analyzing a user's emotional state in real time and dynamically adjusting the testing process based on that analysis.

[0863] MODE FOR CARRYING OUT THE INVENTION

[0864] Our system automates user interface (UI) testing using generative AI models and combines them with an emotion engine to recognize user emotions and dynamically adjust the testing process accordingly. The system consists of four main components: the user, the server, the device, and the emotion engine.

[0865] server

[0866] The server receives the URL or file of the application or web page to be tested uploaded by the user. Specifically, the server uses a browser emulator (e.g., Selenium or Puppeteer) to take a screenshot of the specified URL or file. This screenshot is analyzed by a generative AI model (e.g., OpenAI's GPT-3) to identify UI elements (e.g., buttons, input fields, links, etc.). The location information of the UI elements obtained as a result of the analysis is stored in a database. The server also automatically generates test cases based on the identified UI elements, and the generated test cases are also stored in the database. The server then analyzes the test results sent from the device and determines whether they succeeded or failed. A detailed report is generated about errors and the impact of UI changes and provided to the user.

[0867] Terminal

[0868] The terminal executes the test cases received from the server on the device and environment specified by the user. Specifically, the terminal automatically performs operations according to the test cases and captures the operation results such as screenshots and error logs. The captured results are then sent to the server.

[0869] User

[0870] The user uploads the URL or file of the application or web page to be tested to the server, and then selects the device and environment configuration options required for the test execution. In addition, the emotion engine monitors the user's emotional state in real time. This is achieved by analyzing the user's facial expressions and tone of voice.

[0871] Emotion Engine

[0872] An emotion engine (e.g., Microsoft Azure Cognitive Services) analyzes the user's emotional state in real time and dynamically changes the priority of test cases. For example, if the user is feeling stressed, the system will change the priority of test cases and adjust the system so that more important tests are executed first. The user's emotional state is also reflected in the test result report and is used for analysis and improvement suggestions.

[0873] Specific examples

[0874] For example, if a user wants to test a login screen, they enter the URL of the login screen into the server. The server receives the URL and uses a browser emulator to take a screenshot of the login screen. This screenshot is analyzed by a generative AI model to identify the UI elements: username field, password field, and login button. Based on this, the server automatically generates a test case that says, "Enter username and password and click the login button."

[0875] The device runs this test case, performing operations on the specified device and environment. Screenshots and error logs obtained as a result of the operations are sent to the server, which determines whether the operation was successful or failed and generates a detailed report. The emotion engine analyzes the user's facial expressions and tone of voice, and if it determines that the user is feeling stressed, it adds that information to the report and makes suggestions for improvement.

[0876] Prompt Sentence Examples

[0877] "Write a program like the one below. A user enters the URL of a login screen into the server. The server receives the URL and takes a screenshot of the login screen using a browser emulator. The server analyzes this screenshot with a generative AI model and generates a test case (enter username and password and click the login button) based on the identified UI elements (username field, password field, login button). The generated test case is then sent to the terminal. The terminal executes the test case on the specified device and environment and sends the obtained operation results to the server. The server determines success / failure and generates a detailed report. The emotion engine analyzes the user's emotional state in real time and adjusts the testing process."

[0878] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0879] Step 1: User uploads URL or file to test

[0880] Specific behavior:

[0881] The user uploads the URL or file of the application or web page to be tested to the server, and also selects the device and environment configuration options required for the test execution.

[0882] input:

[0883] Test target URL or file, device and environment configuration options

[0884] output:

[0885] URLs or files and configuration data stored on the server

[0886] Step 2: The server receives the data and takes a screenshot

[0887] Specific behavior:

[0888] The server receives the URL or file provided by the user, then uses a browser emulator (e.g., Selenium, Puppeteer) to take a screenshot of the specified URL or file.

[0889] input:

[0890] URL or file from the user

[0891] output:

[0892] Captured screenshots

[0893] Step 3: Generate screenshots and analyze them with an AI model to identify UI elements

[0894] Specific behavior:

[0895] The server sends the captured screenshot to a generative AI model (e.g., OpenAI's GPT-3) to identify UI elements (buttons, input fields, links, etc.).

[0896] input:

[0897] Captured screenshots

[0898] output:

[0899] The location of the identified UI element

[0900] Step 4: The server generates the test cases and saves them in the database

[0901] Specific behavior:

[0902] The server automatically generates test cases based on the identified UI elements and saves them in a database. For example, if a login screen is detected, a scenario such as "Enter username and password and click login button" is generated.

[0903] input:

[0904] Location information of UI elements

[0905] output:

[0906] Generated test cases, saved test cases

[0907] Step 5: The device receives the test case and prepares to execute it

[0908] Specific behavior:

[0909] The terminal analyzes the test cases received from the server and prepares to execute the test cases on the device and in the environment specified by the user.

[0910] input:

[0911] Test cases received from the server

[0912] output:

[0913] Test cases ready to run

[0914] Step 6: The device performs the operation according to the test case

[0915] Specific behavior:

[0916] The device automatically performs operations according to the test cases, and captures screenshots and error logs as a result of the operations.

[0917] input:

[0918] Test cases ready to run

[0919] output:

[0920] Obtained operation results (screenshots, error logs)

[0921] Step 7: The server analyzes the test results and generates a report

[0922] Specific behavior:

[0923] The server analyzes the test results sent from the device and determines whether the test succeeded or failed. It also generates a detailed report on the impact of errors and UI changes and provides it to the user.

[0924] input:

[0925] Operation results sent from the device

[0926] output:

[0927] Success / failure results and detailed reports

[0928] Step 8: The emotion engine analyzes the user's emotional state and adjusts the process

[0929] Specific behavior:

[0930] The emotion engine analyzes the user's facial expressions and tone of voice to recognize the user's emotional state in real time, and dynamically reprioritizes test cases and adjusts the testing process accordingly.

[0931] input:

[0932] The user's facial expression and tone of voice

[0933] output:

[0934] Analyzed emotional states, adjusted testing process

[0935] (Application example 2)

[0936] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0937] Current UI testing automation systems do not consider the user's emotional state during the testing process, which can lead to a poor user experience. Furthermore, the testing process may not proceed effectively when the user is under stress. Furthermore, because electronic payment services often involve situations where users are prone to experiencing high levels of stress, a more advanced testing system incorporating emotion recognition is needed.

[0938] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to upload a URL or file of an application or webpage to be tested; means for analyzing a screenshot of the test target using a generative AI model and identifying UI elements; means for automatically generating test cases based on the identified UI elements and saving them in a database; means for a terminal to receive the test cases and automatically execute operations; means for analyzing test results sent from the terminal, generating a report, and providing it to the user; and means for recognizing the user's emotional state using an emotion engine that analyzes the user's facial expressions and tone of voice and dynamically changing the priority of test cases according to the emotional state. This enables advanced UI testing automation that takes the user's emotional state into consideration, making it possible to provide a high-quality user experience even in electronic payment services.

[0939] "User" is the person or system that uploads the URL or file of the application or web page under test and initiates the testing process.

[0940] "Generative AI model" refers to an artificial intelligence model used to analyze screenshots and identify UI elements.

[0941] A "screenshot" is an image captured from the screen of the application or web page being tested.

[0942] A "UI element" is an operable component in a user interface, such as a button, input field, or link.

[0943] A "test case" is a scenario of operations that is executed to specifically test a system.

[0944] A "database" is a digital storage for storing information such as test cases and test results.

[0945] A "terminal" is a device that receives test cases and automatically executes the operations.

[0946] An "emotion engine" is a system that analyzes a user's facial expressions and tone of voice to recognize the user's emotional state in real time.

[0947] An "emotional state" is an emotional state that a user is feeling, such as stress, joy, or impatience.

[0948] "Priority" is a concept that indicates the importance and order in which test cases are executed.

[0949] "Dynamic change" means changing the order and importance of test cases in real time according to the user's emotional state.

[0950] A "report" is a document or file that summarizes the test results and the analysis of the user's emotional state.

[0951] The system of the present invention not only automates UI testing using a generative AI model, but also combines it with an emotion engine to recognize user emotions and dynamically adjust the testing process accordingly. Specific embodiments are described below.

[0952] The system consists of four main components: the user, the server, the device, and the emotion engine. First, the user uploads the URL or file of the application or web page to be tested to the server. The user also selects the device and environment configuration options required for the test execution. The emotion engine then analyzes the user's facial expressions and tone of voice to recognize the user's emotional state.

[0953] The server receives the specified URL or uploaded file and takes a screenshot using a browser emulator. This screenshot is then analyzed by a generative AI model. As a result of the analysis, UI elements (buttons, input fields, links, etc.) are identified and stored in a database along with their location information. The server then automatically generates test cases based on the detected UI elements. For example, if a login screen is detected, a scenario such as "Enter your username and password and click the login button" is generated. The generated test cases are then stored in a database.

[0954] The device receives these test cases from the server and prepares to run them on the device and environment specified by the user. The device automatically performs operations according to the test cases, capturing screenshots and error logs as operation results. The captured results are sent to the server. The server analyzes the received test results and determines whether the test succeeded or failed. It also generates a detailed report on the impact of errors and UI changes and provides it to the user. This report includes a summary of success or failure, error logs, and improvement suggestions.

[0955] The emotion engine analyzes the user's emotional state in real time and dynamically changes the priority of test cases based on that information. For example, if the user is feeling stressed, the system will adjust the priority of test cases so that more important tests are executed first. The user's emotional state is also reflected in the test result report and is used for analysis and improvement suggestions.

[0956] As a concrete example, consider the case where a user wants to perform UI testing for an electronic payment app. The user enters the URL of the login screen into the server. The server retrieves a screenshot from the URL and analyzes it with a generative AI model to identify the username field, password field, and login button. Based on this, the server generates a test case that says, "Enter username and password and click login button." The terminal executes this test case, performing the operation in the specified device and environment. The operation results are sent to the server, which determines whether the operation was successful or failed and generates a detailed report. The emotion engine analyzes the user's facial expressions and tone of voice, and if it determines that the user is stressed, it adds that information to the report and makes suggestions for improvement.

[0957] An example of a prompt sentence is, "Identify the login button in this screenshot, enter your username and password, and generate a test case in which you click the button. Also, if the user seems stressed, please run this case first."

[0958] This will improve the quality of electronic payment services and the user experience.

[0959] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0960] Step 1:

[0961] The user uploads the URL or file of the application or web page to be tested to the server. At the same time, the user selects the device and environment configuration options required for test execution. The input is the URL or file of the application or web page and the environment settings, and the output is the URL or file and configuration information saved on the server.

[0962] Step 2:

[0963] The server receives the specified URL or uploaded file and takes a screenshot using a browser emulator. The input is the URL or file uploaded in step 1, and the output is the screenshot taken.

[0964] Step 3:

[0965] Using a generative AI model, the server analyzes the screenshot and identifies UI elements. The input is the screenshot, and the output is the location information of the identified UI elements. The generative AI model uses image recognition techniques to identify UI elements such as buttons and input fields in the screenshot.

[0966] Step 4:

[0967] The server automatically generates test cases based on the identified UI elements and saves the data in a database. The input is the location information of the UI elements, and the output is the generated test case. The generated test case might be a scenario such as "Enter your username and password and click the login button."

[0968] Step 5:

[0969] The terminal receives the generated test case from the server and prepares to execute it on the device and environment specified by the user. The input is the test case and environment setting information, and the output is the status of the test execution preparation. The terminal prepares to execute the test case based on the appropriate environment setting.

[0970] Step 6:

[0971] The terminal automatically performs operations according to the test cases and captures screenshots and error logs as the results of the operations. The input is the test cases, and the output is screenshots and error logs as the results of the operations. The terminal performs operations according to the set scenario and records the results.

[0972] Step 7:

[0973] The server receives the test results sent from the device, analyzes them, and determines whether the test was successful or not. The input is a screenshot of the operation results and an error log, and the output is the success / failure result. The server determines whether the test was successful or not based on the analysis results.

[0974] Step 8:

[0975] The server generates a detailed report of errors and the impact of UI changes and provides it to the user. The input is the analysis results and the output is a detailed report. The report includes a success / failure summary, error log, and improvement suggestions.

[0976] Step 9:

[0977] The emotion engine analyzes the user's facial expressions and tone of voice in real time to recognize the user's emotional state. The input is the user's facial expressions and tone of voice, and the output is the recognized emotional state. The emotion engine uses image analysis and voice analysis technology to determine the user's emotions.

[0978] Step 10:

[0979] The server dynamically changes the priority of test cases according to the user's emotional state based on the output of the emotion engine. The input is the recognized emotional state, and the output is the updated test case priority. If the user is feeling stressed, the system adjusts to prioritize important test cases.

[0980] As a concrete example, if a user wants to test a login screen, the user enters the URL into the server, the server takes a screenshot, analyzes it using a generative AI model, identifies UI elements, generates a test case, the device executes the case, sends the operation results to the server, the server determines success / failure, generates a report, analyzes the user's emotional state using an emotion engine, and changes priorities.

[0981] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0982] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0983] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0984] [Fourth embodiment]

[0985] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0986] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0987] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0988] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0989] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0990] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0991] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0992] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0993] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0994] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0995] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0996] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0997] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0998] The system of the present invention automates UI testing of applications and websites using generative AI models. The system includes three main components: the user, the server, and the terminal.

[0999] First, the user uploads the URL of the application or website or application file to be tested to the server, clearly specifying what should be tested. The user can then further specify specific devices and environments (e.g. different browsers or resolutions).

[1000] The server then receives the specified URL or uploaded file and uses a browser emulator to capture a screenshot, which is then analyzed by the generative AI model. As a result of the analysis, UI elements (buttons, input fields, links, etc.) are identified and stored in a database along with their location information.

[1001] The server then automatically generates test cases based on the detected UI elements. For example, if a login screen is detected, it generates a scenario that says "Enter username and password and click login button." The generated test cases are saved in a database.

[1002] The terminal receives these test cases from the server and executes them on the device and environment specified by the user. The terminal automatically performs operations according to the test cases, taking screenshots and error logs as operation results, and sending the results to the server.

[1003] The server analyzes the received test results and determines whether they passed or failed. It also generates a detailed report of any errors or the impact of UI changes, which is provided to the user. This report includes a summary of success or failure, an error log, and suggestions for improvement. Based on this, the user can make necessary corrections and plan retesting.

[1004] As a concrete example, consider the case where a user wants to test a login screen. The user enters the URL of the login screen into the server. The server retrieves a screenshot from the URL and analyzes it with a generative AI model to identify the username field, password field, and login button. Based on this, the server generates a test case that says, "Enter username and password and click the login button."

[1005] The terminal executes this test case and performs operations on the specified device and environment. The operation results are sent to the server, which determines whether the operation was successful or not and generates a detailed report. The user can check this report to confirm that the login operation was successful.

[1006] The above is a specific embodiment for carrying out the present invention. This system makes it possible to automate UI tests and execute them efficiently and accurately.

[1007] The processing flow will be explained below.

[1008] Step 1:

[1009] The user uploads the URL or file of the application or web page to be tested to the server, and also selects the device and environment configuration options required for the test execution.

[1010] Step 2:

[1011] The server stores the URL or file received from the user in a database. The server configures the browser emulator based on the specified device and environment information.

[1012] Step 3:

[1013] The server uses a browser emulator to open the specified URL, captures a screenshot, and saves it in the appropriate resolution.

[1014] Step 4:

[1015] The server uses the generative AI model to analyze the captured screenshot and identify UI elements (buttons, input fields, links, etc.) The server then stores the UI elements identified as a result of the analysis and their location information in a database.

[1016] Step 5:

[1017] The server automatically generates test cases based on detected UI elements. For example, if a login screen is detected, it generates a scenario that says "Enter your username and password and click the login button." The generated test cases are saved in a database.

[1018] Step 6:

[1019] The terminal receives the test cases from the server and prepares them for execution on the device and environment specified by the user. The terminal prepares the environment according to the specified setting conditions.

[1020] Step 7:

[1021] The device automatically performs UI operations according to the received test cases, such as entering text into text fields and clicking buttons, and records any screen changes or errors that occur during the operation.

[1022] Step 8:

[1023] The device sends screenshots and error logs containing the test results to the server, providing information on which operations were successful and which failed, along with the results of the operations.

[1024] Step 9:

[1025] The server analyzes the test results sent from the device, determines whether the test succeeded or failed, and analyzes the cause of errors and the impact of UI changes.

[1026] Step 10:

[1027] The server generates a detailed report based on the analysis results, including a summary of successes / failures, error logs, and suggestions for improvement, and uploads it to the user's dashboard.

[1028] Step 11:

[1029] Users can access a dashboard to view detailed reports of tests that have been performed, allowing them to plan any necessary corrections or retests.

[1030] Example 1

[1031] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1032] Traditional methods for testing the user interface (UI) of applications and websites require a significant amount of manual work, which is time-consuming and costly. It is also difficult to perform consistent testing across different devices and environments, making it difficult to ensure the reliability and efficiency of testing. This often results in extended release times.

[1033] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1034] In this invention, the server includes a means for a user to upload a URL or file of an application or webpage to be tested, a means for analyzing screenshots of the test target using a generative AI model and identifying user interface elements, a means for automatically generating test cases based on the identified user interface elements and saving them in a database, a means for a terminal to receive the test cases and automatically execute operations, a means for analyzing test results sent from the terminal, generating a report and providing it to the user, and a means for executing tests on different devices and environments. This enables efficient and accurate automation of UI testing of applications and websites.

[1035] "User" means the person or entity who uploads URLs and files to the server, selects configuration options, and receives reports to run tests on an application or website.

[1036] "Server" is a computer system that receives URLs and files submitted by users, analyzes screenshots using a generative AI model to identify user interface elements, generates test cases, stores them in a database, analyzes test results, and generates reports.

[1037] A "generative AI model" is an artificial intelligence model used to analyze captured screenshots and identify user interface elements.

[1038] A "terminal" is a device that receives test cases sent from a server, automatically executes the operations, and sends the results to the server.

[1039] A "test case" is a specific scenario or procedure that is generated based on identified user interface elements and that is used to perform automated testing.

[1040] "User interface elements" are interactive elements such as buttons, input fields, and links that appear on the screen of an application or website.

[1041] A "screenshot" is an image that captures the display state of a specified URL or uploaded file.

[1042] A "device" is a physical or virtual hardware environment for running test cases.

[1043] An "environment" is the settings and conditions for running a test (e.g., browser type, screen resolution, etc.).

[1044] A "report" is a document generated after analyzing test results, which includes a summary of successes / failures, an error log, and suggestions for improvement.

[1045] The system of the present invention automates user interface (UI) testing of applications and websites using generative AI models. In this system, a user specifies a test target, a server and a terminal work together to execute the test, analyze the results, and provide a report. An embodiment of this system is described in detail below.

[1046] First, the user uploads the URL or file of the application or website to be tested to the server. This operation clearly specifies the target to be tested. The upload is performed by accessing the server interface using a standard web browser and operating the "Enter URL" or "Upload file" button. The user can also specify specific devices and environments (e.g., different browsers or resolutions). This sets up the test environment.

[1047] Next, the server receives the specified URL or uploaded application file and takes a screenshot using a browser emulator (e.g., Selenium). Selenium is a browser automation tool that opens the specified URL and captures a screenshot of the screen. The server takes this screenshot and passes it to the generative AI model.

[1048] The server uses a generative AI model to analyze the captured screenshot and identify user interface elements (buttons, input fields, links, etc.). The AI ​​model analyzes the image data of the screenshot and identifies the location and attributes of the UI elements. The analysis results are stored in a database.

[1049] The server then automatically generates test cases based on the identified user interface elements. For example, if a login screen is detected, it generates a test case such as "Enter username and password and click login button." This generated test case is stored in a database and later sent to the device.

[1050] The terminal executes the test cases received from the server on the device and environment specified by the user. Using a testing framework (e.g., Appium), it launches an emulator for the specified device and automatically executes operations according to the test case scenario. The terminal captures screenshots and error logs as operation results and sends them to the server.

[1051] The server analyzes the test results sent from the device and determines whether the test succeeded or failed. Specifically, it compares the expected UI state with the actual operation result, and records any discrepancies as an error. A detailed report is generated based on the analysis results and provided to the user. This report includes a summary of success or failure, an error log, and suggestions for improvement. Based on this report, the user can make any necessary corrections and plan retesting.

[1052] As a concrete example, consider the case where a user wants to test a login screen. The user enters the URL of the login screen into the server and specifies the browser and resolution. The server uses Selenium to obtain a screenshot from the specified URL, and analyzes it with a generative AI model to identify the username field, password field, and login button. Based on this, a test case is generated that says "Enter username and password and click the login button." The device executes this test case and sends the results to the server. The server analyzes the results, generates a report, and provides it to the user.

[1053] An example of a prompt statement might be something like, "Enter the URL of the login screen into the server, perform the operations according to the generated test case, and generate a prompt to evaluate the results."

[1054] By implementing the invention in this manner, it is possible to automate UI testing of applications and websites efficiently and accurately.

[1055] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1056] Step 1:

[1057] The user specifies the test target

[1058] Input: The URL or application file to test.

[1059] Output: The URL or file received by the server.

[1060] Specific operation: A user accesses the server interface using a web browser and operates the "Enter URL" or "Upload file" button. The user enters or uploads the target URL or file and presses the send button. The server receives the sent URL or file.

[1061] Step 2:

[1062] The user specifies the test environment

[1063] Input: Preferences such as device, browser, and resolution.

[1064] Output: The configuration information received by the server.

[1065] What happens: The user selects the appropriate option from the "Device Selection" or "Browser Selection" drop-down list on the server's interface. The user's selection is sent to the server, which receives this preference information.

[1066] Step 3:

[1067] Server takes screenshot

[1068] Input: URLs, application files, and configuration information submitted by the user.

[1069] Output: The captured screenshot image.

[1070] What happens: The server launches a browser emulator, such as Selenium, to open the specified URL or file. The browser emulator opens the URL with the specified configuration (e.g., browser type and resolution) and captures a screenshot of the screen. The server saves the screenshot.

[1071] Step 4:

[1072] The server analyzes the UI elements

[1073] Input: The captured screenshot image.

[1074] Output: Information about the identified UI element (position and attributes).

[1075] How it works: The server passes a screenshot as input to the generative AI model. The generative AI model analyzes the image, identifies UI elements (e.g., buttons, input fields, links, etc.), and extracts their locations and attributes. The analysis results are stored in a database.

[1076] Step 5:

[1077] The server generates the test cases

[1078] Input: Information about the identified UI element.

[1079] Output: Generated test cases.

[1080] Specific operation: The server automatically generates test cases based on UI element information retrieved from the database. For example, if a login screen is detected, a test scenario will be created that says "Enter username and password and click login button." The generated test cases are saved in the database.

[1081] Step 6:

[1082] The device runs the test case

[1083] Input: Test cases received from the server.

[1084] Output: Operation results (screenshots and error logs).

[1085] Specific operation: The terminal executes the test cases received from the server on the device and environment specified by the user. The terminal uses a testing framework (e.g., Appium) to launch the emulator for the specified device and automatically perform operations according to the test case scenario. The terminal captures screenshots and error logs as operation results and sends them to the server.

[1086] Step 7:

[1087] The server analyzes the test results

[1088] Input: Operation results sent from the device (screenshots and error logs).

[1089] Output: Analysis result (success / failure).

[1090] Specific operation: The server analyzes the operation results sent from the device. It compares the expected UI state (for example, whether a button is clickable or a form is submitted) with the actual operation result. This comparison determines whether the test is successful or failed.

[1091] Step 8:

[1092] The server generates the report

[1093] Input: Analysis data of test results.

[1094] Output: Detailed report (success / failure summary, error log, improvement suggestions).

[1095] Specific operation: The server generates a detailed report based on the analysis results, including a summary of the test success / failure, a log of errors that occurred, and suggestions for improvement. The generated report is then provided to the user.

[1096] (Application example 1)

[1097] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1098] Product picking work at logistics centers requires a great deal of time and effort, and is prone to human error. In particular, it is difficult to quickly and accurately pick large quantities of products in a large warehouse, so there is a need to improve work efficiency and reduce errors.

[1099] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1100] In this invention, the server includes a means for a user to upload a URL or file of an application or webpage to be tested, a means for analyzing screenshots of the test target using a generative AI model and identifying UI elements, a means for automatically generating test cases based on the identified UI elements and saving them in a database, a means for a terminal to receive the test cases and automatically execute operations, a means for analyzing test results sent from the terminal, generating a report and providing it to the user, and a means for supporting product picking using smart glasses at a logistics center, thereby enabling improved efficiency and accuracy of picking work at logistics centers.

[1101] "User" refers to the person who uploads the URL or file of the application or web page to be tested and reviews the generated test cases and reports.

[1102] "Generative AI Model" refers to an artificial intelligence model used to analyze a given screenshot and identify UI elements.

[1103] A "test case" refers to a scenario that is automatically generated based on identified UI elements to verify the operation and behavior of the system.

[1104] "Terminal" refers to a device or equipment that receives the generated test cases and automatically executes the operations.

[1105] "Smart glasses" refer to devices worn by workers at logistics centers that display information in real time to assist with picking operations.

[1106] "Picking" refers to the task of removing specified products from shelves at the logistics center and preparing them for shipment.

[1107] "Report" refers to a detailed report provided to the User after analyzing the test results sent from the Terminal.

[1108] A "logistics center" refers to a facility that stores, manages, and ships goods.

[1109] This invention is a system that utilizes a generative AI model and smart glasses to improve the efficiency and accuracy of product picking work in logistics centers. Specifically, the system uses a generative AI model to identify the location of products and shelves, and displays instructions to workers through the smart glasses.

[1110] First, the user uploads the URL or file of the application or web page to be tested to the server, which then recognizes the test target using the generative AI model.

[1111] The server then receives the specified URL or file and captures screenshots from a browser emulator or a real-world distribution center's surveillance cameras. These screenshots are then analyzed by a generative AI model to identify UI elements and shelf and product locations. This data is then stored in a database, and test cases are automatically generated based on the identified UI elements.

[1112] The generated test cases are received by a device (such as smart glasses) and automatically performed in the set environment. The smart glasses are worn by workers and display real-time information on shelves and product locations, as well as picking instructions. This display allows workers to accurately locate products in a short amount of time and perform picking tasks.

[1113] The results of operations performed by the terminal (e.g., the progress of a picking operation) are sent to the server, which analyzes them and generates a report that includes a success / failure summary, an error log, and suggestions for improvement, which the user can review and correct or retest.

[1114] The specific technologies used are OpenCV for image processing and machine learning models for generative AI models. The smart glasses used are commercially available (e.g., Vuzix, Google Glass Enterprise Edition). The entire system is built by combining a browser emulator, a database management system, and a communication protocol.

[1115] An example of a prompt is:

[1116] "To support product picking in the warehouse, please recognize the product and its location from the captured image. Specifically, please analyze the label or barcode and indicate the corresponding product."

[1117] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1118] Step 1:

[1119] The user uploads the URL or file of the application or web page to be tested to the server. This input includes the URL or file path, based on which the server identifies the target for testing.

[1120] Step 2:

[1121] The server receives the specified URL or file and captures a screenshot from a browser emulator or the actual surveillance camera at the distribution center. The input is the specified URL or file, and the captured screenshot is generated as the output. Specifically, the specified URL is opened using a browser emulator and the screen is captured.

[1122] Step 3:

[1123] The server uses a generative AI model to analyze the captured screenshot and identify UI elements, shelves, and product locations. The input is a screenshot, and the output is the identified UI elements, shelves, and product locations. This analysis uses an image recognition method using a machine learning model.

[1124] Step 4:

[1125] The server automatically generates test cases based on the identified UI elements and shelf / product locations and saves them in a database. The input is data on UI elements and shelf / product locations, and the generated test cases are obtained as output. Specifically, a test case is created that specifies which product to pick from which shelf based on the product's barcode and label information.

[1126] Step 5:

[1127] The terminal (smart glasses) receives the test cases generated from the server and automatically executes the operations. The test cases are input and the operation results are output. The smart glasses display shelf and product location information and picking instructions in real time.

[1128] Step 6:

[1129] The terminal sends the results of the operation it performed to the server. The input is the operation result of the smart glasses, and the output is the operation result data sent to the server. This operation result includes the progress of the picking work and an error log.

[1130] Step 7:

[1131] The server analyzes the received operation results, generates a report, and provides it to the user. The operation result data is input, and a report is generated as output. This report includes a summary of success / failure, an error log, and suggestions for improvement. The user can make corrections or retest based on this report.

[1132] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1133] Our system not only automates UI testing using generative AI models, but also combines an emotion engine to recognize user emotions and dynamically adjust the testing process accordingly. This system consists of four main components: the user, the server, the device, and the emotion engine.

[1134] First, the user uploads the URL or file of the application or web page to be tested to the server. The user also selects the device and environment configuration options required for the test execution. The emotion engine then analyzes the user's facial expressions and tone of voice to recognize the user's emotional state.

[1135] The server then receives the specified URL or uploaded file and uses a browser emulator to capture a screenshot, which is then analyzed by the generative AI model. As a result of the analysis, UI elements (buttons, input fields, links, etc.) are identified and stored in a database along with their location information.

[1136] The server then automatically generates test cases based on the detected UI elements. For example, if a login screen is detected, it generates a scenario that says "Enter username and password and click login button." The generated test cases are saved in a database.

[1137] The device receives these test cases from the server and prepares to run them on the device and environment specified by the user. The device automatically performs operations according to the test cases, taking screenshots and error logs as operation results. The obtained results are sent to the server.

[1138] The server analyzes the received test results, determines whether they passed or failed, and generates a detailed report of any errors or the impact of UI changes, which is then provided to the user. This report includes a summary of success or failure, an error log, and suggestions for improvement.

[1139] The emotion engine analyzes the user's emotional state in real time and dynamically changes the priority of test cases based on that information. For example, if the user is feeling stressed, the system will adjust the priority of test cases so that more important tests are executed first. The user's emotional state is also reflected in the test result report and is used for analysis and improvement suggestions.

[1140] As a concrete example, consider the case where a user wants to test a login screen. The user enters the URL of the login screen into the server. The server retrieves a screenshot from the URL and analyzes it with a generative AI model to identify the username field, password field, and login button. Based on this, the server generates a test case that says, "Enter username and password and click the login button."

[1141] The device runs the test cases and performs operations on the specified device and environment. The results of the operations are sent to the server, which determines whether the operation was successful or not and generates a detailed report. The emotion engine analyzes the user's facial expressions and tone of voice, and if it determines that the user is feeling stressed, it adds that information to the report and makes suggestions for improvement.

[1142] The above is a specific embodiment of the present invention, which not only automates UI testing and executes it efficiently and accurately, but also dynamically adjusts the testing process by taking into account the user's emotional state.

[1143] The processing flow will be explained below.

[1144] Step 1:

[1145] The user uploads the URL or file of the application or web page to be tested to the server, and then selects the device and environment configuration options required for the test execution.

[1146] Step 2:

[1147] The emotion engine analyzes the user's facial expressions and tone of voice to recognize the user's emotional state, and stores this information in a database within the system.

[1148] Step 3:

[1149] The server stores the URL or file received from the user in a database. The server configures the browser emulator based on the specified device and environment information.

[1150] Step 4:

[1151] The server uses a browser emulator to open the specified URL, captures a screenshot, and saves it in the appropriate resolution.

[1152] Step 5:

[1153] The server uses the generative AI model to analyze the captured screenshot and identify UI elements (buttons, input fields, links, etc.) The server then stores the UI elements identified as a result of the analysis and their location information in a database.

[1154] Step 6:

[1155] The server automatically generates test cases based on detected UI elements. For example, if a login screen is detected, it generates a scenario that says "Enter your username and password and click the login button." The generated test cases are saved in a database.

[1156] Step 7:

[1157] The terminal receives the test cases from the server and prepares them for execution on the device and environment specified by the user. The terminal prepares the environment according to the specified setting conditions.

[1158] Step 8:

[1159] The device automatically performs UI operations according to the received test cases, such as entering text into text fields and clicking buttons, and records any screen changes or errors that occur during the operation.

[1160] Step 9:

[1161] The device sends screenshots and error logs containing the test results to the server, providing information on which operations were successful and which failed, along with the results of the operations.

[1162] Step 10:

[1163] The server analyzes the test results sent from the device. The server determines whether the test was successful or failed, and analyzes the cause of any errors and the impact of UI changes. The emotion engine also references the user's emotional state and reflects that information in the test result report.

[1164] Step 11:

[1165] The server generates a detailed report based on the analysis results. The report includes a summary of success / failure, an error log, and suggestions for improvement. The report also incorporates data from the emotion engine, and includes the emotions felt by the user during the test. This report is then uploaded to the user's dashboard.

[1166] Step 12:

[1167] Users can access the dashboard and view detailed reports of the tests they have performed. Based on the reports, users can make necessary corrections and plan retests. The reports also reflect the user's emotional state, allowing for more accurate improvements.

[1168] Example 2

[1169] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1170] Conventional UI test automation systems have limited adaptability to multiple devices and environments, and have faced challenges in analyzing test results and generating reports efficiently. Furthermore, they do not dynamically adjust the test process to take into account the user's emotional state. This often leads to user frustration, and improvements to the UX are needed.

[1171] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1172] In this invention, the server includes: a means for a user to upload a URL or file of an application or webpage to be tested; a means for analyzing screenshots of the test target using a generative AI model to identify UI elements; a means for automatically generating test cases based on the identified UI elements and saving them in a database; a means for a terminal to receive the test cases and automatically execute operations; a means for analyzing test results sent from the terminal, generating a report and providing it to the user; and a means for recognizing the user's emotional state in real time and dynamically adjusting the test process based on the result. This improves the efficiency and accuracy of UI testing and enables dynamic adjustment of the test process taking the user's emotional state into account.

[1173] "User" means a person or entity that utilizes the system to upload the URL or file of an application or web page to be tested and select the configuration options required to run the test.

[1174] "Test subject" refers to the URL or file of an application or web page that a user uploads to the system, and refers to the entire software asset that is the subject of testing.

[1175] A "generative AI model" is an artificial intelligence model that analyzes screenshots to identify UI elements and automatically generate test cases.

[1176] A "screenshot" is a screen image of the application or web page under test that the system captures using a browser emulator.

[1177] A "UI element" is a user interface component (e.g., button, input field, link, etc.) identified from a screenshot by a generative AI model.

[1178] A "test case" is a set of operational scenarios that the system executes, automatically generated based on identified UI elements.

[1179] A "database" is an information storage means for saving information such as identified UI elements and generated test cases.

[1180] A "terminal" is an electronic device that executes test cases received from a server on a specified device or in a specified environment.

[1181] A "report" is a detailed report that analyzes the test results sent from the device and includes a success / failure determination, error log, and improvement suggestions.

[1182] "Emotional state" refers to the user's psychological state, recognized in real time by analyzing the user's facial expressions and tone of voice.

[1183] An "emotion engine" is an analysis device or software for analyzing a user's emotional state in real time and dynamically adjusting the testing process based on that analysis.

[1184] MODE FOR CARRYING OUT THE INVENTION

[1185] Our system automates user interface (UI) testing using generative AI models and combines them with an emotion engine to recognize user emotions and dynamically adjust the testing process accordingly. The system consists of four main components: the user, the server, the device, and the emotion engine.

[1186] server

[1187] The server receives the URL or file of the application or web page to be tested uploaded by the user. Specifically, the server uses a browser emulator (e.g., Selenium or Puppeteer) to take a screenshot of the specified URL or file. This screenshot is analyzed by a generative AI model (e.g., OpenAI's GPT-3) to identify UI elements (e.g., buttons, input fields, links, etc.). The location information of the UI elements obtained as a result of the analysis is stored in a database. The server also automatically generates test cases based on the identified UI elements, and the generated test cases are also stored in the database. The server then analyzes the test results sent from the device and determines whether they succeeded or failed. A detailed report is generated about errors and the impact of UI changes and provided to the user.

[1188] Terminal

[1189] The terminal executes the test cases received from the server on the device and environment specified by the user. Specifically, the terminal automatically performs operations according to the test cases and captures the operation results such as screenshots and error logs. The captured results are then sent to the server.

[1190] User

[1191] The user uploads the URL or file of the application or web page to be tested to the server, and then selects the device and environment configuration options required for the test execution. In addition, the emotion engine monitors the user's emotional state in real time. This is achieved by analyzing the user's facial expressions and tone of voice.

[1192] Emotion Engine

[1193] An emotion engine (e.g., Microsoft Azure Cognitive Services) analyzes the user's emotional state in real time and dynamically changes the priority of test cases. For example, if the user is feeling stressed, the system will change the priority of test cases and adjust the system so that more important tests are executed first. The user's emotional state is also reflected in the test result report and is used for analysis and improvement suggestions.

[1194] Specific examples

[1195] For example, if a user wants to test a login screen, they enter the URL of the login screen into the server. The server receives the URL and uses a browser emulator to take a screenshot of the login screen. This screenshot is analyzed by a generative AI model to identify the UI elements: username field, password field, and login button. Based on this, the server automatically generates a test case that says, "Enter username and password and click the login button."

[1196] The device runs this test case, performing operations on the specified device and environment. Screenshots and error logs obtained as a result of the operations are sent to the server, which determines whether the operation was successful or failed and generates a detailed report. The emotion engine analyzes the user's facial expressions and tone of voice, and if it determines that the user is feeling stressed, it adds that information to the report and makes suggestions for improvement.

[1197] Prompt Sentence Examples

[1198] "Write a program like the one below. A user enters the URL of a login screen into the server. The server receives the URL and takes a screenshot of the login screen using a browser emulator. The server analyzes this screenshot with a generative AI model and generates a test case (enter username and password and click the login button) based on the identified UI elements (username field, password field, login button). The generated test case is then sent to the terminal. The terminal executes the test case on the specified device and environment and sends the obtained operation results to the server. The server determines success / failure and generates a detailed report. The emotion engine analyzes the user's emotional state in real time and adjusts the testing process."

[1199] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1200] Step 1: User uploads URL or file to test

[1201] Specific behavior:

[1202] The user uploads the URL or file of the application or web page to be tested to the server, and also selects the device and environment configuration options required for the test execution.

[1203] input:

[1204] Test target URL or file, device and environment configuration options

[1205] output:

[1206] URLs or files and configuration data stored on the server

[1207] Step 2: The server receives the data and takes a screenshot

[1208] Specific behavior:

[1209] The server receives the URL or file provided by the user, then uses a browser emulator (e.g., Selenium, Puppeteer) to take a screenshot of the specified URL or file.

[1210] input:

[1211] URL or file from the user

[1212] output:

[1213] Captured screenshots

[1214] Step 3: Generate screenshots and analyze them with an AI model to identify UI elements

[1215] Specific behavior:

[1216] The server sends the captured screenshot to a generative AI model (e.g., OpenAI's GPT-3) to identify UI elements (buttons, input fields, links, etc.).

[1217] input:

[1218] Captured screenshots

[1219] output:

[1220] The location of the identified UI element

[1221] Step 4: The server generates the test cases and saves them in the database

[1222] Specific behavior:

[1223] The server automatically generates test cases based on the identified UI elements and saves them in a database. For example, if a login screen is detected, a scenario such as "Enter username and password and click login button" is generated.

[1224] input:

[1225] Location information of UI elements

[1226] output:

[1227] Generated test cases, saved test cases

[1228] Step 5: The device receives the test case and prepares to execute it

[1229] Specific behavior:

[1230] The terminal analyzes the test cases received from the server and prepares to execute the test cases on the device and in the environment specified by the user.

[1231] input:

[1232] Test cases received from the server

[1233] output:

[1234] Test cases ready to run

[1235] Step 6: The device performs the operation according to the test case

[1236] Specific behavior:

[1237] The device automatically performs operations according to the test cases, and captures screenshots and error logs as a result of the operations.

[1238] input:

[1239] Test cases ready to run

[1240] output:

[1241] Obtained operation results (screenshots, error logs)

[1242] Step 7: The server analyzes the test results and generates a report

[1243] Specific behavior:

[1244] The server analyzes the test results sent from the device and determines whether the test succeeded or failed. It also generates a detailed report on the impact of errors and UI changes and provides it to the user.

[1245] input:

[1246] Operation results sent from the device

[1247] output:

[1248] Success / failure results and detailed reports

[1249] Step 8: The emotion engine analyzes the user's emotional state and adjusts the process

[1250] Specific behavior:

[1251] The emotion engine analyzes the user's facial expressions and tone of voice to recognize the user's emotional state in real time, and dynamically reprioritizes test cases and adjusts the testing process accordingly.

[1252] input:

[1253] The user's facial expression and tone of voice

[1254] output:

[1255] Analyzed emotional states, adjusted testing process

[1256] (Application example 2)

[1257] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1258] Current UI testing automation systems do not consider the user's emotional state during the testing process, which can lead to a poor user experience. Furthermore, the testing process may not proceed effectively when the user is under stress. Furthermore, because electronic payment services often involve situations where users are prone to experiencing high levels of stress, a more advanced testing system incorporating emotion recognition is needed.

[1259] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to upload a URL or file of an application or webpage to be tested; means for analyzing a screenshot of the test target using a generative AI model and identifying UI elements; means for automatically generating test cases based on the identified UI elements and saving them in a database; means for a terminal to receive the test cases and automatically execute operations; means for analyzing test results sent from the terminal, generating a report, and providing it to the user; and means for recognizing the user's emotional state using an emotion engine that analyzes the user's facial expressions and tone of voice and dynamically changing the priority of test cases according to the emotional state. This enables advanced UI testing automation that takes the user's emotional state into consideration, making it possible to provide a high-quality user experience even in electronic payment services.

[1260] "User" is the person or system that uploads the URL or file of the application or web page under test and initiates the testing process.

[1261] "Generative AI model" refers to an artificial intelligence model used to analyze screenshots and identify UI elements.

[1262] A "screenshot" is an image captured from the screen of the application or web page being tested.

[1263] A "UI element" is an operable component in a user interface, such as a button, input field, or link.

[1264] A "test case" is a scenario of operations that is executed to specifically test a system.

[1265] A "database" is a digital storage for storing information such as test cases and test results.

[1266] A "terminal" is a device that receives test cases and automatically executes the operations.

[1267] An "emotion engine" is a system that analyzes a user's facial expressions and tone of voice to recognize the user's emotional state in real time.

[1268] An "emotional state" is an emotional state that a user is feeling, such as stress, joy, or impatience.

[1269] "Priority" is a concept that indicates the importance and order in which test cases are executed.

[1270] "Dynamic change" means changing the order and importance of test cases in real time according to the user's emotional state.

[1271] A "report" is a document or file that summarizes the test results and the analysis of the user's emotional state.

[1272] The system of the present invention not only automates UI testing using a generative AI model, but also combines it with an emotion engine to recognize user emotions and dynamically adjust the testing process accordingly. Specific embodiments are described below.

[1273] The system consists of four main components: the user, the server, the device, and the emotion engine. First, the user uploads the URL or file of the application or web page to be tested to the server. The user also selects the device and environment configuration options required for the test execution. The emotion engine then analyzes the user's facial expressions and tone of voice to recognize the user's emotional state.

[1274] The server receives the specified URL or uploaded file and takes a screenshot using a browser emulator. This screenshot is then analyzed by a generative AI model. As a result of the analysis, UI elements (buttons, input fields, links, etc.) are identified and stored in a database along with their location information. The server then automatically generates test cases based on the detected UI elements. For example, if a login screen is detected, a scenario such as "Enter your username and password and click the login button" is generated. The generated test cases are then stored in a database.

[1275] The device receives these test cases from the server and prepares to run them on the device and environment specified by the user. The device automatically performs operations according to the test cases, capturing screenshots and error logs as operation results. The captured results are sent to the server. The server analyzes the received test results and determines whether the test succeeded or failed. It also generates a detailed report on the impact of errors and UI changes and provides it to the user. This report includes a summary of success or failure, error logs, and improvement suggestions.

[1276] The emotion engine analyzes the user's emotional state in real time and dynamically changes the priority of test cases based on that information. For example, if the user is feeling stressed, the system will adjust the priority of test cases so that more important tests are executed first. The user's emotional state is also reflected in the test result report and is used for analysis and improvement suggestions.

[1277] As a concrete example, consider the case where a user wants to perform UI testing for an electronic payment app. The user enters the URL of the login screen into the server. The server retrieves a screenshot from the URL and analyzes it with a generative AI model to identify the username field, password field, and login button. Based on this, the server generates a test case that says, "Enter username and password and click login button." The terminal executes this test case, performing the operation in the specified device and environment. The operation results are sent to the server, which determines whether the operation was successful or failed and generates a detailed report. The emotion engine analyzes the user's facial expressions and tone of voice, and if it determines that the user is stressed, it adds that information to the report and makes suggestions for improvement.

[1278] An example of a prompt sentence is, "Identify the login button in this screenshot, enter your username and password, and generate a test case in which you click the button. Also, if the user seems stressed, please run this case first."

[1279] This will improve the quality of electronic payment services and the user experience.

[1280] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1281] Step 1:

[1282] The user uploads the URL or file of the application or web page to be tested to the server. At the same time, the user selects the device and environment configuration options required for test execution. The input is the URL or file of the application or web page and the environment settings, and the output is the URL or file and configuration information saved on the server.

[1283] Step 2:

[1284] The server receives the specified URL or uploaded file and takes a screenshot using a browser emulator. The input is the URL or file uploaded in step 1, and the output is the screenshot taken.

[1285] Step 3:

[1286] Using a generative AI model, the server analyzes the screenshot and identifies UI elements. The input is the screenshot, and the output is the location information of the identified UI elements. The generative AI model uses image recognition techniques to identify UI elements such as buttons and input fields in the screenshot.

[1287] Step 4:

[1288] The server automatically generates test cases based on the identified UI elements and saves the data in a database. The input is the location information of the UI elements, and the output is the generated test case. The generated test case might be a scenario such as "Enter your username and password and click the login button."

[1289] Step 5:

[1290] The terminal receives the generated test case from the server and prepares to execute it on the device and environment specified by the user. The input is the test case and environment setting information, and the output is the status of the test execution preparation. The terminal prepares to execute the test case based on the appropriate environment setting.

[1291] Step 6:

[1292] The terminal automatically performs operations according to the test cases and captures screenshots and error logs as the results of the operations. The input is the test cases, and the output is screenshots and error logs as the results of the operations. The terminal performs operations according to the set scenario and records the results.

[1293] Step 7:

[1294] The server receives the test results sent from the device, analyzes them, and determines whether the test was successful or not. The input is a screenshot of the operation results and an error log, and the output is the success / failure result. The server determines whether the test was successful or not based on the analysis results.

[1295] Step 8:

[1296] The server generates a detailed report of errors and the impact of UI changes and provides it to the user. The input is the analysis results and the output is a detailed report. The report includes a success / failure summary, error log, and improvement suggestions.

[1297] Step 9:

[1298] The emotion engine analyzes the user's facial expressions and tone of voice in real time to recognize the user's emotional state. The input is the user's facial expressions and tone of voice, and the output is the recognized emotional state. The emotion engine uses image analysis and voice analysis technology to determine the user's emotions.

[1299] Step 10:

[1300] The server dynamically changes the priority of test cases according to the user's emotional state based on the output of the emotion engine. The input is the recognized emotional state, and the output is the updated test case priority. If the user is feeling stressed, the system adjusts to prioritize important test cases.

[1301] As a concrete example, if a user wants to test a login screen, the user enters the URL into the server, the server takes a screenshot, analyzes it using a generative AI model, identifies UI elements, generates a test case, the device executes the case, sends the operation results to the server, the server determines success / failure, generates a report, analyzes the user's emotional state using an emotion engine, and changes priorities.

[1302] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1303] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1304] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1305] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1306] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1307] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1308] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1309] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1310] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1311] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1312] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1313] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1314] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1315] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1316] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1317] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1318] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1319] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1320] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1321] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1322] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1323] The following is further disclosed regarding the above embodiment.

[1324] (Claim 1)

[1325] A means for the user to upload the URL or file of the application or web page to be tested;

[1326] A means of analyzing screenshots of test subjects and identifying UI elements using a generative AI model;

[1327] A means for automatically generating test cases based on the identified UI elements and storing them in a database;

[1328] means for the terminal to receive the test case and automatically execute the operation;

[1329] means for analyzing the test results sent from the terminal, generating a report, and providing the report to the user;

[1330] A system including:

[1331] (Claim 2)

[1332] The system of claim 1, characterized in that the generative AI model has the ability to adapt test cases to different devices and environments.

[1333] (Claim 3)

[1334] 2. The system according to claim 1, further comprising means for a user to select a setting option and configure the device and environment to be tested based on the selected setting option.

[1335] "Example 1"

[1336] (Claim 1)

[1337] A means for the user to upload the URL or file of the application or web page to be tested;

[1338] a means for analyzing screenshots of a test subject using a generative AI model to identify user interface elements;

[1339] means for automatically generating and storing test cases in a database based on the identified user interface elements;

[1340] means for the terminal to receive the test case and automatically execute the operation;

[1341] means for analyzing the test results sent from the terminal, generating a report, and providing the report to the user;

[1342] A means to run tests against different devices and environments;

[1343] A system including:

[1344] (Claim 2)

[1345] The system of claim 1, characterized in that the generative AI model has the ability to adapt test cases for different devices and environments.

[1346] (Claim 3)

[1347] 2. The system according to claim 1, further comprising means for a user to select a setting option and configure the device and environment to be tested based on the selected setting option.

[1348] "Application Example 1"

[1349] (Claim 1)

[1350] A means for the user to upload the URL or file of the application or web page to be tested;

[1351] A means of analyzing screenshots of test subjects and identifying UI elements using a generative AI model;

[1352] A means for automatically generating test cases based on the identified UI elements and storing them in a database;

[1353] means for the terminal to receive the test case and automatically execute the operation;

[1354] means for analyzing the test results sent from the terminal, generating a report, and providing the report to the user;

[1355] A system including a means for assisting product picking in a logistics center using smart glasses.

[1356] (Claim 2)

[1357] The system of claim 1, characterized in that the generative AI model has the ability to adapt test cases to different devices and environments.

[1358] (Claim 3)

[1359] 2. The system according to claim 1, further comprising means for a user to select a setting option and configure the device and environment to be tested based on the selected setting option.

[1360] "Example 2: Combining Emotion Engines"

[1361] (Claim 1)

[1362] A means for the user to upload the URL or file of the application or web page to be tested;

[1363] A means of analyzing screenshots of test subjects and identifying UI elements using a generative AI model;

[1364] A means for automatically generating test cases based on the identified UI elements and storing them in a database;

[1365] means for the terminal to receive the test case and automatically execute the operation;

[1366] means for analyzing the test results sent from the terminal, generating a report, and providing the report to the user;

[1367] a means for recognizing a user's emotional state in real time and dynamically adjusting the testing process based thereon;

[1368] A system including:

[1369] (Claim 2)

[1370] 2. The system of claim 1, wherein the generative AI model has the ability to adapt test cases for different devices and environments.

[1371] (Claim 3)

[1372] 10. The system of claim 1, further comprising means for a user to select configuration options and configure the device and environment under test based on the selected options.

[1373] "Application example 2 when combining emotion engines"

[1374] (Claim 1)

[1375] A means for the user to upload the URL or file of the application or web page to be tested;

[1376] A means of analyzing screenshots of test subjects and identifying UI elements using a generative AI model;

[1377] A means for automatically generating test cases based on the identified UI elements and storing them in a database;

[1378] means for the terminal to receive the test case and automatically execute the operation;

[1379] means for analyzing the test results sent from the terminal, generating a report, and providing the report to the user;

[1380] a means for recognizing a user's emotional state using an emotion engine that analyzes the user's facial expressions and tone of voice, and dynamically changing the priority of test cases according to the emotional state;

[1381] A system including:

[1382] (Claim 2)

[1383] The system of claim 1, characterized in that the generative AI model has the ability to adapt test cases to different devices and environments.

[1384] (Claim 3)

[1385] 2. The system according to claim 1, further comprising means for a user to select a setting option and configure the device and environment to be tested based on the selected setting option. [Explanation of symbols]

[1386] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for the user to upload the URL or file of the application or web page to be tested; A means of analyzing screenshots of test subjects and identifying UI elements using a generative AI model; A means for automatically generating test cases based on the identified UI elements and storing them in a database; means for the terminal to receive the test case and automatically execute the operation; means for analyzing the test results sent from the terminal, generating a report, and providing the report to the user; A system including:

2. The system of claim 1, wherein the generative AI model has the ability to adapt test cases for different devices and environments.

3. 2. The system according to claim 1, further comprising means for allowing a user to select a setting option and to set a device or environment to be tested based on the selected setting option.

Images