System

Abstract

A system is provided.SOLUTION: A system, comprising: means for loading a generative artificial intelligence model; means for generating a test scenario using the generative artificial intelligence model; means for sending the test scenario to a terminal; means for the terminal to operate an application according to the test scenario; means for collecting and sending a result of the test to a server; means for analyzing the test result and providing a failure cause and a correction method at the server; and means for retesting the corrected application.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional application testing is often performed manually, requiring significant effort and cost to create test cases, execute them, and analyze the results. Even when outsourcing, cost and timeline constraints can prevent a wide variety of test cases from being adequately covered. Furthermore, because analysis and suggested fixes after test failures are performed manually, problem resolution is time-consuming and inefficient. [Means for solving the problem]

[0005] We provide a system that automatically generates test scenarios using a generative artificial intelligence model and allows a terminal to operate an application based on those scenarios. This system collects test results and sends them to a server, which analyzes the results and provides the cause of failure and how to correct it. It also includes a means for retesting the corrected application. This system automates and streamlines testing, enabling the rapid coverage of a variety of test cases.

[0006] A "generative artificial intelligence model" is a machine learning or deep learning model used to automatically generate test scenarios for an application.

[0007] A "test scenario" represents a series of test cases including user interface operations and inputs of an application, and is a file generated in JSON format or the like.

[0008] A "terminal" is a physical or virtual computing device used to actually operate an application according to test scenarios and collect results.

[0009] An "application" is software that can be operated by a user and is a program to be tested.

[0010] "Test results" refers to information such as success, failure, and error messages obtained when a terminal executes a test scenario.

[0011] A "server" is a central control system that generates test scenarios, collects results, and analyzes them.

[0012] The "cause of failure" is the specific cause of an error or defect that occurred during the execution of a test scenario.

[0013] "Correction methods" are specific steps or suggestions for resolving the discovered causes of failure.

[0014] "Retesting" is the process of re-executing test scenarios on a modified application to confirm the effectiveness of the modifications. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention is a system that uses a generative artificial intelligence model to automate application testing and reduce the effort and cost. Specific embodiments of this system are described below.

[0037] First, the server loads the generative AI model, which uses deep learning techniques and is used to generate test scenarios and analyze the results.

[0038] Next, the user inputs information about the application under test into the server from their device, including the application path, settings, and test scope. Once the server receives the information from the user, it uses a generative AI model to generate a test scenario. This scenario includes, for example, various UI operations based on different input values ​​and operation patterns.

[0039] The server sends the generated test scenario to the terminal. The terminal receives the test scenario, launches the application, and performs operations according to the specified scenario. As a concrete example, consider testing a user registration form. The terminal enters a username, email address, and password in the correct format and clicks the registration button. This also includes entering an invalid email address or a short password. This covers both normal and error behavior.

[0040] As a result of the test, the device collects logs of successes, failures, error messages, etc. and sends them to the server. The server analyzes the received log data and identifies the cause of the failure. For example, if there is a problem with the validation of an email address, it will detect this and generate advice such as "the validation logic needs to be corrected."

[0041] The application is modified based on the modification information provided by the user. Once the modification is complete, the server generates a new test scenario, sends it to the device, and performs a retest. Through the retest, it is confirmed whether the modification was performed properly, and the results are collected and analyzed again before a final report is made.

[0042] The above is a specific embodiment of the system based on the present invention. This system utilizes AI to automatically perform a variety of tests, thereby improving the efficiency and quality of tests.

[0043] The processing flow will be explained below.

[0044] Step 1:

[0045] The server loads the generative AI model. Specifically, it extracts the file of the generative AI model into memory and makes the model ready for use.

[0046] Step 2:

[0047] The user enters information about the application to be tested into the server from the terminal. Specifically, the user enters the application path (e.g., " / path / to / application") and initial settings (e.g., test scope and input item definitions), and sends them to the server.

[0048] Step 3:

[0049] The server generates test scenarios using a generative AI model based on the application information it receives. For example, it creates test scenarios for valid input (username, email address, password in the correct format) and invalid input (invalid email address, password that is too short).

[0050] Step 4:

[0051] The server saves the generated test scenario in JSON format, ensuring that the scenario is used consistently in subsequent processing.

[0052] Step 5:

[0053] The server sends the saved test scenario to the device, and passes the scenario to the device's API endpoint.

[0054] Step 6:

[0055] The terminal loads the test scenario received from the server and launches the application. Specifically, it opens the application's executable file and makes the GUI operable.

[0056] Step 7:

[0057] The device automatically executes UI operations according to the test scenario, such as entering a username, email address, and password, and clicking the send button.

[0058] Step 8:

[0059] The device collects the results of the test, including successful and unsuccessful operations and any error messages displayed.

[0060] Step 9:

[0061] The test results collected by the device are sent to the server. Specifically, the test results are compiled in JSON format and sent to the server's receiving API.

[0062] Step 10:

[0063] The server analyzes the test results it receives and uses a generative AI model to identify the specific cause of the failure and suggest a fix based on that. For example, it generates a message such as "Email address validation logic is required."

[0064] Step 11:

[0065] The server notifies the user of the proposed correction method. Possible notification methods include displaying the result on the dashboard or sending an email.

[0066] Step 12:

[0067] The user modifies the application based on the provided fixes, specifically by modifying the code in the development environment and building the modified version.

[0068] Step 13:

[0069] The user notifies the server that the modification is complete, for example, by sending the modified application path to the server.

[0070] Step 14:

[0071] The server then generates a new test scenario and sends it to the device, which then runs a retest to verify whether the modifications were made properly.

[0072] Step 15:

[0073] The retest results are collected, analyzed, and finally reported to the user. After confirming that the problems discovered in the previous process have been resolved, a final report is made.

[0074] The above are the specific processing steps of the system.

[0075] Example 1

[0076] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0077] The application testing process is not automated enough, resulting in labor-intensive and costly tasks. Creating test scenarios and analyzing test results takes a lot of time, reducing overall testing efficiency. Furthermore, human error and insufficient test coverage make it difficult to improve application quality.

[0078] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0079] In this invention, the server includes means for loading a generative AI model using a generating means, means for a user to input application information to be tested, means for generating a test scenario using the generative AI model, means for transmitting the test scenario to a terminal, means for the terminal to launch and operate the application according to the test scenario, means for testing various input values ​​to test both normal and abnormal operation, means for collecting test results and transmitting them to the server, means for the server to analyze the test results and provide causes of failure and correction methods, means for retesting the corrected application based on the correction information, and means for reporting and analyzing the final results through the retest and generating a report. This makes it possible to automate the application testing process, reduce effort and cost, and improve overall testing efficiency.

[0080] "Generating means" refers to a device or program that has the function of loading a generating artificial intelligence model into the system.

[0081] A "generative artificial intelligence model" refers to a model that uses deep learning technology to generate test scenarios and analyze test results.

[0082] The term "means for the user to input information about the application to be tested" refers to a device or program that allows the user to input information such as the path, settings, and test scope of the application to be tested to the server.

[0083] A "test scenario" refers to a scenario that includes a series of operation procedures and input value actions that are generated for the application under test.

[0084] "Terminal" refers to a device that performs operations according to the test scenario received from the server and collects the test results.

[0085] "Means for testing various input values ​​to test both normal and abnormal operation" refers to a function for attempting both correct and incorrect input to an application and checking its operation.

[0086] "Means for collecting test results and sending them to a server" refers to a device or program for collecting result data such as logs and screenshots after test execution and sending them to a server.

[0087] "Means for analyzing test results on the server and providing the cause of failure and how to correct it" refers to the function for the server to analyze the test result data received, identify the cause of the error and how to correct it, and provide the information to the user.

[0088] "Means for retesting" refers to the function of generating new test scenarios for the modified application and running the tests again.

[0089] The "means for generating a report" refers to a function for compiling the final test results and generating a document to be provided to the user.

[0090] The present invention is a system that uses a generative artificial intelligence model to automate application testing and reduce the effort and cost. Specific embodiments of this system are described below.

[0091] First, the server loads a generative AI model using a generator. This generative AI model employs deep learning techniques and is pre-trained using libraries such as TensorFlow and PyTorch. This model is then used to generate test scenarios and analyze the results.

[0092] Next, the user enters the application information to be tested into the server from the terminal. This input includes the application path, configuration information, test scope, etc. For example, the user can use a browser to enter information such as " / path / to / application", "configuration information", and "Module A to be tested" into the form and click the submit button.

[0093] When the server receives input information from the user, it uses a generative AI model to generate a test scenario. The generated test scenario includes a variety of UI operations based on different input values ​​and operation patterns. At this time, a prompt such as "Please create a test case for the user registration form" is passed to the generative AI model, and the returned test scenario is obtained.

[0094] The server sends the generated test scenario to the terminal. This scenario is sent in JSON format and is deployed on the terminal. The terminal receives the test scenario and launches the application and performs operations according to the scenario specified in the test environment. For example, to test a user registration form, enter "testuser", "testuser@example.com", and "password123" and click the Register button. Invalid email addresses and short passwords are also entered to test both normal and error behavior.

[0095] The device collects logs of the test results, including success, failure, and error messages, and sends them to the server. The server analyzes the received log data and identifies the cause of the failure. For example, if there is a problem with email address validation, it will detect this and generate advice such as "the validation logic needs to be corrected."

[0096] The user modifies the application source code based on the provided modification information. After the modification is complete, the server generates a new test scenario and sends it to the terminal for re-testing. Through the re-test, it is confirmed whether the modification was properly made, and the results are collected again, analyzed, and a final report is generated. The user receives the final report and can identify areas for improvement or problems.

[0097] This system automates the application testing process, reducing the effort and cost involved, and also improves the quality and efficiency of testing by automatically generating test scenarios using generative AI models and analyzing the results.

[0098] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0099] Step 1:

[0100] The server loads the generative AI model using a generator, initializes a specific library (e.g., TensorFlow or PyTorch), and loads a pre-trained model file. The input is the model file path, and the output is the generative AI model deployed in memory.

[0101] Specific behavior:

[0102] The server reads " / path / to / model / file" and loads the model using the TensorFlow API.

[0103] Step 2:

[0104] The user inputs the application information to be tested from the terminal to the server. This information includes the application path, settings, and test scope. The input is the information entered by the user, and the output is the test information sent to the server.

[0105] Specific behavior:

[0106] The user enters " / path / to / application," "configuration information," and "module A to be tested" into the browser form and clicks the "Submit" button.

[0107] Step 3:

[0108] The server generates test scenarios using a generative AI model based on the received information. The input is the application information and prompts provided by the user, and the output is the generated test scenario.

[0109] Specific behavior:

[0110] The server passes the prompt "Please create a test case for the user registration form" to the generative AI model and obtains the returned test scenario.

[0111] Step 4:

[0112] The server sends the generated test scenario to the terminal. The input is the test scenario, and the output is the scenario sent to the terminal.

[0113] Specific behavior:

[0114] The server saves the generated scenario as " / tests / scenario1.json" and provides a download link to the device.

[0115] Step 5:

[0116] The device executes the application in the test environment according to the received test scenario, with the test scenario as input and the execution result log and screenshots as output.

[0117] Specific behavior:

[0118] The terminal opens the "User Registration Form", enters "testuser", "testuser@example.com", and "password123", and clicks the "Register" button.

[0119] Step 6:

[0120] The terminal sends the collected test results to the server. The input is the execution result, and the output is the log and screenshots sent to the server.

[0121] Specific behavior:

[0122] The device sends files such as " / logs / test1.log" and " / screenshots / test1.png" to the server via an HTTP POST request.

[0123] Step 7:

[0124] The server analyzes the received test results and provides the cause of failure and how to fix it. The input is logs and screenshots, and the output is a report containing fix information.

[0125] Specific behavior:

[0126] The server parses the log file, detects the error message "Email address validation failed" and generates an advice "Fix the validation logic".

[0127] Step 8:

[0128] The user modifies the application based on the modification information from the server. The input is the modification information from the server, and the output is the modified application.

[0129] Specific behavior:

[0130] The user opens a code editor, modifies the "email_validation.py" file, and rebuilds it.

[0131] Step 9:

[0132] The server generates a new test scenario for the modified application and sends it to the terminal to re-execute the test. The input is the modified application information and prompt, and the output is the re-generated test scenario.

[0133] Specific behavior:

[0134] The server generates " / tests / scenario2.json" and sends it to the device, which runs the test again and collects new results.

[0135] Step 10:

[0136] The server collects and analyzes the retest results and generates the final report. The input is the retest results and the output is the final report.

[0137] Specific behavior:

[0138] The server generates " / reports / final_report.pdf" and provides the user with a download link.

[0139] (Application example 1)

[0140] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0141] Conventional product inspection systems rely on manual inspection, which often results in reduced inspection efficiency and accuracy. Furthermore, manual inspection is time-consuming and prone to human error, creating a need for improved quality control. Furthermore, existing automated inspection systems lack flexibility, making it difficult to accommodate a wide variety of products and inspection items. A new system is needed to resolve these issues, streamline product inspection within factories, and improve quality.

[0142] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0143] In this invention, the server includes means for loading a generative artificial intelligence model, means for generating a test scenario using the generative artificial intelligence model, means for transmitting the test scenario to an operation means, means for the operation means to operate an application according to the test scenario, means for collecting and transmitting test results to the server, means for analyzing the test results in the server and providing causes of failure and correction methods, means for retesting the corrected application, means for generating a product inspection scenario and having an in-factory device execute it, means for collecting and analyzing the inspection results and identifying defective products and providing points to be corrected, and means for generating and re-executing a new test scenario for reinspection, thereby enabling improvement in the efficiency and accuracy of product inspection.

[0144] A "generative artificial intelligence model" is a collection of algorithms that use AI techniques such as deep learning to process data for a specific task and make predictions or decisions.

[0145] A "test scenario" is a plan that defines specific operational procedures and input data to verify the operation and functionality of software or a system.

[0146] "Operation means" refers to devices or programs for operating applications or systems according to test scenarios.

[0147] "Test results" are information on operation confirmation such as success, failure, and error messages obtained when a test scenario is executed.

[0148] A "server" is a computer system that provides services such as data processing and storage over a network.

[0149] "Correction methods" are specific countermeasures or procedures for resolving problems or defects discovered during testing.

[0150] "Retesting" is the process of running tests again after a fix has been made to verify that the fix was successful.

[0151] "Factory equipment" refers to machines and robot systems used in factories to perform product inspections and production work.

[0152] "Inspection results" refers to data such as measurement values ​​and appearance check results obtained during product inspection.

[0153] A "defective product" is a product that does not meet standards or is determined to be defective during product inspection.

[0154] "Correction points" are specific locations and means for correcting identified defects or flaws based on the inspection results.

[0155] The present invention provides a system for automating product inspection during the production process, thereby improving efficiency and accuracy. Specific embodiments for carrying out the present invention will be described below.

[0156] The server first loads a generative artificial intelligence model. This generative AI model is based on deep learning technology and is optimized for product inspection tasks. The server also provides an interface for receiving product information and inspection items entered by factory staff. Product information includes model numbers, specifications, and details of inspection items, and generates test scenarios based on this information.

[0157] The generated test scenario is sent to the operating means in the factory. The operating means can be a robot or measuring device for product inspection. Specifically, it can be, for example, a laser tape measure for measuring dimensions or an inspection robot equipped with a high-resolution camera. These devices inspect the products according to the test scenario sent from the server.

[0158] Inspection results are sent from the operation device to the server. The server analyzes the received inspection results and identifies defective products and provides correction points. For example, if dimensions are outside the standard range or if surface scratches are detected in camera images, specific correction points will be presented. This allows factory staff to quickly carry out correction work.

[0159] After the corrections are complete, the server again uses the generative AI model to generate new test scenarios and conduct a retest to confirm whether the corrections were effective. The results of the retest are also sent to the server for final analysis.

[0160] To explain the effectiveness of this system, let's use a concrete example. For example, in the case of product inspection of smartphone cases, the prompt sentence to check the dimensions of the case and the presence of scratches on the surface is as follows:

[0161] "Measure the dimensions of your smartphone case and make sure it meets the standard values ​​(height: 150mm, width: 75mm, depth: 10mm). Also, check for scratches on the surface and report any defects."

[0162] Based on the prompts, the server generates a test scenario, and the operating device executes the test. The test results are collected on the server for detailed analysis. The test results include dimensional measurements and the presence or absence of surface scratches, and if the standards are not met, specific correction points are provided.

[0163] In this way, the system based on the present invention realizes efficient product inspection and improved quality.

[0164] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0165] Step 1:

[0166] The server loads the generative AI model, which is based on deep learning technology and optimized for product inspection tasks. Once the model has been loaded, the server prepares to accept product information and inspection items.

[0167] Input: Deep learning model file

[0168] Output: The loaded generative AI model

[0169] How it works: The server loads the generative AI model from file storage and deploys it in memory. This model includes neural networks for performing tasks such as object detection, dimension measurement, and visual inspection.

[0170] Step 2:

[0171] The user uses a tablet or PC to input product information (model number, specifications, inspection items) into the server, including details of the product to be inspected.

[0172] Input: Product information (model number, specifications, inspection items)

[0173] Output: Product information received by the server

[0174] How it works: The user enters product information into a dedicated interface and sends it to the server, which then prepares the information for input into the generative AI model.

[0175] Step 3:

[0176] The server uses the generative AI model to generate test scenarios, which include specific testing procedures for each product.

[0177] Input:Product Information

[0178] Output: Test scenario

[0179] How it works: The server inputs product information into the generative AI model and generates test scenarios, including procedures for measuring dimensions and inspecting surfaces. For example, "Measurement procedures to confirm a height of 150 mm, width of 75 mm, and depth of 10 mm."

[0180] Step 4:

[0181] The server sends the generated test scenario to the operating means, which is usually a robot or inspection device in a factory.

[0182] Input: Test scenario

[0183] Output: Test scenario sent to the control

[0184] Specific operation: The server transmits the test scenario to the operation means via the network, using a data transfer protocol to ensure reliable communication.

[0185] Step 5:

[0186] The operating means (for example, an inspection robot) performs inspections according to the test scenario, including dimensional measurements and surface inspections.

[0187] Input: Test scenario

[0188] Output: Inspection results (measurement values, abnormality detection information, etc.)

[0189] Specific operation: The operating means uses a laser tape measure and a high-resolution camera to inspect the product based on the test scenario. For example, the robot moves through each part of the product to measure its dimensions, and a high-resolution camera detects surface scratches.

[0190] Step 6:

[0191] The test results are transmitted from the operating means to the server, and the server receives them.

[0192] Input: Test results

[0193] Output: Inspection results aggregated on the server

[0194] Specific operation: The operating means sends the inspection results (e.g., size measurements and the presence or absence of surface abnormalities) to the server. The server stores this data as a log and prepares it for analysis.

[0195] Step 7:

[0196] The server analyzes the inspection results and identifies defects and provides correction points.

[0197] Input: Test results

[0198] Output: Analysis results (defective product identification, correction points)

[0199] Specific operation: The server uses machine learning algorithms to analyze the inspection results and identify defective products. It also generates specific instructions for correction, such as "This product's height exceeds the specified value, so it needs to be reprocessed."

[0200] Step 8:

[0201] The user receives instructions from the server and performs the work to correct the product.

[0202] Input: Correction instructions from the server

[0203] Output: Modified product

[0204] Specific operation: The user checks the correction instructions provided by the server and reprocesses or reinspects the product.

[0205] Step 9:

[0206] For the corrected product, new test scenarios are generated for retesting, and the product is retested.

[0207] Input: Revised product information

[0208] Output: Retest scenario, retest results

[0209] Specific operation: The server generates a new test scenario based on the corrected product information and has the operating means re-execute it. The retest results are sent to the server, and a final pass / fail decision is made.

[0210] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0211] This invention is a system that combines a generative artificial intelligence model and an emotion engine to automate application testing and take user emotions into consideration. Specific embodiments of this system are described below.

[0212] First, the server loads the generative AI model and emotion engine. The generative AI model is used to generate test scenarios, and the emotion engine recognizes user emotions in real time.

[0213] Next, the user provides information about the application under test to the server from their device. This information includes the application path, settings, and test scope. The server uses the received information to generate test scenarios using a generative AI model. For example, it creates test scenarios with various patterns that include input in the correct format and errors.

[0214] The server sends the generated test scenario to the terminal, which receives the scenario, launches the application, and performs the specified operations. As a concrete example, consider testing a user registration form. The terminal enters a username, email address, and password in the correct format and clicks the submit button. At the same time, the emotion engine obtains emotional data from the user's facial expressions and voice and records it in a log. If the emotional data indicates, for example, "dissatisfaction" or "confusion," this is collected as a test result.

[0215] As a result of the test, the device sends a log containing success or failure results, error messages, and emotional data to the server. The server analyzes the received log data and identifies the cause of the failure and how to correct it based on the user's emotional state. For example, it provides information such as "There was a problem with email address validation, which caused a lot of dissatisfaction."

[0216] The application is modified based on the modification information provided by the user. Once the modification is complete, the server generates a new test scenario, sends it to the device, and performs a retest. At this time, the emotion engine is again used to collect and analyze the overall test results, including emotion data. This allows the system to confirm that the modifications have been made appropriately and in a way that satisfies the user.

[0217] The above is a specific embodiment of the system based on the present invention, which combines AI and emotion recognition technology to achieve more user-friendly and effective test automation.

[0218] The processing flow will be explained below.

[0219] Step 1:

[0220] The server loads the generative AI model and emotion engine. Specifically, it expands the program files for the generative AI model and emotion engine into memory, making both models available for use.

[0221] Step 2:

[0222] The user enters information about the application to be tested into the server from the terminal, including the application path (e.g., " / path / to / application"), the test scope, and initial setting items.

[0223] Step 3:

[0224] The server generates test scenarios using a generative AI model based on the application information it receives. For example, for a user registration form, multiple scenarios for valid and invalid inputs are created in JSON format.

[0225] Step 4:

[0226] The server stores the generated test scenarios in a database, ensuring that the scenarios are used consistently in subsequent processes.

[0227] Step 5:

[0228] The server sends the saved test scenario to the device, which passes the test scenario to the device via an API endpoint.

[0229] Step 6:

[0230] The device loads the test scenario received from the server and launches the application. For example, it opens the application's executable file and makes the UI operable.

[0231] Step 7:

[0232] The device automatically executes UI operations according to the test scenario. At the same time, the emotion engine is activated and collects emotional data in real time from the user's facial expressions and voice. For example, if the user shows signs of "confusion" or "dissatisfaction" while entering their username, email address, and password, and then performing the send operation, this data is also recorded.

[0233] Step 8:

[0234] The device collects test results and emotional data, including successful and unsuccessful operations, displayed error messages, and emotional data.

[0235] Step 9:

[0236] The device sends the collected test results and emotion data to the server, which includes compiling the test results and emotion data in JSON format and sending them to the server's receiving API.

[0237] Step 10:

[0238] The server analyzes the test results and emotion data it receives. It uses a generative AI model to identify the specific cause of the failure and extracts the user's perceived issues based on the emotion data. For example, if there is a problem with email address validation, it generates a message saying, "The validation logic needs to be corrected."

[0239] Step 11:

[0240] The server notifies the user of the proposed correction method. Possible notification methods include displaying the information on the dashboard or sending an email. Specifically, it presents in detail the areas that need to be corrected and how to do so.

[0241] Step 12:

[0242] The user modifies the application based on the provided fixes, modifies the code in the development environment, and builds the modified version.

[0243] Step 13:

[0244] The user notifies the server that the modification is complete, for example, by sending the modified application path to the server.

[0245] Step 14:

[0246] The server generates a new test scenario again and sends it to the terminal. The retest runs a test to check whether the corrections have been made properly.

[0247] Step 15:

[0248] The device will then conduct the retest and send the results to the server. The device will also use the emotion engine to collect emotional data during the retest and send the results, including this data, to the server.

[0249] Step 16:

[0250] The server finally analyzes all test results and emotion data and reports back to the user, ensuring that the corrections were made properly and providing the user with a final report.

[0251] The above are the specific processing steps of the system.

[0252] Example 2

[0253] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0254] Traditional application testing is often done manually, requiring a large amount of work and time. Furthermore, there is no system for evaluating test results that takes user emotions into account, making it difficult to identify problems based on user experience. Therefore, there is a need for more efficient, user-centric testing by introducing test automation and a system that reflects user emotions in real time.

[0255] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for loading a generative AI model, means for loading an emotion engine, means for generating a test scenario using the generative AI model, means for transmitting the generated test scenario to a terminal, means for the terminal to run an application according to the test scenario, means for acquiring user emotion data using the emotion engine, means for collecting test results and emotion data and transmitting them to the server, means for analyzing the test results and emotion data in the server and providing causes of failure and correction methods, and means for retesting the corrected application. This enables test automation and comprehensive test evaluation that takes user emotions into consideration.

[0256] "Generative artificial intelligence model" refers to an algorithm or system that uses artificial intelligence to automatically generate test scenarios and other artifacts.

[0257] An "emotion engine" refers to a technology or system that analyzes a user's facial expressions and voice to recognize and record emotions in real time.

[0258] A "test scenario" refers to a plan containing a series of steps or procedures used in performing operational testing of an application.

[0259] "Terminal" refers to a device that receives instructions from a server, executes applications, and obtains operation and test results.

[0260] "Server" refers to the central computer system that loads the generative AI models and emotion engines, generates and transmits test scenarios, and analyzes collected data.

[0261] "User emotion data" refers to data indicating the user's emotional state obtained using an emotion engine.

[0262] "Test results" refers to the result information such as success / failure and error messages when testing an application.

[0263] "Correction method" refers to the specific means or steps to resolve application or system problems based on the analysis results.

[0264] "Retesting" refers to the process of testing an application again after it has been modified to ensure that the modifications function correctly.

[0265] MODE FOR CARRYING OUT THE INVENTION

[0266] This invention is a system that combines a generative artificial intelligence model and an emotion engine to automate application testing and take user emotions into consideration. Specific embodiments of this system are described below.

[0267] Hardware and software used

[0268] Server: The central computer system of the system, which loads the generative AI model and emotion engine, generates and transmits test scenarios, and analyzes the collected data.

[0269] Example of use: Server machines with high performance processors, such as Linux servers.

[0270] Terminal: A device that receives instructions from the server, executes applications, and obtains operation and test results.

[0271] Example of use: A typical personal computer (Windows or macOS) or a virtual machine.

[0272] Generative AI models: Artificial intelligence algorithms that automatically generate test scenarios and other artifacts.

[0273] Usage example: Using "GPT-3" and "BERT".

[0274] Emotion engine: Technology that analyzes the user's facial expressions and voice to recognize and record emotions in real time.

[0275] Usage examples: "Affectiva" and "Microsoft Azure Emotion API".

[0276] Specific examples of processing

[0277] First, the server loads the generative AI model and emotion engine. The generative AI model is used to generate test scenarios, and the emotion engine is responsible for recognizing user emotions in real time.

[0278] Next, the user uses the terminal to provide information about the application under test to the server, including the application path, settings, test scope, etc.

[0279] The server then uses the received information to generate appropriate test scenarios using a generative AI model. For example, it sends a prompt to the AI, such as "Please generate a test scenario for when a user enters an incorrect format in the email address field," and uses the response to create the scenario.

[0280] The server sends the generated test scenario to the terminal, and the terminal launches the application according to the scenario and performs specific operations. For example, run " / path / to / app", enter "testuser" in the username field, "wrong_email_format" in the email address field, and "password123" in the password field, and click the submit button.

[0281] At the same time, the emotion engine analyzes the user's facial expressions and voice to obtain emotional data. For example, if the user makes a "confused" expression, this is recorded as emotional data in the log. The device then saves this in a log file.

[0282] Once the test is complete, the device sends a log containing the test success / failure, error messages, and emotion data to the server. The server analyzes the received log data and identifies the cause of the failure and how to fix it. For example, it may conclude that "email address validation errors caused many users to feel confused."

[0283] The user modifies the application based on the analysis results provided by the server. For example, the user makes specific modifications such as "strengthening the validation function for email addresses."

[0284] Finally, the server generates a new test scenario using the generative AI model again and sends it to the device for retesting. This retest also uses the emotion engine, and collects and analyzes the overall test results.

[0285] This allows for comprehensive testing that takes user emotions into account, resulting in the development of higher quality, more user-friendly applications.

[0286] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0287] Step 1:

[0288] The server loads the generative AI model and emotion engine. First, the server loads the generative AI model and emotion engine from existing data storage into memory and prepares them. GPT-3 or BERT is used as the generative AI model, and Affectiva or Microsoft Azure Emotion API is used as the emotion engine. This enables the server to generate test scenarios and recognize user emotions. The file paths of the generative AI model and emotion engine are required as input, and the loaded model and engine are obtained as output.

[0289] Step 2:

[0290] The user uses a terminal to enter information about the application under test and provides it to the server. The user enters the application path, settings, test scope, etc. into the terminal and clicks the send button. The terminal sends the entered information to the server. The application path, setting file, etc. are required as input, and this information is obtained as output to be sent to the server.

[0291] Step 3:

[0292] The server generates a test scenario using the generative AI model. Based on the application information provided by the user, the server inputs a prompt statement to the generative AI model to generate an appropriate test scenario. For example, the server uses the prompt statement "Please generate a test scenario for when the user enters an incorrect format in the email address field" to obtain the generated scenario. The input of this step is the prompt statement and application information, and the output is the generated test scenario.

[0293] Step 4:

[0294] The server sends the generated test scenario to the terminal. The server packages the generated test scenario and sends it to the terminal. The generated test scenario is required as input, and the scenario sent to the terminal is obtained as output.

[0295] Step 5:

[0296] The terminal launches the application according to the received test scenario and performs the specified operation. The terminal first launches the application from the specified path, and then performs the input operations according to the scenario. Specifically, it enters "testuser" in the username field, "wrong_email_format" in the email address field, and "password123" in the password field, and then clicks the send button. The input for this step is the test scenario and application information, and the output is the result of the executed operation.

[0297] Step 6:

[0298] The emotion engine analyzes the user's facial expressions and voice to obtain emotional data. The device uses the emotion engine to analyze the user's real-time facial expressions and voice to collect emotional data. If the emotional data indicates "confusion" or "dissatisfaction," this is recorded in a log. The user's facial expressions and voice data are required as input, and the collected emotional data is obtained as output.

[0299] Step 7:

[0300] The device sends a log containing the success or failure of the test, error messages, and emotional data to the server. The device saves the executed operation results and emotional data together in a log file and sends the log to the server. The operation results and emotional data are required as input, and the log data sent to the server is obtained as output.

[0301] Step 8:

[0302] The server analyzes the received log data to identify the cause of the failure and how to fix it. The server analyzes the log data to identify the problem and its cause. For example, it may conclude that "email address validation errors caused frequent user confusion." The input to this step is the log data, and the output is the analysis results and how to fix it.

[0303] Step 9:

[0304] The user modifies the application based on the analysis results provided by the server. The user then modifies the application code according to the instructions provided by the server. For example, the user makes a specific modification such as "strengthening the email address validation function." The input requires the analysis results and instructions, and the output is the modified application.

[0305] Step 10:

[0306] The server then uses the generative AI model to generate a new test scenario, sends it to the device, and runs the test again. All steps are repeated to verify that the modifications function properly. The inputs are the modified application information and a new prompt, and the output is the retest result.

[0307] (Application example 2)

[0308] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0309] Traditional application testing systems focused on test automation but failed to consider user emotions and usage. As a result, while testing could detect technical defects, it was unable to adequately address user experience improvements. Furthermore, the inability to reflect user confusion and dissatisfaction in real time limited UI / UX optimization.

[0310] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for loading a generative AI model, a means for generating a test scenario, and a means for analyzing test results and emotion data. This makes it possible to collect and analyze user emotion data in real time and test and improve applications taking user experience into consideration.

[0311] A "generative artificial intelligence model" is a type of artificial intelligence used to generate test scenarios, and has the ability to automatically create appropriate scenarios based on user input and operations.

[0312] A "test scenario" refers to a set of pre-defined operational procedures or conditions for verifying the application's functions and user interface.

[0313] "Device" refers to an electronic device such as a computer, smartphone, or tablet on which the application being tested runs.

[0314] "User emotion data" refers to emotion information obtained in real time from the user's facial expressions and voice, and includes emotions such as "satisfaction," "dissatisfaction," and "confusion."

[0315] "Log" refers to data that records the application's operating status, user operation history, emotional data, etc.

[0316] "Server" refers to a centralized computer system that generates test scenarios, analyzes the results, collects and stores emotional data, and so on.

[0317] The "cause of failure" refers to the cause of an application not operating normally.

[0318] "Correction method" refers to the specific methods and procedures for solving the discovered cause of failure.

[0319] "Retesting" refers to the process of verifying the modified application again according to the test scenarios to confirm that the improvements have been properly implemented.

[0320] This invention is a system for automating application testing and taking user emotions into account. Specifically, it uses a system that combines a generative artificial intelligence model and an emotion engine.

[0321] First, the server loads a generative AI model. This model generates test scenarios based on user input and operation history. For example, it generates a test scenario for the process of searching for a specific product and adding it to a cart.

[0322] The server then loads the emotion engine, which has the ability to acquire emotion data in real time from the user's facial expressions and voice. The data is collected from devices such as smartphones and tablets.

[0323] When a user provides information about the application under test to the server from their device, the server uses a generative artificial intelligence model to generate a test scenario and sends it to the device. For example, a scenario in which a user searches for a product in a virtual store and adds it to a cart is considered.

[0324] The device operates the application according to the test scenario sent from the server. During the test, the emotion engine collects emotional data in real time from the user's facial expressions and voice. The device records these test results and emotional data in a log and sends them to the server.

[0325] The server analyzes the received log data to identify the cause of the application failure and how to fix it. For example, if the user expresses "confusion" or "dissatisfaction," the server will suggest improvements to the UI / UX based on that emotion data.

[0326] The modified application is then retested, again using the emotion engine to verify whether user satisfaction has improved. To ensure the modified application behaves as expected, retest scenarios are also generated using the generative artificial intelligence model.

[0327] This system allows for comprehensive application testing that takes into account user emotions, rather than simply fixing technical bugs, and allows for application improvements that satisfy users.

[0328] Examples:

[0329] For example, if a user expresses "confusion" in the process of searching for a specific product in a virtual store app and adding it to the cart, a log of this emotion can be collected and suggestions for improving the UI / UX can be generated.An example of a prompt to input to the generative AI model could be, "Generate a test scenario for the virtual store application where the user searches for a specific product and adds it to the shopping cart. Include various steps and possible user inputs, including both successful and erroneous entries."

[0330] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0331] Step 1:

[0332] The server loads the generative AI model. The input is the path of the generative AI model, and the output is the loaded model object. This model object is used to prepare for generating test scenarios.

[0333] Step 2:

[0334] The server loads the emotion engine. The input is the path to the emotion engine, and the output is the loaded engine object. This engine object is used to prepare for collecting user emotion data in real time. Specifically, the parameters of the emotion recognition model are set.

[0335] Step 3:

[0336] The user provides information about the application under test to the server from their device. The input is the information about the application under test (application path, settings, test scope), and the output is the information object received by the server. The server uses this to input prompts to the generative AI model.

[0337] Step 4:

[0338] The server generates test scenarios using a generative AI model. The input is the application information sent by the user, and the output is the generated test scenario. The generative AI model outputs a specific test scenario using a prompt statement.

[0339] Step 5:

[0340] The server sends the generated test scenario to the terminal. The input is the generated test scenario, and the output is confirmation data of the transmission result. The server verifies whether the test scenario was sent correctly to the terminal.

[0341] Step 6:

[0342] The terminal operates the application according to the test scenario. The input is the test scenario received from the server, and the output is the test result. Specifically, the terminal automatically executes the specified actions and records the operation log.

[0343] Step 7:

[0344] The device collects the user's emotional data in real time. The input is real-time data from the camera and microphone, and the output is emotional data. The device inputs this data into the emotion engine and obtains the emotion recognition results.

[0345] Step 8:

[0346] The device sends a log containing the test results and emotional data to the server. The input is the log containing the test results and emotional data, and the output is a transmission confirmation data sent to the server. The server performs analysis based on this.

[0347] Step 9:

[0348] The server analyzes the received log data and identifies the cause of the application failure and how to fix it. The input is the log data sent from the device, and the output is the analysis results including how to fix it. This makes it clear what specific points need to be fixed.

[0349] Step 10:

[0350] The server generates a new test scenario for retesting the modified application and sends it to the terminal. The input is the modified application information, and the output is the new test scenario. The same process is repeated to check the user's satisfaction.

[0351] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0352] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0353] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0354] [Second embodiment]

[0355] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0356] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0357] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0358] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0359] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0360] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0361] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0362] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0363] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0364] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0365] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0366] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0367] The present invention is a system that uses a generative artificial intelligence model to automate application testing and reduce the effort and cost. Specific embodiments of this system are described below.

[0368] First, the server loads the generative AI model, which uses deep learning techniques and is used to generate test scenarios and analyze the results.

[0369] Next, the user inputs information about the application under test into the server from their device, including the application path, settings, and test scope. Once the server receives the information from the user, it uses a generative AI model to generate a test scenario. This scenario includes, for example, various UI operations based on different input values ​​and operation patterns.

[0370] The server sends the generated test scenario to the terminal. The terminal receives the test scenario, launches the application, and performs operations according to the specified scenario. As a concrete example, consider testing a user registration form. The terminal enters a username, email address, and password in the correct format and clicks the registration button. This also includes entering an invalid email address or a short password. This covers both normal and error behavior.

[0371] As a result of the test, the device collects logs of successes, failures, error messages, etc. and sends them to the server. The server analyzes the received log data and identifies the cause of the failure. For example, if there is a problem with the validation of an email address, it will detect this and generate advice such as "the validation logic needs to be corrected."

[0372] The application is modified based on the modification information provided by the user. Once the modification is complete, the server generates a new test scenario, sends it to the device, and performs a retest. Through the retest, it is confirmed whether the modification was performed properly, and the results are collected and analyzed again before a final report is made.

[0373] The above is a specific embodiment of the system based on the present invention. This system utilizes AI to automatically perform a variety of tests, thereby improving the efficiency and quality of tests.

[0374] The processing flow will be explained below.

[0375] Step 1:

[0376] The server loads the generative AI model. Specifically, it extracts the file of the generative AI model into memory and makes the model ready for use.

[0377] Step 2:

[0378] The user enters information about the application to be tested into the server from the terminal. Specifically, the user enters the application path (e.g., " / path / to / application") and initial settings (e.g., test scope and input item definitions), and sends them to the server.

[0379] Step 3:

[0380] The server generates test scenarios using a generative AI model based on the application information it receives. For example, it creates test scenarios for valid input (username, email address, password in the correct format) and invalid input (invalid email address, password that is too short).

[0381] Step 4:

[0382] The server saves the generated test scenario in JSON format, ensuring that the scenario is used consistently in subsequent processing.

[0383] Step 5:

[0384] The server sends the saved test scenario to the device, and passes the scenario to the device's API endpoint.

[0385] Step 6:

[0386] The terminal loads the test scenario received from the server and launches the application. Specifically, it opens the application's executable file and makes the GUI operable.

[0387] Step 7:

[0388] The device automatically executes UI operations according to the test scenario, such as entering a username, email address, and password, and clicking the send button.

[0389] Step 8:

[0390] The device collects the results of the test, including successful and unsuccessful operations and any error messages displayed.

[0391] Step 9:

[0392] The test results collected by the device are sent to the server. Specifically, the test results are compiled in JSON format and sent to the server's receiving API.

[0393] Step 10:

[0394] The server analyzes the test results it receives and uses a generative AI model to identify the specific cause of the failure and suggest a fix based on that. For example, it generates a message such as "Email address validation logic is required."

[0395] Step 11:

[0396] The server notifies the user of the proposed correction method. Possible notification methods include displaying the result on the dashboard or sending an email.

[0397] Step 12:

[0398] The user modifies the application based on the provided fixes, specifically by modifying the code in the development environment and building the modified version.

[0399] Step 13:

[0400] The user notifies the server that the modification is complete, for example, by sending the modified application path to the server.

[0401] Step 14:

[0402] The server then generates a new test scenario and sends it to the device, which then runs a retest to verify whether the modifications were made properly.

[0403] Step 15:

[0404] The retest results are collected, analyzed, and finally reported to the user. After confirming that the problems discovered in the previous process have been resolved, a final report is made.

[0405] The above are the specific processing steps of the system.

[0406] Example 1

[0407] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0408] The application testing process is not automated enough, resulting in labor-intensive and costly tasks. Creating test scenarios and analyzing test results takes a lot of time, reducing overall testing efficiency. Furthermore, human error and insufficient test coverage make it difficult to improve application quality.

[0409] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0410] In this invention, the server includes means for loading a generative AI model using a generating means, means for a user to input application information to be tested, means for generating a test scenario using the generative AI model, means for transmitting the test scenario to a terminal, means for the terminal to launch and operate the application according to the test scenario, means for testing various input values ​​to test both normal and abnormal operation, means for collecting test results and transmitting them to the server, means for the server to analyze the test results and provide causes of failure and correction methods, means for retesting the corrected application based on the correction information, and means for reporting and analyzing the final results through the retest and generating a report. This makes it possible to automate the application testing process, reduce effort and cost, and improve overall testing efficiency.

[0411] "Generating means" refers to a device or program that has the function of loading a generating artificial intelligence model into the system.

[0412] A "generative artificial intelligence model" refers to a model that uses deep learning technology to generate test scenarios and analyze test results.

[0413] The term "means for the user to input information about the application to be tested" refers to a device or program that allows the user to input information such as the path, settings, and test scope of the application to be tested to the server.

[0414] A "test scenario" refers to a scenario that includes a series of operation procedures and input value actions that are generated for the application under test.

[0415] "Terminal" refers to a device that performs operations according to the test scenario received from the server and collects the test results.

[0416] "Means for testing various input values ​​to test both normal and abnormal operation" refers to a function for attempting both correct and incorrect input to an application and checking its operation.

[0417] "Means for collecting test results and sending them to a server" refers to a device or program for collecting result data such as logs and screenshots after test execution and sending them to a server.

[0418] "Means for analyzing test results on the server and providing the cause of failure and how to correct it" refers to the function for the server to analyze the test result data received, identify the cause of the error and how to correct it, and provide the information to the user.

[0419] "Means for retesting" refers to the function of generating new test scenarios for the modified application and running the tests again.

[0420] The "means for generating a report" refers to a function for compiling the final test results and generating a document to be provided to the user.

[0421] The present invention is a system that uses a generative artificial intelligence model to automate application testing and reduce the effort and cost. Specific embodiments of this system are described below.

[0422] First, the server loads a generative AI model using a generator. This generative AI model employs deep learning techniques and is pre-trained using libraries such as TensorFlow and PyTorch. This model is then used to generate test scenarios and analyze the results.

[0423] Next, the user enters the application information to be tested into the server from the terminal. This input includes the application path, configuration information, test scope, etc. For example, the user can use a browser to enter information such as " / path / to / application", "configuration information", and "Module A to be tested" into the form and click the submit button.

[0424] When the server receives input information from the user, it uses a generative AI model to generate a test scenario. The generated test scenario includes a variety of UI operations based on different input values ​​and operation patterns. At this time, a prompt such as "Please create a test case for the user registration form" is passed to the generative AI model, and the returned test scenario is obtained.

[0425] The server sends the generated test scenario to the terminal. This scenario is sent in JSON format and is deployed on the terminal. The terminal receives the test scenario and launches the application and performs operations according to the scenario specified in the test environment. For example, to test a user registration form, enter "testuser", "testuser@example.com", and "password123" and click the Register button. Invalid email addresses and short passwords are also entered to test both normal and error behavior.

[0426] The device collects logs of the test results, including success, failure, and error messages, and sends them to the server. The server analyzes the received log data and identifies the cause of the failure. For example, if there is a problem with email address validation, it will detect this and generate advice such as "the validation logic needs to be corrected."

[0427] The user modifies the application source code based on the provided modification information. After the modification is complete, the server generates a new test scenario and sends it to the terminal for re-testing. Through the re-test, it is confirmed whether the modification was properly made, and the results are collected again, analyzed, and a final report is generated. The user receives the final report and can identify areas for improvement or problems.

[0428] This system automates the application testing process, reducing the effort and cost involved, and also improves the quality and efficiency of testing by automatically generating test scenarios using generative AI models and analyzing the results.

[0429] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0430] Step 1:

[0431] The server loads the generative AI model using a generator, initializes a specific library (e.g., TensorFlow or PyTorch), and loads a pre-trained model file. The input is the model file path, and the output is the generative AI model deployed in memory.

[0432] Specific behavior:

[0433] The server reads " / path / to / model / file" and loads the model using the TensorFlow API.

[0434] Step 2:

[0435] The user inputs the application information to be tested from the terminal to the server. This information includes the application path, settings, and test scope. The input is the information entered by the user, and the output is the test information sent to the server.

[0436] Specific behavior:

[0437] The user enters " / path / to / application," "configuration information," and "module A to be tested" into the browser form and clicks the "Submit" button.

[0438] Step 3:

[0439] The server generates test scenarios using a generative AI model based on the received information. The input is the application information and prompts provided by the user, and the output is the generated test scenario.

[0440] Specific behavior:

[0441] The server passes the prompt "Please create a test case for the user registration form" to the generative AI model and obtains the returned test scenario.

[0442] Step 4:

[0443] The server sends the generated test scenario to the terminal. The input is the test scenario, and the output is the scenario sent to the terminal.

[0444] Specific behavior:

[0445] The server saves the generated scenario as " / tests / scenario1.json" and provides a download link to the device.

[0446] Step 5:

[0447] The device executes the application in the test environment according to the received test scenario, with the test scenario as input and the execution result log and screenshots as output.

[0448] Specific behavior:

[0449] The terminal opens the "User Registration Form", enters "testuser", "testuser@example.com", and "password123", and clicks the "Register" button.

[0450] Step 6:

[0451] The terminal sends the collected test results to the server. The input is the execution result, and the output is the log and screenshots sent to the server.

[0452] Specific behavior:

[0453] The device sends files such as " / logs / test1.log" and " / screenshots / test1.png" to the server via an HTTP POST request.

[0454] Step 7:

[0455] The server analyzes the received test results and provides the cause of failure and how to fix it. The input is logs and screenshots, and the output is a report containing fix information.

[0456] Specific behavior:

[0457] The server parses the log file, detects the error message "Email address validation failed" and generates an advice "Fix the validation logic".

[0458] Step 8:

[0459] The user modifies the application based on the modification information from the server. The input is the modification information from the server, and the output is the modified application.

[0460] Specific behavior:

[0461] The user opens a code editor, modifies the "email_validation.py" file, and rebuilds it.

[0462] Step 9:

[0463] The server generates a new test scenario for the modified application and sends it to the terminal to re-execute the test. The input is the modified application information and prompt, and the output is the re-generated test scenario.

[0464] Specific behavior:

[0465] The server generates " / tests / scenario2.json" and sends it to the device, which runs the test again and collects new results.

[0466] Step 10:

[0467] The server collects and analyzes the retest results and generates the final report. The input is the retest results and the output is the final report.

[0468] Specific behavior:

[0469] The server generates " / reports / final_report.pdf" and provides the user with a download link.

[0470] (Application example 1)

[0471] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0472] Conventional product inspection systems rely on manual inspection, which often results in reduced inspection efficiency and accuracy. Furthermore, manual inspection is time-consuming and prone to human error, creating a need for improved quality control. Furthermore, existing automated inspection systems lack flexibility, making it difficult to accommodate a wide variety of products and inspection items. A new system is needed to resolve these issues, streamline product inspection within factories, and improve quality.

[0473] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0474] In this invention, the server includes means for loading a generative artificial intelligence model, means for generating a test scenario using the generative artificial intelligence model, means for transmitting the test scenario to an operation means, means for the operation means to operate an application according to the test scenario, means for collecting and transmitting test results to the server, means for analyzing the test results in the server and providing causes of failure and correction methods, means for retesting the corrected application, means for generating a product inspection scenario and having an in-factory device execute it, means for collecting and analyzing the inspection results and identifying defective products and providing points to be corrected, and means for generating and re-executing a new test scenario for reinspection, thereby enabling improvement in the efficiency and accuracy of product inspection.

[0475] A "generative artificial intelligence model" is a collection of algorithms that use AI techniques such as deep learning to process data for a specific task and make predictions or decisions.

[0476] A "test scenario" is a plan that defines specific operational procedures and input data to verify the operation and functionality of software or a system.

[0477] "Operation means" refers to devices or programs for operating applications or systems according to test scenarios.

[0478] "Test results" are information on operation confirmation such as success, failure, and error messages obtained when a test scenario is executed.

[0479] A "server" is a computer system that provides services such as data processing and storage over a network.

[0480] "Correction methods" are specific countermeasures or procedures for resolving problems or defects discovered during testing.

[0481] "Retesting" is the process of running tests again after a fix has been made to verify that the fix was successful.

[0482] "Factory equipment" refers to machines and robot systems used in factories to perform product inspections and production work.

[0483] "Inspection results" refers to data such as measurement values ​​and appearance check results obtained during product inspection.

[0484] A "defective product" is a product that does not meet standards or is determined to be defective during product inspection.

[0485] "Correction points" are specific locations and means for correcting identified defects or flaws based on the inspection results.

[0486] The present invention provides a system for automating product inspection during the production process, thereby improving efficiency and accuracy. Specific embodiments for carrying out the present invention will be described below.

[0487] The server first loads a generative artificial intelligence model. This generative AI model is based on deep learning technology and is optimized for product inspection tasks. The server also provides an interface for receiving product information and inspection items entered by factory staff. Product information includes model numbers, specifications, and details of inspection items, and generates test scenarios based on this information.

[0488] The generated test scenario is sent to the operating means in the factory. The operating means can be a robot or measuring device for product inspection. Specifically, it can be, for example, a laser tape measure for measuring dimensions or an inspection robot equipped with a high-resolution camera. These devices inspect the products according to the test scenario sent from the server.

[0489] Inspection results are sent from the operation device to the server. The server analyzes the received inspection results and identifies defective products and provides correction points. For example, if dimensions are outside the standard range or if surface scratches are detected in camera images, specific correction points will be presented. This allows factory staff to quickly carry out correction work.

[0490] After the corrections are complete, the server again uses the generative AI model to generate new test scenarios and conduct a retest to confirm whether the corrections were effective. The results of the retest are also sent to the server for final analysis.

[0491] To explain the effectiveness of this system, let's use a concrete example. For example, in the case of product inspection of smartphone cases, the prompt sentence to check the dimensions of the case and the presence of scratches on the surface is as follows:

[0492] "Measure the dimensions of your smartphone case and make sure it meets the standard values ​​(height: 150mm, width: 75mm, depth: 10mm). Also, check for scratches on the surface and report any defects."

[0493] Based on the prompts, the server generates a test scenario, and the operating device executes the test. The test results are collected on the server for detailed analysis. The test results include dimensional measurements and the presence or absence of surface scratches, and if the standards are not met, specific correction points are provided.

[0494] In this way, the system based on the present invention realizes efficient product inspection and improved quality.

[0495] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0496] Step 1:

[0497] The server loads the generative AI model, which is based on deep learning technology and optimized for product inspection tasks. Once the model has been loaded, the server prepares to accept product information and inspection items.

[0498] Input: Deep learning model file

[0499] Output: The loaded generative AI model

[0500] How it works: The server loads the generative AI model from file storage and deploys it in memory. This model includes neural networks for performing tasks such as object detection, dimension measurement, and visual inspection.

[0501] Step 2:

[0502] The user uses a tablet or PC to input product information (model number, specifications, inspection items) into the server, including details of the product to be inspected.

[0503] Input: Product information (model number, specifications, inspection items)

[0504] Output: Product information received by the server

[0505] How it works: The user enters product information into a dedicated interface and sends it to the server, which then prepares the information for input into the generative AI model.

[0506] Step 3:

[0507] The server uses the generative AI model to generate test scenarios, which include specific testing procedures for each product.

[0508] Input:Product Information

[0509] Output: Test scenario

[0510] How it works: The server inputs product information into the generative AI model and generates test scenarios, including procedures for measuring dimensions and inspecting surfaces. For example, "Measurement procedures to confirm a height of 150 mm, width of 75 mm, and depth of 10 mm."

[0511] Step 4:

[0512] The server sends the generated test scenario to the operating means, which is usually a robot or inspection device in a factory.

[0513] Input: Test scenario

[0514] Output: Test scenario sent to the control

[0515] Specific operation: The server transmits the test scenario to the operation means via the network, using a data transfer protocol to ensure reliable communication.

[0516] Step 5:

[0517] The operating means (for example, an inspection robot) performs inspections according to the test scenario, including dimensional measurements and surface inspections.

[0518] Input: Test scenario

[0519] Output: Inspection results (measurement values, abnormality detection information, etc.)

[0520] Specific operation: The operating means uses a laser tape measure and a high-resolution camera to inspect the product based on the test scenario. For example, the robot moves through each part of the product to measure its dimensions, and a high-resolution camera detects surface scratches.

[0521] Step 6:

[0522] The test results are transmitted from the operating means to the server, and the server receives them.

[0523] Input: Test results

[0524] Output: Inspection results aggregated on the server

[0525] Specific operation: The operating means sends the inspection results (e.g., size measurements and the presence or absence of surface abnormalities) to the server. The server stores this data as a log and prepares it for analysis.

[0526] Step 7:

[0527] The server analyzes the inspection results and identifies defects and provides correction points.

[0528] Input: Test results

[0529] Output: Analysis results (defective product identification, correction points)

[0530] Specific operation: The server uses machine learning algorithms to analyze the inspection results and identify defective products. It also generates specific instructions for correction, such as "This product's height exceeds the specified value, so it needs to be reprocessed."

[0531] Step 8:

[0532] The user receives instructions from the server and performs the work to correct the product.

[0533] Input: Correction instructions from the server

[0534] Output: Modified product

[0535] Specific operation: The user checks the correction instructions provided by the server and reprocesses or reinspects the product.

[0536] Step 9:

[0537] For the corrected product, new test scenarios are generated for retesting, and the product is retested.

[0538] Input: Revised product information

[0539] Output: Retest scenario, retest results

[0540] Specific operation: The server generates a new test scenario based on the corrected product information and has the operating means re-execute it. The retest results are sent to the server, and a final pass / fail decision is made.

[0541] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0542] This invention is a system that combines a generative artificial intelligence model and an emotion engine to automate application testing and take user emotions into consideration. Specific embodiments of this system are described below.

[0543] First, the server loads the generative AI model and emotion engine. The generative AI model is used to generate test scenarios, and the emotion engine recognizes user emotions in real time.

[0544] Next, the user provides information about the application under test to the server from their device. This information includes the application path, settings, and test scope. The server uses the received information to generate test scenarios using a generative AI model. For example, it creates test scenarios with various patterns that include input in the correct format and errors.

[0545] The server sends the generated test scenario to the terminal, which receives the scenario, launches the application, and performs the specified operations. As a concrete example, consider testing a user registration form. The terminal enters a username, email address, and password in the correct format and clicks the submit button. At the same time, the emotion engine obtains emotional data from the user's facial expressions and voice and records it in a log. If the emotional data indicates, for example, "dissatisfaction" or "confusion," this is collected as a test result.

[0546] As a result of the test, the device sends a log containing success or failure results, error messages, and emotional data to the server. The server analyzes the received log data and identifies the cause of the failure and how to correct it based on the user's emotional state. For example, it provides information such as "There was a problem with email address validation, which caused a lot of dissatisfaction."

[0547] The application is modified based on the modification information provided by the user. Once the modification is complete, the server generates a new test scenario, sends it to the device, and performs a retest. At this time, the emotion engine is again used to collect and analyze the overall test results, including emotion data. This allows the system to confirm that the modifications have been made appropriately and in a way that satisfies the user.

[0548] The above is a specific embodiment of the system based on the present invention, which combines AI and emotion recognition technology to achieve more user-friendly and effective test automation.

[0549] The processing flow will be explained below.

[0550] Step 1:

[0551] The server loads the generative AI model and emotion engine. Specifically, it expands the program files for the generative AI model and emotion engine into memory, making both models available for use.

[0552] Step 2:

[0553] The user enters information about the application to be tested into the server from the terminal, including the application path (e.g., " / path / to / application"), the test scope, and initial setting items.

[0554] Step 3:

[0555] The server generates test scenarios using a generative AI model based on the application information it receives. For example, for a user registration form, multiple scenarios for valid and invalid inputs are created in JSON format.

[0556] Step 4:

[0557] The server stores the generated test scenarios in a database, ensuring that the scenarios are used consistently in subsequent processes.

[0558] Step 5:

[0559] The server sends the saved test scenario to the device, which passes the test scenario to the device via an API endpoint.

[0560] Step 6:

[0561] The device loads the test scenario received from the server and launches the application. For example, it opens the application's executable file and makes the UI operable.

[0562] Step 7:

[0563] The device automatically executes UI operations according to the test scenario. At the same time, the emotion engine is activated and collects emotional data in real time from the user's facial expressions and voice. For example, if the user shows signs of "confusion" or "dissatisfaction" while entering their username, email address, and password, and then performing the send operation, this data is also recorded.

[0564] Step 8:

[0565] The device collects test results and emotional data, including successful and unsuccessful operations, displayed error messages, and emotional data.

[0566] Step 9:

[0567] The device sends the collected test results and emotion data to the server, which includes compiling the test results and emotion data in JSON format and sending them to the server's receiving API.

[0568] Step 10:

[0569] The server analyzes the test results and emotion data it receives. It uses a generative AI model to identify the specific cause of the failure and extracts the user's perceived issues based on the emotion data. For example, if there is a problem with email address validation, it generates a message saying, "The validation logic needs to be corrected."

[0570] Step 11:

[0571] The server notifies the user of the proposed correction method. Possible notification methods include displaying the information on the dashboard or sending an email. Specifically, it presents in detail the areas that need to be corrected and how to do so.

[0572] Step 12:

[0573] The user modifies the application based on the provided fixes, modifies the code in the development environment, and builds the modified version.

[0574] Step 13:

[0575] The user notifies the server that the modification is complete, for example, by sending the modified application path to the server.

[0576] Step 14:

[0577] The server generates a new test scenario again and sends it to the terminal. The retest runs a test to check whether the corrections have been made properly.

[0578] Step 15:

[0579] The device will then conduct the retest and send the results to the server. The device will also use the emotion engine to collect emotional data during the retest and send the results, including this data, to the server.

[0580] Step 16:

[0581] The server finally analyzes all test results and emotion data and reports back to the user, ensuring that the corrections were made properly and providing the user with a final report.

[0582] The above are the specific processing steps of the system.

[0583] Example 2

[0584] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0585] Traditional application testing is often done manually, requiring a large amount of work and time. Furthermore, there is no system for evaluating test results that takes user emotions into account, making it difficult to identify problems based on user experience. Therefore, there is a need for more efficient, user-centric testing by introducing test automation and a system that reflects user emotions in real time.

[0586] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for loading a generative AI model, means for loading an emotion engine, means for generating a test scenario using the generative AI model, means for transmitting the generated test scenario to a terminal, means for the terminal to run an application according to the test scenario, means for acquiring user emotion data using the emotion engine, means for collecting test results and emotion data and transmitting them to the server, means for analyzing the test results and emotion data in the server and providing causes of failure and correction methods, and means for retesting the corrected application. This enables test automation and comprehensive test evaluation that takes user emotions into consideration.

[0587] "Generative artificial intelligence model" refers to an algorithm or system that uses artificial intelligence to automatically generate test scenarios and other artifacts.

[0588] An "emotion engine" refers to a technology or system that analyzes a user's facial expressions and voice to recognize and record emotions in real time.

[0589] A "test scenario" refers to a plan containing a series of steps or procedures used in performing operational testing of an application.

[0590] "Terminal" refers to a device that receives instructions from a server, executes applications, and obtains operation and test results.

[0591] "Server" refers to the central computer system that loads the generative AI models and emotion engines, generates and transmits test scenarios, and analyzes collected data.

[0592] "User emotion data" refers to data indicating the user's emotional state obtained using an emotion engine.

[0593] "Test results" refers to the result information such as success / failure and error messages when testing an application.

[0594] "Correction method" refers to the specific means or steps to resolve application or system problems based on the analysis results.

[0595] "Retesting" refers to the process of testing an application again after it has been modified to ensure that the modifications function correctly.

[0596] MODE FOR CARRYING OUT THE INVENTION

[0597] This invention is a system that combines a generative artificial intelligence model and an emotion engine to automate application testing and take user emotions into consideration. Specific embodiments of this system are described below.

[0598] Hardware and software used

[0599] Server: The central computer system of the system, which loads the generative AI model and emotion engine, generates and transmits test scenarios, and analyzes the collected data.

[0600] Example of use: Server machines with high performance processors, such as Linux servers.

[0601] Terminal: A device that receives instructions from the server, executes applications, and obtains operation and test results.

[0602] Example of use: A typical personal computer (Windows or macOS) or a virtual machine.

[0603] Generative AI models: Artificial intelligence algorithms that automatically generate test scenarios and other artifacts.

[0604] Usage example: Using "GPT-3" and "BERT".

[0605] Emotion engine: Technology that analyzes the user's facial expressions and voice to recognize and record emotions in real time.

[0606] Usage examples: "Affectiva" and "Microsoft Azure Emotion API".

[0607] Specific examples of processing

[0608] First, the server loads the generative AI model and emotion engine. The generative AI model is used to generate test scenarios, and the emotion engine is responsible for recognizing user emotions in real time.

[0609] Next, the user uses the terminal to provide information about the application under test to the server, including the application path, settings, test scope, etc.

[0610] The server then uses the received information to generate appropriate test scenarios using a generative AI model. For example, it sends a prompt to the AI, such as "Please generate a test scenario for when a user enters an incorrect format in the email address field," and uses the response to create the scenario.

[0611] The server sends the generated test scenario to the terminal, and the terminal launches the application according to the scenario and performs specific operations. For example, run " / path / to / app", enter "testuser" in the username field, "wrong_email_format" in the email address field, and "password123" in the password field, and click the submit button.

[0612] At the same time, the emotion engine analyzes the user's facial expressions and voice to obtain emotional data. For example, if the user makes a "confused" expression, this is recorded as emotional data in the log. The device then saves this in a log file.

[0613] Once the test is complete, the device sends a log containing the test success / failure, error messages, and emotion data to the server. The server analyzes the received log data and identifies the cause of the failure and how to fix it. For example, it may conclude that "email address validation errors caused many users to feel confused."

[0614] The user modifies the application based on the analysis results provided by the server. For example, the user makes specific modifications such as "strengthening the validation function for email addresses."

[0615] Finally, the server generates a new test scenario using the generative AI model again and sends it to the device for retesting. This retest also uses the emotion engine, and collects and analyzes the overall test results.

[0616] This allows for comprehensive testing that takes user emotions into account, resulting in the development of higher quality, more user-friendly applications.

[0617] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0618] Step 1:

[0619] The server loads the generative AI model and emotion engine. First, the server loads the generative AI model and emotion engine from existing data storage into memory and prepares them. GPT-3 or BERT is used as the generative AI model, and Affectiva or Microsoft Azure Emotion API is used as the emotion engine. This enables the server to generate test scenarios and recognize user emotions. The file paths of the generative AI model and emotion engine are required as input, and the loaded model and engine are obtained as output.

[0620] Step 2:

[0621] The user uses a terminal to enter information about the application under test and provides it to the server. The user enters the application path, settings, test scope, etc. into the terminal and clicks the send button. The terminal sends the entered information to the server. The application path, setting file, etc. are required as input, and this information is obtained as output to be sent to the server.

[0622] Step 3:

[0623] The server generates a test scenario using the generative AI model. Based on the application information provided by the user, the server inputs a prompt statement to the generative AI model to generate an appropriate test scenario. For example, the server uses the prompt statement "Please generate a test scenario for when the user enters an incorrect format in the email address field" to obtain the generated scenario. The input of this step is the prompt statement and application information, and the output is the generated test scenario.

[0624] Step 4:

[0625] The server sends the generated test scenario to the terminal. The server packages the generated test scenario and sends it to the terminal. The generated test scenario is required as input, and the scenario sent to the terminal is obtained as output.

[0626] Step 5:

[0627] The terminal launches the application according to the received test scenario and performs the specified operation. The terminal first launches the application from the specified path, and then performs the input operations according to the scenario. Specifically, it enters "testuser" in the username field, "wrong_email_format" in the email address field, and "password123" in the password field, and then clicks the send button. The input for this step is the test scenario and application information, and the output is the result of the executed operation.

[0628] Step 6:

[0629] The emotion engine analyzes the user's facial expressions and voice to obtain emotional data. The device uses the emotion engine to analyze the user's real-time facial expressions and voice to collect emotional data. If the emotional data indicates "confusion" or "dissatisfaction," this is recorded in a log. The user's facial expressions and voice data are required as input, and the collected emotional data is obtained as output.

[0630] Step 7:

[0631] The device sends a log containing the success or failure of the test, error messages, and emotional data to the server. The device saves the executed operation results and emotional data together in a log file and sends the log to the server. The operation results and emotional data are required as input, and the log data sent to the server is obtained as output.

[0632] Step 8:

[0633] The server analyzes the received log data to identify the cause of the failure and how to fix it. The server analyzes the log data to identify the problem and its cause. For example, it may conclude that "email address validation errors caused frequent user confusion." The input to this step is the log data, and the output is the analysis results and how to fix it.

[0634] Step 9:

[0635] The user modifies the application based on the analysis results provided by the server. The user then modifies the application code according to the instructions provided by the server. For example, the user makes a specific modification such as "strengthening the email address validation function." The input requires the analysis results and instructions, and the output is the modified application.

[0636] Step 10:

[0637] The server then uses the generative AI model to generate a new test scenario, sends it to the device, and runs the test again. All steps are repeated to verify that the modifications function properly. The inputs are the modified application information and a new prompt, and the output is the retest result.

[0638] (Application example 2)

[0639] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0640] Traditional application testing systems focused on test automation but failed to consider user emotions and usage. As a result, while testing could detect technical defects, it was unable to adequately address user experience improvements. Furthermore, the inability to reflect user confusion and dissatisfaction in real time limited UI / UX optimization.

[0641] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for loading a generative AI model, a means for generating a test scenario, and a means for analyzing test results and emotion data. This makes it possible to collect and analyze user emotion data in real time and test and improve applications taking user experience into consideration.

[0642] A "generative artificial intelligence model" is a type of artificial intelligence used to generate test scenarios, and has the ability to automatically create appropriate scenarios based on user input and operations.

[0643] A "test scenario" refers to a set of pre-defined operational procedures or conditions for verifying the application's functions and user interface.

[0644] "Device" refers to an electronic device such as a computer, smartphone, or tablet on which the application being tested runs.

[0645] "User emotion data" refers to emotion information obtained in real time from the user's facial expressions and voice, and includes emotions such as "satisfaction," "dissatisfaction," and "confusion."

[0646] "Log" refers to data that records the application's operating status, user operation history, emotional data, etc.

[0647] "Server" refers to a centralized computer system that generates test scenarios, analyzes the results, collects and stores emotional data, and so on.

[0648] The "cause of failure" refers to the cause of an application not operating normally.

[0649] "Correction method" refers to the specific methods and procedures for solving the discovered cause of failure.

[0650] "Retesting" refers to the process of verifying the modified application again according to the test scenarios to confirm that the improvements have been properly implemented.

[0651] This invention is a system for automating application testing and taking user emotions into account. Specifically, it uses a system that combines a generative artificial intelligence model and an emotion engine.

[0652] First, the server loads a generative AI model. This model generates test scenarios based on user input and operation history. For example, it generates a test scenario for the process of searching for a specific product and adding it to a cart.

[0653] The server then loads the emotion engine, which has the ability to acquire emotion data in real time from the user's facial expressions and voice. The data is collected from devices such as smartphones and tablets.

[0654] When a user provides information about the application under test to the server from their device, the server uses a generative artificial intelligence model to generate a test scenario and sends it to the device. For example, a scenario in which a user searches for a product in a virtual store and adds it to a cart is considered.

[0655] The device operates the application according to the test scenario sent from the server. During the test, the emotion engine collects emotional data in real time from the user's facial expressions and voice. The device records these test results and emotional data in a log and sends them to the server.

[0656] The server analyzes the received log data to identify the cause of the application failure and how to fix it. For example, if the user expresses "confusion" or "dissatisfaction," the server will suggest improvements to the UI / UX based on that emotion data.

[0657] The modified application is then retested, again using the emotion engine to verify whether user satisfaction has improved. To ensure the modified application behaves as expected, retest scenarios are also generated using the generative artificial intelligence model.

[0658] This system allows for comprehensive application testing that takes into account user emotions, rather than simply fixing technical bugs, and allows for application improvements that satisfy users.

[0659] Examples:

[0660] For example, if a user expresses "confusion" in the process of searching for a specific product in a virtual store app and adding it to the cart, a log of this emotion can be collected and suggestions for improving the UI / UX can be generated.An example of a prompt to input to the generative AI model could be, "Generate a test scenario for the virtual store application where the user searches for a specific product and adds it to the shopping cart. Include various steps and possible user inputs, including both successful and erroneous entries."

[0661] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0662] Step 1:

[0663] The server loads the generative AI model. The input is the path of the generative AI model, and the output is the loaded model object. This model object is used to prepare for generating test scenarios.

[0664] Step 2:

[0665] The server loads the emotion engine. The input is the path to the emotion engine, and the output is the loaded engine object. This engine object is used to prepare for collecting user emotion data in real time. Specifically, the parameters of the emotion recognition model are set.

[0666] Step 3:

[0667] The user provides information about the application under test to the server from their device. The input is the information about the application under test (application path, settings, test scope), and the output is the information object received by the server. The server uses this to input prompts to the generative AI model.

[0668] Step 4:

[0669] The server generates test scenarios using a generative AI model. The input is the application information sent by the user, and the output is the generated test scenario. The generative AI model outputs a specific test scenario using a prompt statement.

[0670] Step 5:

[0671] The server sends the generated test scenario to the terminal. The input is the generated test scenario, and the output is confirmation data of the transmission result. The server verifies whether the test scenario was sent correctly to the terminal.

[0672] Step 6:

[0673] The terminal operates the application according to the test scenario. The input is the test scenario received from the server, and the output is the test result. Specifically, the terminal automatically executes the specified actions and records the operation log.

[0674] Step 7:

[0675] The device collects the user's emotional data in real time. The input is real-time data from the camera and microphone, and the output is emotional data. The device inputs this data into the emotion engine and obtains the emotion recognition results.

[0676] Step 8:

[0677] The device sends a log containing the test results and emotional data to the server. The input is the log containing the test results and emotional data, and the output is a transmission confirmation data sent to the server. The server performs analysis based on this.

[0678] Step 9:

[0679] The server analyzes the received log data and identifies the cause of the application failure and how to fix it. The input is the log data sent from the device, and the output is the analysis results including how to fix it. This makes it clear what specific points need to be fixed.

[0680] Step 10:

[0681] The server generates a new test scenario for retesting the modified application and sends it to the terminal. The input is the modified application information, and the output is the new test scenario. The same process is repeated to check the user's satisfaction.

[0682] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0683] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0684] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0685] [Third embodiment]

[0686] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0687] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0688] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0689] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0690] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0691] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0692] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0693] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0694] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0695] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0696] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0697] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0698] The present invention is a system that uses a generative artificial intelligence model to automate application testing and reduce the effort and cost. Specific embodiments of this system are described below.

[0699] First, the server loads the generative AI model, which uses deep learning techniques and is used to generate test scenarios and analyze the results.

[0700] Next, the user inputs information about the application under test into the server from their device, including the application path, settings, and test scope. Once the server receives the information from the user, it uses a generative AI model to generate a test scenario. This scenario includes, for example, various UI operations based on different input values ​​and operation patterns.

[0701] The server sends the generated test scenario to the terminal. The terminal receives the test scenario, launches the application, and performs operations according to the specified scenario. As a concrete example, consider testing a user registration form. The terminal enters a username, email address, and password in the correct format and clicks the registration button. This also includes entering an invalid email address or a short password. This covers both normal and error behavior.

[0702] As a result of the test, the device collects logs of successes, failures, error messages, etc. and sends them to the server. The server analyzes the received log data and identifies the cause of the failure. For example, if there is a problem with the validation of an email address, it will detect this and generate advice such as "the validation logic needs to be corrected."

[0703] The application is modified based on the modification information provided by the user. Once the modification is complete, the server generates a new test scenario, sends it to the device, and performs a retest. Through the retest, it is confirmed whether the modification was performed properly, and the results are collected and analyzed again before a final report is made.

[0704] The above is a specific embodiment of the system based on the present invention. This system utilizes AI to automatically perform a variety of tests, thereby improving the efficiency and quality of tests.

[0705] The processing flow will be explained below.

[0706] Step 1:

[0707] The server loads the generative AI model. Specifically, it extracts the file of the generative AI model into memory and makes the model ready for use.

[0708] Step 2:

[0709] The user enters information about the application to be tested into the server from the terminal. Specifically, the user enters the application path (e.g., " / path / to / application") and initial settings (e.g., test scope and input item definitions), and sends them to the server.

[0710] Step 3:

[0711] The server generates test scenarios using a generative AI model based on the application information it receives. For example, it creates test scenarios for valid input (username, email address, password in the correct format) and invalid input (invalid email address, password that is too short).

[0712] Step 4:

[0713] The server saves the generated test scenario in JSON format, ensuring that the scenario is used consistently in subsequent processing.

[0714] Step 5:

[0715] The server sends the saved test scenario to the device, and passes the scenario to the device's API endpoint.

[0716] Step 6:

[0717] The terminal loads the test scenario received from the server and launches the application. Specifically, it opens the application's executable file and makes the GUI operable.

[0718] Step 7:

[0719] The device automatically executes UI operations according to the test scenario, such as entering a username, email address, and password, and clicking the send button.

[0720] Step 8:

[0721] The device collects the results of the test, including successful and unsuccessful operations and any error messages displayed.

[0722] Step 9:

[0723] The test results collected by the device are sent to the server. Specifically, the test results are compiled in JSON format and sent to the server's receiving API.

[0724] Step 10:

[0725] The server analyzes the test results it receives and uses a generative AI model to identify the specific cause of the failure and suggest a fix based on that. For example, it generates a message such as "Email address validation logic is required."

[0726] Step 11:

[0727] The server notifies the user of the proposed correction method. Possible notification methods include displaying the result on the dashboard or sending an email.

[0728] Step 12:

[0729] The user modifies the application based on the provided fixes, specifically by modifying the code in the development environment and building the modified version.

[0730] Step 13:

[0731] The user notifies the server that the modification is complete, for example, by sending the modified application path to the server.

[0732] Step 14:

[0733] The server then generates a new test scenario and sends it to the device, which then runs a retest to verify whether the modifications were made properly.

[0734] Step 15:

[0735] The retest results are collected, analyzed, and finally reported to the user. After confirming that the problems discovered in the previous process have been resolved, a final report is made.

[0736] The above are the specific processing steps of the system.

[0737] Example 1

[0738] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0739] The application testing process is not automated enough, resulting in labor-intensive and costly tasks. Creating test scenarios and analyzing test results takes a lot of time, reducing overall testing efficiency. Furthermore, human error and insufficient test coverage make it difficult to improve application quality.

[0740] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0741] In this invention, the server includes means for loading a generative AI model using a generating means, means for a user to input application information to be tested, means for generating a test scenario using the generative AI model, means for transmitting the test scenario to a terminal, means for the terminal to launch and operate the application according to the test scenario, means for testing various input values ​​to test both normal and abnormal operation, means for collecting test results and transmitting them to the server, means for the server to analyze the test results and provide causes of failure and correction methods, means for retesting the corrected application based on the correction information, and means for reporting and analyzing the final results through the retest and generating a report. This makes it possible to automate the application testing process, reduce effort and cost, and improve overall testing efficiency.

[0742] "Generating means" refers to a device or program that has the function of loading a generating artificial intelligence model into the system.

[0743] A "generative artificial intelligence model" refers to a model that uses deep learning technology to generate test scenarios and analyze test results.

[0744] The term "means for the user to input information about the application to be tested" refers to a device or program that allows the user to input information such as the path, settings, and test scope of the application to be tested to the server.

[0745] A "test scenario" refers to a scenario that includes a series of operation procedures and input value actions that are generated for the application under test.

[0746] "Terminal" refers to a device that performs operations according to the test scenario received from the server and collects the test results.

[0747] "Means for testing various input values ​​to test both normal and abnormal operation" refers to a function for attempting both correct and incorrect input to an application and checking its operation.

[0748] "Means for collecting test results and sending them to a server" refers to a device or program for collecting result data such as logs and screenshots after test execution and sending them to a server.

[0749] "Means for analyzing test results on the server and providing the cause of failure and how to correct it" refers to the function for the server to analyze the test result data received, identify the cause of the error and how to correct it, and provide the information to the user.

[0750] "Means for retesting" refers to the function of generating new test scenarios for the modified application and running the tests again.

[0751] The "means for generating a report" refers to a function for compiling the final test results and generating a document to be provided to the user.

[0752] The present invention is a system that uses a generative artificial intelligence model to automate application testing and reduce the effort and cost. Specific embodiments of this system are described below.

[0753] First, the server loads a generative AI model using a generator. This generative AI model employs deep learning techniques and is pre-trained using libraries such as TensorFlow and PyTorch. This model is then used to generate test scenarios and analyze the results.

[0754] Next, the user enters the application information to be tested into the server from the terminal. This input includes the application path, configuration information, test scope, etc. For example, the user can use a browser to enter information such as " / path / to / application", "configuration information", and "Module A to be tested" into the form and click the submit button.

[0755] When the server receives input information from the user, it uses a generative AI model to generate a test scenario. The generated test scenario includes a variety of UI operations based on different input values ​​and operation patterns. At this time, a prompt such as "Please create a test case for the user registration form" is passed to the generative AI model, and the returned test scenario is obtained.

[0756] The server sends the generated test scenario to the terminal. This scenario is sent in JSON format and is deployed on the terminal. The terminal receives the test scenario and launches the application and performs operations according to the scenario specified in the test environment. For example, to test a user registration form, enter "testuser", "testuser@example.com", and "password123" and click the Register button. Invalid email addresses and short passwords are also entered to test both normal and error behavior.

[0757] The device collects logs of the test results, including success, failure, and error messages, and sends them to the server. The server analyzes the received log data and identifies the cause of the failure. For example, if there is a problem with email address validation, it will detect this and generate advice such as "the validation logic needs to be corrected."

[0758] The user modifies the application source code based on the provided modification information. After the modification is complete, the server generates a new test scenario and sends it to the terminal for re-testing. Through the re-test, it is confirmed whether the modification was properly made, and the results are collected again, analyzed, and a final report is generated. The user receives the final report and can identify areas for improvement or problems.

[0759] This system automates the application testing process, reducing the effort and cost involved, and also improves the quality and efficiency of testing by automatically generating test scenarios using generative AI models and analyzing the results.

[0760] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0761] Step 1:

[0762] The server loads the generative AI model using a generator, initializes a specific library (e.g., TensorFlow or PyTorch), and loads a pre-trained model file. The input is the model file path, and the output is the generative AI model deployed in memory.

[0763] Specific behavior:

[0764] The server reads " / path / to / model / file" and loads the model using the TensorFlow API.

[0765] Step 2:

[0766] The user inputs the application information to be tested from the terminal to the server. This information includes the application path, settings, and test scope. The input is the information entered by the user, and the output is the test information sent to the server.

[0767] Specific behavior:

[0768] The user enters " / path / to / application," "configuration information," and "module A to be tested" into the browser form and clicks the "Submit" button.

[0769] Step 3:

[0770] The server generates test scenarios using a generative AI model based on the received information. The input is the application information and prompts provided by the user, and the output is the generated test scenario.

[0771] Specific behavior:

[0772] The server passes the prompt "Please create a test case for the user registration form" to the generative AI model and obtains the returned test scenario.

[0773] Step 4:

[0774] The server sends the generated test scenario to the terminal. The input is the test scenario, and the output is the scenario sent to the terminal.

[0775] Specific behavior:

[0776] The server saves the generated scenario as " / tests / scenario1.json" and provides a download link to the device.

[0777] Step 5:

[0778] The device executes the application in the test environment according to the received test scenario, with the test scenario as input and the execution result log and screenshots as output.

[0779] Specific behavior:

[0780] The terminal opens the "User Registration Form", enters "testuser", "testuser@example.com", and "password123", and clicks the "Register" button.

[0781] Step 6:

[0782] The terminal sends the collected test results to the server. The input is the execution result, and the output is the log and screenshots sent to the server.

[0783] Specific behavior:

[0784] The device sends files such as " / logs / test1.log" and " / screenshots / test1.png" to the server via an HTTP POST request.

[0785] Step 7:

[0786] The server analyzes the received test results and provides the cause of failure and how to fix it. The input is logs and screenshots, and the output is a report containing fix information.

[0787] Specific behavior:

[0788] The server parses the log file, detects the error message "Email address validation failed" and generates an advice "Fix the validation logic".

[0789] Step 8:

[0790] The user modifies the application based on the modification information from the server. The input is the modification information from the server, and the output is the modified application.

[0791] Specific behavior:

[0792] The user opens a code editor, modifies the "email_validation.py" file, and rebuilds it.

[0793] Step 9:

[0794] The server generates a new test scenario for the modified application and sends it to the terminal to re-execute the test. The input is the modified application information and prompt, and the output is the re-generated test scenario.

[0795] Specific behavior:

[0796] The server generates " / tests / scenario2.json" and sends it to the device, which runs the test again and collects new results.

[0797] Step 10:

[0798] The server collects and analyzes the retest results and generates the final report. The input is the retest results and the output is the final report.

[0799] Specific behavior:

[0800] The server generates " / reports / final_report.pdf" and provides the user with a download link.

[0801] (Application example 1)

[0802] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0803] Conventional product inspection systems rely on manual inspection, which often results in reduced inspection efficiency and accuracy. Furthermore, manual inspection is time-consuming and prone to human error, creating a need for improved quality control. Furthermore, existing automated inspection systems lack flexibility, making it difficult to accommodate a wide variety of products and inspection items. A new system is needed to resolve these issues, streamline product inspection within factories, and improve quality.

[0804] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0805] In this invention, the server includes means for loading a generative artificial intelligence model, means for generating a test scenario using the generative artificial intelligence model, means for transmitting the test scenario to an operation means, means for the operation means to operate an application according to the test scenario, means for collecting and transmitting test results to the server, means for analyzing the test results in the server and providing causes of failure and correction methods, means for retesting the corrected application, means for generating a product inspection scenario and having an in-factory device execute it, means for collecting and analyzing the inspection results and identifying defective products and providing points to be corrected, and means for generating and re-executing a new test scenario for reinspection, thereby enabling improvement in the efficiency and accuracy of product inspection.

[0806] A "generative artificial intelligence model" is a collection of algorithms that use AI techniques such as deep learning to process data for a specific task and make predictions or decisions.

[0807] A "test scenario" is a plan that defines specific operational procedures and input data to verify the operation and functionality of software or a system.

[0808] "Operation means" refers to devices or programs for operating applications or systems according to test scenarios.

[0809] "Test results" are information on operation confirmation such as success, failure, and error messages obtained when a test scenario is executed.

[0810] A "server" is a computer system that provides services such as data processing and storage over a network.

[0811] "Correction methods" are specific countermeasures or procedures for resolving problems or defects discovered during testing.

[0812] "Retesting" is the process of running tests again after a fix has been made to verify that the fix was successful.

[0813] "Factory equipment" refers to machines and robot systems used in factories to perform product inspections and production work.

[0814] "Inspection results" refers to data such as measurement values ​​and appearance check results obtained during product inspection.

[0815] A "defective product" is a product that does not meet standards or is determined to be defective during product inspection.

[0816] "Correction points" are specific locations and means for correcting identified defects or flaws based on the inspection results.

[0817] The present invention provides a system for automating product inspection during the production process, thereby improving efficiency and accuracy. Specific embodiments for carrying out the present invention will be described below.

[0818] The server first loads a generative artificial intelligence model. This generative AI model is based on deep learning technology and is optimized for product inspection tasks. The server also provides an interface for receiving product information and inspection items entered by factory staff. Product information includes model numbers, specifications, and details of inspection items, and generates test scenarios based on this information.

[0819] The generated test scenario is sent to the operating means in the factory. The operating means can be a robot or measuring device for product inspection. Specifically, it can be, for example, a laser tape measure for measuring dimensions or an inspection robot equipped with a high-resolution camera. These devices inspect the products according to the test scenario sent from the server.

[0820] Inspection results are sent from the operation device to the server. The server analyzes the received inspection results and identifies defective products and provides correction points. For example, if dimensions are outside the standard range or if surface scratches are detected in camera images, specific correction points will be presented. This allows factory staff to quickly carry out correction work.

[0821] After the corrections are complete, the server again uses the generative AI model to generate new test scenarios and conduct a retest to confirm whether the corrections were effective. The results of the retest are also sent to the server for final analysis.

[0822] To explain the effectiveness of this system, let's use a concrete example. For example, in the case of product inspection of smartphone cases, the prompt sentence to check the dimensions of the case and the presence of scratches on the surface is as follows:

[0823] "Measure the dimensions of your smartphone case and make sure it meets the standard values ​​(height: 150mm, width: 75mm, depth: 10mm). Also, check for scratches on the surface and report any defects."

[0824] Based on the prompts, the server generates a test scenario, and the operating device executes the test. The test results are collected on the server for detailed analysis. The test results include dimensional measurements and the presence or absence of surface scratches, and if the standards are not met, specific correction points are provided.

[0825] In this way, the system based on the present invention realizes efficient product inspection and improved quality.

[0826] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0827] Step 1:

[0828] The server loads the generative AI model, which is based on deep learning technology and optimized for product inspection tasks. Once the model has been loaded, the server prepares to accept product information and inspection items.

[0829] Input: Deep learning model file

[0830] Output: The loaded generative AI model

[0831] How it works: The server loads the generative AI model from file storage and deploys it in memory. This model includes neural networks for performing tasks such as object detection, dimension measurement, and visual inspection.

[0832] Step 2:

[0833] The user uses a tablet or PC to input product information (model number, specifications, inspection items) into the server, including details of the product to be inspected.

[0834] Input: Product information (model number, specifications, inspection items)

[0835] Output: Product information received by the server

[0836] How it works: The user enters product information into a dedicated interface and sends it to the server, which then prepares the information for input into the generative AI model.

[0837] Step 3:

[0838] The server uses the generative AI model to generate test scenarios, which include specific testing procedures for each product.

[0839] Input:Product Information

[0840] Output: Test scenario

[0841] How it works: The server inputs product information into the generative AI model and generates test scenarios, including procedures for measuring dimensions and inspecting surfaces. For example, "Measurement procedures to confirm a height of 150 mm, width of 75 mm, and depth of 10 mm."

[0842] Step 4:

[0843] The server sends the generated test scenario to the operating means, which is usually a robot or inspection device in a factory.

[0844] Input: Test scenario

[0845] Output: Test scenario sent to the control

[0846] Specific operation: The server transmits the test scenario to the operation means via the network, using a data transfer protocol to ensure reliable communication.

[0847] Step 5:

[0848] The operating means (for example, an inspection robot) performs inspections according to the test scenario, including dimensional measurements and surface inspections.

[0849] Input: Test scenario

[0850] Output: Inspection results (measurement values, abnormality detection information, etc.)

[0851] Specific operation: The operating means uses a laser tape measure and a high-resolution camera to inspect the product based on the test scenario. For example, the robot moves through each part of the product to measure its dimensions, and a high-resolution camera detects surface scratches.

[0852] Step 6:

[0853] The test results are transmitted from the operating means to the server, and the server receives them.

[0854] Input: Test results

[0855] Output: Inspection results aggregated on the server

[0856] Specific operation: The operating means sends the inspection results (e.g., size measurements and the presence or absence of surface abnormalities) to the server. The server stores this data as a log and prepares it for analysis.

[0857] Step 7:

[0858] The server analyzes the inspection results and identifies defects and provides correction points.

[0859] Input: Test results

[0860] Output: Analysis results (defective product identification, correction points)

[0861] Specific operation: The server uses machine learning algorithms to analyze the inspection results and identify defective products. It also generates specific instructions for correction, such as "This product's height exceeds the specified value, so it needs to be reprocessed."

[0862] Step 8:

[0863] The user receives instructions from the server and performs the work to correct the product.

[0864] Input: Correction instructions from the server

[0865] Output: Modified product

[0866] Specific operation: The user checks the correction instructions provided by the server and reprocesses or reinspects the product.

[0867] Step 9:

[0868] For the corrected product, new test scenarios are generated for retesting, and the product is retested.

[0869] Input: Revised product information

[0870] Output: Retest scenario, retest results

[0871] Specific operation: The server generates a new test scenario based on the corrected product information and has the operating means re-execute it. The retest results are sent to the server, and a final pass / fail decision is made.

[0872] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0873] This invention is a system that combines a generative artificial intelligence model and an emotion engine to automate application testing and take user emotions into consideration. Specific embodiments of this system are described below.

[0874] First, the server loads the generative AI model and emotion engine. The generative AI model is used to generate test scenarios, and the emotion engine recognizes user emotions in real time.

[0875] Next, the user provides information about the application under test to the server from their device. This information includes the application path, settings, and test scope. The server uses the received information to generate test scenarios using a generative AI model. For example, it creates test scenarios with various patterns that include input in the correct format and errors.

[0876] The server sends the generated test scenario to the terminal, which receives the scenario, launches the application, and performs the specified operations. As a concrete example, consider testing a user registration form. The terminal enters a username, email address, and password in the correct format and clicks the submit button. At the same time, the emotion engine obtains emotional data from the user's facial expressions and voice and records it in a log. If the emotional data indicates, for example, "dissatisfaction" or "confusion," this is collected as a test result.

[0877] As a result of the test, the device sends a log containing success or failure results, error messages, and emotional data to the server. The server analyzes the received log data and identifies the cause of the failure and how to correct it based on the user's emotional state. For example, it provides information such as "There was a problem with email address validation, which caused a lot of dissatisfaction."

[0878] The application is modified based on the modification information provided by the user. Once the modification is complete, the server generates a new test scenario, sends it to the device, and performs a retest. At this time, the emotion engine is again used to collect and analyze the overall test results, including emotion data. This allows the system to confirm that the modifications have been made appropriately and in a way that satisfies the user.

[0879] The above is a specific embodiment of the system based on the present invention, which combines AI and emotion recognition technology to achieve more user-friendly and effective test automation.

[0880] The processing flow will be explained below.

[0881] Step 1:

[0882] The server loads the generative AI model and emotion engine. Specifically, it expands the program files for the generative AI model and emotion engine into memory, making both models available for use.

[0883] Step 2:

[0884] The user enters information about the application to be tested into the server from the terminal, including the application path (e.g., " / path / to / application"), the test scope, and initial setting items.

[0885] Step 3:

[0886] The server generates test scenarios using a generative AI model based on the application information it receives. For example, for a user registration form, multiple scenarios for valid and invalid inputs are created in JSON format.

[0887] Step 4:

[0888] The server stores the generated test scenarios in a database, ensuring that the scenarios are used consistently in subsequent processes.

[0889] Step 5:

[0890] The server sends the saved test scenario to the device, which passes the test scenario to the device via an API endpoint.

[0891] Step 6:

[0892] The device loads the test scenario received from the server and launches the application. For example, it opens the application's executable file and makes the UI operable.

[0893] Step 7:

[0894] The device automatically executes UI operations according to the test scenario. At the same time, the emotion engine is activated and collects emotional data in real time from the user's facial expressions and voice. For example, if the user shows signs of "confusion" or "dissatisfaction" while entering their username, email address, and password, and then performing the send operation, this data is also recorded.

[0895] Step 8:

[0896] The device collects test results and emotional data, including successful and unsuccessful operations, displayed error messages, and emotional data.

[0897] Step 9:

[0898] The device sends the collected test results and emotion data to the server, which includes compiling the test results and emotion data in JSON format and sending them to the server's receiving API.

[0899] Step 10:

[0900] The server analyzes the test results and emotion data it receives. It uses a generative AI model to identify the specific cause of the failure and extracts the user's perceived issues based on the emotion data. For example, if there is a problem with email address validation, it generates a message saying, "The validation logic needs to be corrected."

[0901] Step 11:

[0902] The server notifies the user of the proposed correction method. Possible notification methods include displaying the information on the dashboard or sending an email. Specifically, it presents in detail the areas that need to be corrected and how to do so.

[0903] Step 12:

[0904] The user modifies the application based on the provided fixes, modifies the code in the development environment, and builds the modified version.

[0905] Step 13:

[0906] The user notifies the server that the modification is complete, for example, by sending the modified application path to the server.

[0907] Step 14:

[0908] The server generates a new test scenario again and sends it to the terminal. The retest runs a test to check whether the corrections have been made properly.

[0909] Step 15:

[0910] The device will then conduct the retest and send the results to the server. The device will also use the emotion engine to collect emotional data during the retest and send the results, including this data, to the server.

[0911] Step 16:

[0912] The server finally analyzes all test results and emotion data and reports back to the user, ensuring that the corrections were made properly and providing the user with a final report.

[0913] The above are the specific processing steps of the system.

[0914] Example 2

[0915] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0916] Traditional application testing is often done manually, requiring a large amount of work and time. Furthermore, there is no system for evaluating test results that takes user emotions into account, making it difficult to identify problems based on user experience. Therefore, there is a need for more efficient, user-centric testing by introducing test automation and a system that reflects user emotions in real time.

[0917] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for loading a generative AI model, means for loading an emotion engine, means for generating a test scenario using the generative AI model, means for transmitting the generated test scenario to a terminal, means for the terminal to run an application according to the test scenario, means for acquiring user emotion data using the emotion engine, means for collecting test results and emotion data and transmitting them to the server, means for analyzing the test results and emotion data in the server and providing causes of failure and correction methods, and means for retesting the corrected application. This enables test automation and comprehensive test evaluation that takes user emotions into consideration.

[0918] "Generative artificial intelligence model" refers to an algorithm or system that uses artificial intelligence to automatically generate test scenarios and other artifacts.

[0919] An "emotion engine" refers to a technology or system that analyzes a user's facial expressions and voice to recognize and record emotions in real time.

[0920] A "test scenario" refers to a plan containing a series of steps or procedures used in performing operational testing of an application.

[0921] "Terminal" refers to a device that receives instructions from a server, executes applications, and obtains operation and test results.

[0922] "Server" refers to the central computer system that loads the generative AI models and emotion engines, generates and transmits test scenarios, and analyzes collected data.

[0923] "User emotion data" refers to data indicating the user's emotional state obtained using an emotion engine.

[0924] "Test results" refers to the result information such as success / failure and error messages when testing an application.

[0925] "Correction method" refers to the specific means or steps to resolve application or system problems based on the analysis results.

[0926] "Retesting" refers to the process of testing an application again after it has been modified to ensure that the modifications function correctly.

[0927] MODE FOR CARRYING OUT THE INVENTION

[0928] This invention is a system that combines a generative artificial intelligence model and an emotion engine to automate application testing and take user emotions into consideration. Specific embodiments of this system are described below.

[0929] Hardware and software used

[0930] Server: The central computer system of the system, which loads the generative AI model and emotion engine, generates and transmits test scenarios, and analyzes the collected data.

[0931] Example of use: Server machines with high performance processors, such as Linux servers.

[0932] Terminal: A device that receives instructions from the server, executes applications, and obtains operation and test results.

[0933] Example of use: A typical personal computer (Windows or macOS) or a virtual machine.

[0934] Generative AI models: Artificial intelligence algorithms that automatically generate test scenarios and other artifacts.

[0935] Usage example: Using "GPT-3" and "BERT".

[0936] Emotion engine: Technology that analyzes the user's facial expressions and voice to recognize and record emotions in real time.

[0937] Usage examples: "Affectiva" and "Microsoft Azure Emotion API".

[0938] Specific examples of processing

[0939] First, the server loads the generative AI model and emotion engine. The generative AI model is used to generate test scenarios, and the emotion engine is responsible for recognizing user emotions in real time.

[0940] Next, the user uses the terminal to provide information about the application under test to the server, including the application path, settings, test scope, etc.

[0941] The server then uses the received information to generate appropriate test scenarios using a generative AI model. For example, it sends a prompt to the AI, such as "Please generate a test scenario for when a user enters an incorrect format in the email address field," and uses the response to create the scenario.

[0942] The server sends the generated test scenario to the terminal, and the terminal launches the application according to the scenario and performs specific operations. For example, run " / path / to / app", enter "testuser" in the username field, "wrong_email_format" in the email address field, and "password123" in the password field, and click the submit button.

[0943] At the same time, the emotion engine analyzes the user's facial expressions and voice to obtain emotional data. For example, if the user makes a "confused" expression, this is recorded as emotional data in the log. The device then saves this in a log file.

[0944] Once the test is complete, the device sends a log containing the test success / failure, error messages, and emotion data to the server. The server analyzes the received log data and identifies the cause of the failure and how to fix it. For example, it may conclude that "email address validation errors caused many users to feel confused."

[0945] The user modifies the application based on the analysis results provided by the server. For example, the user makes specific modifications such as "strengthening the validation function for email addresses."

[0946] Finally, the server generates a new test scenario using the generative AI model again and sends it to the device for retesting. This retest also uses the emotion engine, and collects and analyzes the overall test results.

[0947] This allows for comprehensive testing that takes user emotions into account, resulting in the development of higher quality, more user-friendly applications.

[0948] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0949] Step 1:

[0950] The server loads the generative AI model and emotion engine. First, the server loads the generative AI model and emotion engine from existing data storage into memory and prepares them. GPT-3 or BERT is used as the generative AI model, and Affectiva or Microsoft Azure Emotion API is used as the emotion engine. This enables the server to generate test scenarios and recognize user emotions. The file paths of the generative AI model and emotion engine are required as input, and the loaded model and engine are obtained as output.

[0951] Step 2:

[0952] The user uses a terminal to enter information about the application under test and provides it to the server. The user enters the application path, settings, test scope, etc. into the terminal and clicks the send button. The terminal sends the entered information to the server. The application path, setting file, etc. are required as input, and this information is obtained as output to be sent to the server.

[0953] Step 3:

[0954] The server generates a test scenario using the generative AI model. Based on the application information provided by the user, the server inputs a prompt statement to the generative AI model to generate an appropriate test scenario. For example, the server uses the prompt statement "Please generate a test scenario for when the user enters an incorrect format in the email address field" to obtain the generated scenario. The input of this step is the prompt statement and application information, and the output is the generated test scenario.

[0955] Step 4:

[0956] The server sends the generated test scenario to the terminal. The server packages the generated test scenario and sends it to the terminal. The generated test scenario is required as input, and the scenario sent to the terminal is obtained as output.

[0957] Step 5:

[0958] The terminal launches the application according to the received test scenario and performs the specified operation. The terminal first launches the application from the specified path, and then performs the input operations according to the scenario. Specifically, it enters "testuser" in the username field, "wrong_email_format" in the email address field, and "password123" in the password field, and then clicks the send button. The input for this step is the test scenario and application information, and the output is the result of the executed operation.

[0959] Step 6:

[0960] The emotion engine analyzes the user's facial expressions and voice to obtain emotional data. The device uses the emotion engine to analyze the user's real-time facial expressions and voice to collect emotional data. If the emotional data indicates "confusion" or "dissatisfaction," this is recorded in a log. The user's facial expressions and voice data are required as input, and the collected emotional data is obtained as output.

[0961] Step 7:

[0962] The device sends a log containing the success or failure of the test, error messages, and emotional data to the server. The device saves the executed operation results and emotional data together in a log file and sends the log to the server. The operation results and emotional data are required as input, and the log data sent to the server is obtained as output.

[0963] Step 8:

[0964] The server analyzes the received log data to identify the cause of the failure and how to fix it. The server analyzes the log data to identify the problem and its cause. For example, it may conclude that "email address validation errors caused frequent user confusion." The input to this step is the log data, and the output is the analysis results and how to fix it.

[0965] Step 9:

[0966] The user modifies the application based on the analysis results provided by the server. The user then modifies the application code according to the instructions provided by the server. For example, the user makes a specific modification such as "strengthening the email address validation function." The input requires the analysis results and instructions, and the output is the modified application.

[0967] Step 10:

[0968] The server then uses the generative AI model to generate a new test scenario, sends it to the device, and runs the test again. All steps are repeated to verify that the modifications function properly. The inputs are the modified application information and a new prompt, and the output is the retest result.

[0969] (Application example 2)

[0970] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0971] Traditional application testing systems focused on test automation but failed to consider user emotions and usage. As a result, while testing could detect technical defects, it was unable to adequately address user experience improvements. Furthermore, the inability to reflect user confusion and dissatisfaction in real time limited UI / UX optimization.

[0972] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for loading a generative AI model, a means for generating a test scenario, and a means for analyzing test results and emotion data. This makes it possible to collect and analyze user emotion data in real time and test and improve applications taking user experience into consideration.

[0973] A "generative artificial intelligence model" is a type of artificial intelligence used to generate test scenarios, and has the ability to automatically create appropriate scenarios based on user input and operations.

[0974] A "test scenario" refers to a set of pre-defined operational procedures or conditions for verifying the application's functions and user interface.

[0975] "Device" refers to an electronic device such as a computer, smartphone, or tablet on which the application being tested runs.

[0976] "User emotion data" refers to emotion information obtained in real time from the user's facial expressions and voice, and includes emotions such as "satisfaction," "dissatisfaction," and "confusion."

[0977] "Log" refers to data that records the application's operating status, user operation history, emotional data, etc.

[0978] "Server" refers to a centralized computer system that generates test scenarios, analyzes the results, collects and stores emotional data, and so on.

[0979] The "cause of failure" refers to the cause of an application not operating normally.

[0980] "Correction method" refers to the specific methods and procedures for solving the discovered cause of failure.

[0981] "Retesting" refers to the process of verifying the modified application again according to the test scenarios to confirm that the improvements have been properly implemented.

[0982] This invention is a system for automating application testing and taking user emotions into account. Specifically, it uses a system that combines a generative artificial intelligence model and an emotion engine.

[0983] First, the server loads a generative AI model. This model generates test scenarios based on user input and operation history. For example, it generates a test scenario for the process of searching for a specific product and adding it to a cart.

[0984] The server then loads the emotion engine, which has the ability to acquire emotion data in real time from the user's facial expressions and voice. The data is collected from devices such as smartphones and tablets.

[0985] When a user provides information about the application under test to the server from their device, the server uses a generative artificial intelligence model to generate a test scenario and sends it to the device. For example, a scenario in which a user searches for a product in a virtual store and adds it to a cart is considered.

[0986] The device operates the application according to the test scenario sent from the server. During the test, the emotion engine collects emotional data in real time from the user's facial expressions and voice. The device records these test results and emotional data in a log and sends them to the server.

[0987] The server analyzes the received log data to identify the cause of the application failure and how to fix it. For example, if the user expresses "confusion" or "dissatisfaction," the server will suggest improvements to the UI / UX based on that emotion data.

[0988] The modified application is then retested, again using the emotion engine to verify whether user satisfaction has improved. To ensure the modified application behaves as expected, retest scenarios are also generated using the generative artificial intelligence model.

[0989] This system allows for comprehensive application testing that takes into account user emotions, rather than simply fixing technical bugs, and allows for application improvements that satisfy users.

[0990] Examples:

[0991] For example, if a user expresses "confusion" in the process of searching for a specific product in a virtual store app and adding it to the cart, a log of this emotion can be collected and suggestions for improving the UI / UX can be generated.An example of a prompt to input to the generative AI model could be, "Generate a test scenario for the virtual store application where the user searches for a specific product and adds it to the shopping cart. Include various steps and possible user inputs, including both successful and erroneous entries."

[0992] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0993] Step 1:

[0994] The server loads the generative AI model. The input is the path of the generative AI model, and the output is the loaded model object. This model object is used to prepare for generating test scenarios.

[0995] Step 2:

[0996] The server loads the emotion engine. The input is the path to the emotion engine, and the output is the loaded engine object. This engine object is used to prepare for collecting user emotion data in real time. Specifically, the parameters of the emotion recognition model are set.

[0997] Step 3:

[0998] The user provides information about the application under test to the server from their device. The input is the information about the application under test (application path, settings, test scope), and the output is the information object received by the server. The server uses this to input prompts to the generative AI model.

[0999] Step 4:

[1000] The server generates test scenarios using a generative AI model. The input is the application information sent by the user, and the output is the generated test scenario. The generative AI model outputs a specific test scenario using a prompt statement.

[1001] Step 5:

[1002] The server sends the generated test scenario to the terminal. The input is the generated test scenario, and the output is confirmation data of the transmission result. The server verifies whether the test scenario was sent correctly to the terminal.

[1003] Step 6:

[1004] The terminal operates the application according to the test scenario. The input is the test scenario received from the server, and the output is the test result. Specifically, the terminal automatically executes the specified actions and records the operation log.

[1005] Step 7:

[1006] The device collects the user's emotional data in real time. The input is real-time data from the camera and microphone, and the output is emotional data. The device inputs this data into the emotion engine and obtains the emotion recognition results.

[1007] Step 8:

[1008] The device sends a log containing the test results and emotional data to the server. The input is the log containing the test results and emotional data, and the output is a transmission confirmation data sent to the server. The server performs analysis based on this.

[1009] Step 9:

[1010] The server analyzes the received log data and identifies the cause of the application failure and how to fix it. The input is the log data sent from the device, and the output is the analysis results including how to fix it. This makes it clear what specific points need to be fixed.

[1011] Step 10:

[1012] The server generates a new test scenario for retesting the modified application and sends it to the terminal. The input is the modified application information, and the output is the new test scenario. The same process is repeated to check the user's satisfaction.

[1013] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1014] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1015] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1016] [Fourth embodiment]

[1017] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1018] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1019] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1020] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1021] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1022] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1023] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1024] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1025] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1026] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1027] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1028] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1029] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1030] The present invention is a system that uses a generative artificial intelligence model to automate application testing and reduce the effort and cost. Specific embodiments of this system are described below.

[1031] First, the server loads the generative AI model, which uses deep learning techniques and is used to generate test scenarios and analyze the results.

[1032] Next, the user inputs information about the application under test into the server from their device, including the application path, settings, and test scope. Once the server receives the information from the user, it uses a generative AI model to generate a test scenario. This scenario includes, for example, various UI operations based on different input values ​​and operation patterns.

[1033] The server sends the generated test scenario to the terminal. The terminal receives the test scenario, launches the application, and performs operations according to the specified scenario. As a concrete example, consider testing a user registration form. The terminal enters a username, email address, and password in the correct format and clicks the registration button. This also includes entering an invalid email address or a short password. This covers both normal and error behavior.

[1034] As a result of the test, the device collects logs of successes, failures, error messages, etc. and sends them to the server. The server analyzes the received log data and identifies the cause of the failure. For example, if there is a problem with the validation of an email address, it will detect this and generate advice such as "the validation logic needs to be corrected."

[1035] The application is modified based on the modification information provided by the user. Once the modification is complete, the server generates a new test scenario, sends it to the device, and performs a retest. Through the retest, it is confirmed whether the modification was performed properly, and the results are collected and analyzed again before a final report is made.

[1036] The above is a specific embodiment of the system based on the present invention. This system utilizes AI to automatically perform a variety of tests, thereby improving the efficiency and quality of tests.

[1037] The processing flow will be explained below.

[1038] Step 1:

[1039] The server loads the generative AI model. Specifically, it extracts the file of the generative AI model into memory and makes the model ready for use.

[1040] Step 2:

[1041] The user enters information about the application to be tested into the server from the terminal. Specifically, the user enters the application path (e.g., " / path / to / application") and initial settings (e.g., test scope and input item definitions), and sends them to the server.

[1042] Step 3:

[1043] The server generates test scenarios using a generative AI model based on the application information it receives. For example, it creates test scenarios for valid input (username, email address, password in the correct format) and invalid input (invalid email address, password that is too short).

[1044] Step 4:

[1045] The server saves the generated test scenario in JSON format, ensuring that the scenario is used consistently in subsequent processing.

[1046] Step 5:

[1047] The server sends the saved test scenario to the device, and passes the scenario to the device's API endpoint.

[1048] Step 6:

[1049] The terminal loads the test scenario received from the server and launches the application. Specifically, it opens the application's executable file and makes the GUI operable.

[1050] Step 7:

[1051] The device automatically executes UI operations according to the test scenario, such as entering a username, email address, and password, and clicking the send button.

[1052] Step 8:

[1053] The device collects the results of the test, including successful and unsuccessful operations and any error messages displayed.

[1054] Step 9:

[1055] The test results collected by the device are sent to the server. Specifically, the test results are compiled in JSON format and sent to the server's receiving API.

[1056] Step 10:

[1057] The server analyzes the test results it receives and uses a generative AI model to identify the specific cause of the failure and suggest a fix based on that. For example, it generates a message such as "Email address validation logic is required."

[1058] Step 11:

[1059] The server notifies the user of the proposed correction method. Possible notification methods include displaying the result on the dashboard or sending an email.

[1060] Step 12:

[1061] The user modifies the application based on the provided fixes, specifically by modifying the code in the development environment and building the modified version.

[1062] Step 13:

[1063] The user notifies the server that the modification is complete, for example, by sending the modified application path to the server.

[1064] Step 14:

[1065] The server then generates a new test scenario and sends it to the device, which then runs a retest to verify whether the modifications were made properly.

[1066] Step 15:

[1067] The retest results are collected, analyzed, and finally reported to the user. After confirming that the problems discovered in the previous process have been resolved, a final report is made.

[1068] The above are the specific processing steps of the system.

[1069] Example 1

[1070] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1071] The application testing process is not automated enough, resulting in labor-intensive and costly tasks. Creating test scenarios and analyzing test results takes a lot of time, reducing overall testing efficiency. Furthermore, human error and insufficient test coverage make it difficult to improve application quality.

[1072] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1073] In this invention, the server includes means for loading a generative AI model using a generating means, means for a user to input application information to be tested, means for generating a test scenario using the generative AI model, means for transmitting the test scenario to a terminal, means for the terminal to launch and operate the application according to the test scenario, means for testing various input values ​​to test both normal and abnormal operation, means for collecting test results and transmitting them to the server, means for the server to analyze the test results and provide causes of failure and correction methods, means for retesting the corrected application based on the correction information, and means for reporting and analyzing the final results through the retest and generating a report. This makes it possible to automate the application testing process, reduce effort and cost, and improve overall testing efficiency.

[1074] "Generating means" refers to a device or program that has the function of loading a generating artificial intelligence model into the system.

[1075] A "generative artificial intelligence model" refers to a model that uses deep learning technology to generate test scenarios and analyze test results.

[1076] The term "means for the user to input information about the application to be tested" refers to a device or program that allows the user to input information such as the path, settings, and test scope of the application to be tested to the server.

[1077] A "test scenario" refers to a scenario that includes a series of operation procedures and input value actions that are generated for the application under test.

[1078] "Terminal" refers to a device that performs operations according to the test scenario received from the server and collects the test results.

[1079] "Means for testing various input values ​​to test both normal and abnormal operation" refers to a function for attempting both correct and incorrect input to an application and checking its operation.

[1080] "Means for collecting test results and sending them to a server" refers to a device or program for collecting result data such as logs and screenshots after test execution and sending them to a server.

[1081] "Means for analyzing test results on the server and providing the cause of failure and how to correct it" refers to the function for the server to analyze the test result data received, identify the cause of the error and how to correct it, and provide the information to the user.

[1082] "Means for retesting" refers to the function of generating new test scenarios for the modified application and running the tests again.

[1083] The "means for generating a report" refers to a function for compiling the final test results and generating a document to be provided to the user.

[1084] The present invention is a system that uses a generative artificial intelligence model to automate application testing and reduce the effort and cost. Specific embodiments of this system are described below.

[1085] First, the server loads a generative AI model using a generator. This generative AI model employs deep learning techniques and is pre-trained using libraries such as TensorFlow and PyTorch. This model is then used to generate test scenarios and analyze the results.

[1086] Next, the user enters the application information to be tested into the server from the terminal. This input includes the application path, configuration information, test scope, etc. For example, the user can use a browser to enter information such as " / path / to / application", "configuration information", and "Module A to be tested" into the form and click the submit button.

[1087] When the server receives input information from the user, it uses a generative AI model to generate a test scenario. The generated test scenario includes a variety of UI operations based on different input values ​​and operation patterns. At this time, a prompt such as "Please create a test case for the user registration form" is passed to the generative AI model, and the returned test scenario is obtained.

[1088] The server sends the generated test scenario to the terminal. This scenario is sent in JSON format and is deployed on the terminal. The terminal receives the test scenario and launches the application and performs operations according to the scenario specified in the test environment. For example, to test a user registration form, enter "testuser", "testuser@example.com", and "password123" and click the Register button. Invalid email addresses and short passwords are also entered to test both normal and error behavior.

[1089] The device collects logs of the test results, including success, failure, and error messages, and sends them to the server. The server analyzes the received log data and identifies the cause of the failure. For example, if there is a problem with email address validation, it will detect this and generate advice such as "the validation logic needs to be corrected."

[1090] The user modifies the application source code based on the provided modification information. After the modification is complete, the server generates a new test scenario and sends it to the terminal for re-testing. Through the re-test, it is confirmed whether the modification was properly made, and the results are collected again, analyzed, and a final report is generated. The user receives the final report and can identify areas for improvement or problems.

[1091] This system automates the application testing process, reducing the effort and cost involved, and also improves the quality and efficiency of testing by automatically generating test scenarios using generative AI models and analyzing the results.

[1092] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1093] Step 1:

[1094] The server loads the generative AI model using a generator, initializes a specific library (e.g., TensorFlow or PyTorch), and loads a pre-trained model file. The input is the model file path, and the output is the generative AI model deployed in memory.

[1095] Specific behavior:

[1096] The server reads " / path / to / model / file" and loads the model using the TensorFlow API.

[1097] Step 2:

[1098] The user inputs the application information to be tested from the terminal to the server. This information includes the application path, settings, and test scope. The input is the information entered by the user, and the output is the test information sent to the server.

[1099] Specific behavior:

[1100] The user enters " / path / to / application," "configuration information," and "module A to be tested" into the browser form and clicks the "Submit" button.

[1101] Step 3:

[1102] The server generates test scenarios using a generative AI model based on the received information. The input is the application information and prompts provided by the user, and the output is the generated test scenario.

[1103] Specific behavior:

[1104] The server passes the prompt "Please create a test case for the user registration form" to the generative AI model and obtains the returned test scenario.

[1105] Step 4:

[1106] The server sends the generated test scenario to the terminal. The input is the test scenario, and the output is the scenario sent to the terminal.

[1107] Specific behavior:

[1108] The server saves the generated scenario as " / tests / scenario1.json" and provides a download link to the device.

[1109] Step 5:

[1110] The device executes the application in the test environment according to the received test scenario, with the test scenario as input and the execution result log and screenshots as output.

[1111] Specific behavior:

[1112] The terminal opens the "User Registration Form", enters "testuser", "testuser@example.com", and "password123", and clicks the "Register" button.

[1113] Step 6:

[1114] The terminal sends the collected test results to the server. The input is the execution result, and the output is the log and screenshots sent to the server.

[1115] Specific behavior:

[1116] The device sends files such as " / logs / test1.log" and " / screenshots / test1.png" to the server via an HTTP POST request.

[1117] Step 7:

[1118] The server analyzes the received test results and provides the cause of failure and how to fix it. The input is logs and screenshots, and the output is a report containing fix information.

[1119] Specific behavior:

[1120] The server parses the log file, detects the error message "Email address validation failed" and generates an advice "Fix the validation logic".

[1121] Step 8:

[1122] The user modifies the application based on the modification information from the server. The input is the modification information from the server, and the output is the modified application.

[1123] Specific behavior:

[1124] The user opens a code editor, modifies the "email_validation.py" file, and rebuilds it.

[1125] Step 9:

[1126] The server generates a new test scenario for the modified application and sends it to the terminal to re-execute the test. The input is the modified application information and prompt, and the output is the re-generated test scenario.

[1127] Specific behavior:

[1128] The server generates " / tests / scenario2.json" and sends it to the device, which runs the test again and collects new results.

[1129] Step 10:

[1130] The server collects and analyzes the retest results and generates the final report. The input is the retest results and the output is the final report.

[1131] Specific behavior:

[1132] The server generates " / reports / final_report.pdf" and provides the user with a download link.

[1133] (Application example 1)

[1134] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1135] Conventional product inspection systems rely on manual inspection, which often results in reduced inspection efficiency and accuracy. Furthermore, manual inspection is time-consuming and prone to human error, creating a need for improved quality control. Furthermore, existing automated inspection systems lack flexibility, making it difficult to accommodate a wide variety of products and inspection items. A new system is needed to resolve these issues, streamline product inspection within factories, and improve quality.

[1136] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1137] In this invention, the server includes means for loading a generative artificial intelligence model, means for generating a test scenario using the generative artificial intelligence model, means for transmitting the test scenario to an operation means, means for the operation means to operate an application according to the test scenario, means for collecting and transmitting test results to the server, means for analyzing the test results in the server and providing causes of failure and correction methods, means for retesting the corrected application, means for generating a product inspection scenario and having an in-factory device execute it, means for collecting and analyzing the inspection results and identifying defective products and providing points to be corrected, and means for generating and re-executing a new test scenario for reinspection, thereby enabling improvement in the efficiency and accuracy of product inspection.

[1138] A "generative artificial intelligence model" is a collection of algorithms that use AI techniques such as deep learning to process data for a specific task and make predictions or decisions.

[1139] A "test scenario" is a plan that defines specific operational procedures and input data to verify the operation and functionality of software or a system.

[1140] "Operation means" refers to devices or programs for operating applications or systems according to test scenarios.

[1141] "Test results" are information on operation confirmation such as success, failure, and error messages obtained when a test scenario is executed.

[1142] A "server" is a computer system that provides services such as data processing and storage over a network.

[1143] "Correction methods" are specific countermeasures or procedures for resolving problems or defects discovered during testing.

[1144] "Retesting" is the process of running tests again after a fix has been made to verify that the fix was successful.

[1145] "Factory equipment" refers to machines and robot systems used in factories to perform product inspections and production work.

[1146] "Inspection results" refers to data such as measurement values ​​and appearance check results obtained during product inspection.

[1147] A "defective product" is a product that does not meet standards or is determined to be defective during product inspection.

[1148] "Correction points" are specific locations and means for correcting identified defects or flaws based on the inspection results.

[1149] The present invention provides a system for automating product inspection during the production process, thereby improving efficiency and accuracy. Specific embodiments for carrying out the present invention will be described below.

[1150] The server first loads a generative artificial intelligence model. This generative AI model is based on deep learning technology and is optimized for product inspection tasks. The server also provides an interface for receiving product information and inspection items entered by factory staff. Product information includes model numbers, specifications, and details of inspection items, and generates test scenarios based on this information.

[1151] The generated test scenario is sent to the operating means in the factory. The operating means can be a robot or measuring device for product inspection. Specifically, it can be, for example, a laser tape measure for measuring dimensions or an inspection robot equipped with a high-resolution camera. These devices inspect the products according to the test scenario sent from the server.

[1152] Inspection results are sent from the operation device to the server. The server analyzes the received inspection results and identifies defective products and provides correction points. For example, if dimensions are outside the standard range or if surface scratches are detected in camera images, specific correction points will be presented. This allows factory staff to quickly carry out correction work.

[1153] After the corrections are complete, the server again uses the generative AI model to generate new test scenarios and conduct a retest to confirm whether the corrections were effective. The results of the retest are also sent to the server for final analysis.

[1154] To explain the effectiveness of this system, let's use a concrete example. For example, in the case of product inspection of smartphone cases, the prompt sentence to check the dimensions of the case and the presence of scratches on the surface is as follows:

[1155] "Measure the dimensions of your smartphone case and make sure it meets the standard values ​​(height: 150mm, width: 75mm, depth: 10mm). Also, check for scratches on the surface and report any defects."

[1156] Based on the prompts, the server generates a test scenario, and the operating device executes the test. The test results are collected on the server for detailed analysis. The test results include dimensional measurements and the presence or absence of surface scratches, and if the standards are not met, specific correction points are provided.

[1157] In this way, the system based on the present invention realizes efficient product inspection and improved quality.

[1158] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1159] Step 1:

[1160] The server loads the generative AI model, which is based on deep learning technology and optimized for product inspection tasks. Once the model has been loaded, the server prepares to accept product information and inspection items.

[1161] Input: Deep learning model file

[1162] Output: The loaded generative AI model

[1163] How it works: The server loads the generative AI model from file storage and deploys it in memory. This model includes neural networks for performing tasks such as object detection, dimension measurement, and visual inspection.

[1164] Step 2:

[1165] The user uses a tablet or PC to input product information (model number, specifications, inspection items) into the server, including details of the product to be inspected.

[1166] Input: Product information (model number, specifications, inspection items)

[1167] Output: Product information received by the server

[1168] How it works: The user enters product information into a dedicated interface and sends it to the server, which then prepares the information for input into the generative AI model.

[1169] Step 3:

[1170] The server uses the generative AI model to generate test scenarios, which include specific testing procedures for each product.

[1171] Input:Product Information

[1172] Output: Test scenario

[1173] How it works: The server inputs product information into the generative AI model and generates test scenarios, including procedures for measuring dimensions and inspecting surfaces. For example, "Measurement procedures to confirm a height of 150 mm, width of 75 mm, and depth of 10 mm."

[1174] Step 4:

[1175] The server sends the generated test scenario to the operating means, which is usually a robot or inspection device in a factory.

[1176] Input: Test scenario

[1177] Output: Test scenario sent to the control

[1178] Specific operation: The server transmits the test scenario to the operation means via the network, using a data transfer protocol to ensure reliable communication.

[1179] Step 5:

[1180] The operating means (for example, an inspection robot) performs inspections according to the test scenario, including dimensional measurements and surface inspections.

[1181] Input: Test scenario

[1182] Output: Inspection results (measurement values, abnormality detection information, etc.)

[1183] Specific operation: The operating means uses a laser tape measure and a high-resolution camera to inspect the product based on the test scenario. For example, the robot moves through each part of the product to measure its dimensions, and a high-resolution camera detects surface scratches.

[1184] Step 6:

[1185] The test results are transmitted from the operating means to the server, and the server receives them.

[1186] Input: Test results

[1187] Output: Inspection results aggregated on the server

[1188] Specific operation: The operating means sends the inspection results (e.g., size measurements and the presence or absence of surface abnormalities) to the server. The server stores this data as a log and prepares it for analysis.

[1189] Step 7:

[1190] The server analyzes the inspection results and identifies defects and provides correction points.

[1191] Input: Test results

[1192] Output: Analysis results (defective product identification, correction points)

[1193] Specific operation: The server uses machine learning algorithms to analyze the inspection results and identify defective products. It also generates specific instructions for correction, such as "This product's height exceeds the specified value, so it needs to be reprocessed."

[1194] Step 8:

[1195] The user receives instructions from the server and performs the work to correct the product.

[1196] Input: Correction instructions from the server

[1197] Output: Modified product

[1198] Specific operation: The user checks the correction instructions provided by the server and reprocesses or reinspects the product.

[1199] Step 9:

[1200] For the corrected product, new test scenarios are generated for retesting, and the product is retested.

[1201] Input: Revised product information

[1202] Output: Retest scenario, retest results

[1203] Specific operation: The server generates a new test scenario based on the corrected product information and has the operating means re-execute it. The retest results are sent to the server, and a final pass / fail decision is made.

[1204] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1205] This invention is a system that combines a generative artificial intelligence model and an emotion engine to automate application testing and take user emotions into consideration. Specific embodiments of this system are described below.

[1206] First, the server loads the generative AI model and emotion engine. The generative AI model is used to generate test scenarios, and the emotion engine recognizes user emotions in real time.

[1207] Next, the user provides information about the application under test to the server from their device. This information includes the application path, settings, and test scope. The server uses the received information to generate test scenarios using a generative AI model. For example, it creates test scenarios with various patterns that include input in the correct format and errors.

[1208] The server sends the generated test scenario to the terminal, which receives the scenario, launches the application, and performs the specified operations. As a concrete example, consider testing a user registration form. The terminal enters a username, email address, and password in the correct format and clicks the submit button. At the same time, the emotion engine obtains emotional data from the user's facial expressions and voice and records it in a log. If the emotional data indicates, for example, "dissatisfaction" or "confusion," this is collected as a test result.

[1209] As a result of the test, the device sends a log containing success or failure results, error messages, and emotional data to the server. The server analyzes the received log data and identifies the cause of the failure and how to correct it based on the user's emotional state. For example, it provides information such as "There was a problem with email address validation, which caused a lot of dissatisfaction."

[1210] The application is modified based on the modification information provided by the user. Once the modification is complete, the server generates a new test scenario, sends it to the device, and performs a retest. At this time, the emotion engine is again used to collect and analyze the overall test results, including emotion data. This allows the system to confirm that the modifications have been made appropriately and in a way that satisfies the user.

[1211] The above is a specific embodiment of the system based on the present invention, which combines AI and emotion recognition technology to achieve more user-friendly and effective test automation.

[1212] The processing flow will be explained below.

[1213] Step 1:

[1214] The server loads the generative AI model and emotion engine. Specifically, it expands the program files for the generative AI model and emotion engine into memory, making both models available for use.

[1215] Step 2:

[1216] The user enters information about the application to be tested into the server from the terminal, including the application path (e.g., " / path / to / application"), the test scope, and initial setting items.

[1217] Step 3:

[1218] The server generates test scenarios using a generative AI model based on the application information it receives. For example, for a user registration form, multiple scenarios for valid and invalid inputs are created in JSON format.

[1219] Step 4:

[1220] The server stores the generated test scenarios in a database, ensuring that the scenarios are used consistently in subsequent processes.

[1221] Step 5:

[1222] The server sends the saved test scenario to the device, which passes the test scenario to the device via an API endpoint.

[1223] Step 6:

[1224] The device loads the test scenario received from the server and launches the application. For example, it opens the application's executable file and makes the UI operable.

[1225] Step 7:

[1226] The device automatically executes UI operations according to the test scenario. At the same time, the emotion engine is activated and collects emotional data in real time from the user's facial expressions and voice. For example, if the user shows signs of "confusion" or "dissatisfaction" while entering their username, email address, and password, and then performing the send operation, this data is also recorded.

[1227] Step 8:

[1228] The device collects test results and emotional data, including successful and unsuccessful operations, displayed error messages, and emotional data.

[1229] Step 9:

[1230] The device sends the collected test results and emotion data to the server, which includes compiling the test results and emotion data in JSON format and sending them to the server's receiving API.

[1231] Step 10:

[1232] The server analyzes the test results and emotion data it receives. It uses a generative AI model to identify the specific cause of the failure and extracts the user's perceived issues based on the emotion data. For example, if there is a problem with email address validation, it generates a message saying, "The validation logic needs to be corrected."

[1233] Step 11:

[1234] The server notifies the user of the proposed correction method. Possible notification methods include displaying the information on the dashboard or sending an email. Specifically, it presents in detail the areas that need to be corrected and how to do so.

[1235] Step 12:

[1236] The user modifies the application based on the provided fixes, modifies the code in the development environment, and builds the modified version.

[1237] Step 13:

[1238] The user notifies the server that the modification is complete, for example, by sending the modified application path to the server.

[1239] Step 14:

[1240] The server generates a new test scenario again and sends it to the terminal. The retest runs a test to check whether the corrections have been made properly.

[1241] Step 15:

[1242] The device will then conduct the retest and send the results to the server. The device will also use the emotion engine to collect emotional data during the retest and send the results, including this data, to the server.

[1243] Step 16:

[1244] The server finally analyzes all test results and emotion data and reports back to the user, ensuring that the corrections were made properly and providing the user with a final report.

[1245] The above are the specific processing steps of the system.

[1246] Example 2

[1247] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1248] Traditional application testing is often done manually, requiring a large amount of work and time. Furthermore, there is no system for evaluating test results that takes user emotions into account, making it difficult to identify problems based on user experience. Therefore, there is a need for more efficient, user-centric testing by introducing test automation and a system that reflects user emotions in real time.

[1249] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for loading a generative AI model, means for loading an emotion engine, means for generating a test scenario using the generative AI model, means for transmitting the generated test scenario to a terminal, means for the terminal to run an application according to the test scenario, means for acquiring user emotion data using the emotion engine, means for collecting test results and emotion data and transmitting them to the server, means for analyzing the test results and emotion data in the server and providing causes of failure and correction methods, and means for retesting the corrected application. This enables test automation and comprehensive test evaluation that takes user emotions into consideration.

[1250] "Generative artificial intelligence model" refers to an algorithm or system that uses artificial intelligence to automatically generate test scenarios and other artifacts.

[1251] An "emotion engine" refers to a technology or system that analyzes a user's facial expressions and voice to recognize and record emotions in real time.

[1252] A "test scenario" refers to a plan containing a series of steps or procedures used in performing operational testing of an application.

[1253] "Terminal" refers to a device that receives instructions from a server, executes applications, and obtains operation and test results.

[1254] "Server" refers to the central computer system that loads the generative AI models and emotion engines, generates and transmits test scenarios, and analyzes collected data.

[1255] "User emotion data" refers to data indicating the user's emotional state obtained using an emotion engine.

[1256] "Test results" refers to the result information such as success / failure and error messages when testing an application.

[1257] "Correction method" refers to the specific means or steps to resolve application or system problems based on the analysis results.

[1258] "Retesting" refers to the process of testing an application again after it has been modified to ensure that the modifications function correctly.

[1259] MODE FOR CARRYING OUT THE INVENTION

[1260] This invention is a system that combines a generative artificial intelligence model and an emotion engine to automate application testing and take user emotions into consideration. Specific embodiments of this system are described below.

[1261] Hardware and software used

[1262] Server: The central computer system of the system, which loads the generative AI model and emotion engine, generates and transmits test scenarios, and analyzes the collected data.

[1263] Example of use: Server machines with high performance processors, such as Linux servers.

[1264] Terminal: A device that receives instructions from the server, executes applications, and obtains operation and test results.

[1265] Example of use: A typical personal computer (Windows or macOS) or a virtual machine.

[1266] Generative AI models: Artificial intelligence algorithms that automatically generate test scenarios and other artifacts.

[1267] Usage example: Using "GPT-3" and "BERT".

[1268] Emotion engine: Technology that analyzes the user's facial expressions and voice to recognize and record emotions in real time.

[1269] Usage examples: "Affectiva" and "Microsoft Azure Emotion API".

[1270] Specific examples of processing

[1271] First, the server loads the generative AI model and emotion engine. The generative AI model is used to generate test scenarios, and the emotion engine is responsible for recognizing user emotions in real time.

[1272] Next, the user uses the terminal to provide information about the application under test to the server, including the application path, settings, test scope, etc.

[1273] The server then uses the received information to generate appropriate test scenarios using a generative AI model. For example, it sends a prompt to the AI, such as "Please generate a test scenario for when a user enters an incorrect format in the email address field," and uses the response to create the scenario.

[1274] The server sends the generated test scenario to the terminal, and the terminal launches the application according to the scenario and performs specific operations. For example, run " / path / to / app", enter "testuser" in the username field, "wrong_email_format" in the email address field, and "password123" in the password field, and click the submit button.

[1275] At the same time, the emotion engine analyzes the user's facial expressions and voice to obtain emotional data. For example, if the user makes a "confused" expression, this is recorded as emotional data in the log. The device then saves this in a log file.

[1276] Once the test is complete, the device sends a log containing the test success / failure, error messages, and emotion data to the server. The server analyzes the received log data and identifies the cause of the failure and how to fix it. For example, it may conclude that "email address validation errors caused many users to feel confused."

[1277] The user modifies the application based on the analysis results provided by the server. For example, the user makes specific modifications such as "strengthening the validation function for email addresses."

[1278] Finally, the server generates a new test scenario using the generative AI model again and sends it to the device for retesting. This retest also uses the emotion engine, and collects and analyzes the overall test results.

[1279] This allows for comprehensive testing that takes user emotions into account, resulting in the development of higher quality, more user-friendly applications.

[1280] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1281] Step 1:

[1282] The server loads the generative AI model and emotion engine. First, the server loads the generative AI model and emotion engine from existing data storage into memory and prepares them. GPT-3 or BERT is used as the generative AI model, and Affectiva or Microsoft Azure Emotion API is used as the emotion engine. This enables the server to generate test scenarios and recognize user emotions. The file paths of the generative AI model and emotion engine are required as input, and the loaded model and engine are obtained as output.

[1283] Step 2:

[1284] The user uses a terminal to enter information about the application under test and provides it to the server. The user enters the application path, settings, test scope, etc. into the terminal and clicks the send button. The terminal sends the entered information to the server. The application path, setting file, etc. are required as input, and this information is obtained as output to be sent to the server.

[1285] Step 3:

[1286] The server generates a test scenario using the generative AI model. Based on the application information provided by the user, the server inputs a prompt statement to the generative AI model to generate an appropriate test scenario. For example, the server uses the prompt statement "Please generate a test scenario for when the user enters an incorrect format in the email address field" to obtain the generated scenario. The input of this step is the prompt statement and application information, and the output is the generated test scenario.

[1287] Step 4:

[1288] The server sends the generated test scenario to the terminal. The server packages the generated test scenario and sends it to the terminal. The generated test scenario is required as input, and the scenario sent to the terminal is obtained as output.

[1289] Step 5:

[1290] The terminal launches the application according to the received test scenario and performs the specified operation. The terminal first launches the application from the specified path, and then performs the input operations according to the scenario. Specifically, it enters "testuser" in the username field, "wrong_email_format" in the email address field, and "password123" in the password field, and then clicks the send button. The input for this step is the test scenario and application information, and the output is the result of the executed operation.

[1291] Step 6:

[1292] The emotion engine analyzes the user's facial expressions and voice to obtain emotional data. The device uses the emotion engine to analyze the user's real-time facial expressions and voice to collect emotional data. If the emotional data indicates "confusion" or "dissatisfaction," this is recorded in a log. The user's facial expressions and voice data are required as input, and the collected emotional data is obtained as output.

[1293] Step 7:

[1294] The device sends a log containing the success or failure of the test, error messages, and emotional data to the server. The device saves the executed operation results and emotional data together in a log file and sends the log to the server. The operation results and emotional data are required as input, and the log data sent to the server is obtained as output.

[1295] Step 8:

[1296] The server analyzes the received log data to identify the cause of the failure and how to fix it. The server analyzes the log data to identify the problem and its cause. For example, it may conclude that "email address validation errors caused frequent user confusion." The input to this step is the log data, and the output is the analysis results and how to fix it.

[1297] Step 9:

[1298] The user modifies the application based on the analysis results provided by the server. The user then modifies the application code according to the instructions provided by the server. For example, the user makes a specific modification such as "strengthening the email address validation function." The input requires the analysis results and instructions, and the output is the modified application.

[1299] Step 10:

[1300] The server then uses the generative AI model to generate a new test scenario, sends it to the device, and runs the test again. All steps are repeated to verify that the modifications function properly. The inputs are the modified application information and a new prompt, and the output is the retest result.

[1301] (Application example 2)

[1302] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1303] Traditional application testing systems focused on test automation but failed to consider user emotions and usage. As a result, while testing could detect technical defects, it was unable to adequately address user experience improvements. Furthermore, the inability to reflect user confusion and dissatisfaction in real time limited UI / UX optimization.

[1304] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for loading a generative AI model, a means for generating a test scenario, and a means for analyzing test results and emotion data. This makes it possible to collect and analyze user emotion data in real time and test and improve applications taking user experience into consideration.

[1305] A "generative artificial intelligence model" is a type of artificial intelligence used to generate test scenarios, and has the ability to automatically create appropriate scenarios based on user input and operations.

[1306] A "test scenario" refers to a set of pre-defined operational procedures or conditions for verifying the application's functions and user interface.

[1307] "Device" refers to an electronic device such as a computer, smartphone, or tablet on which the application being tested runs.

[1308] "User emotion data" refers to emotion information obtained in real time from the user's facial expressions and voice, and includes emotions such as "satisfaction," "dissatisfaction," and "confusion."

[1309] "Log" refers to data that records the application's operating status, user operation history, emotional data, etc.

[1310] "Server" refers to a centralized computer system that generates test scenarios, analyzes the results, collects and stores emotional data, and so on.

[1311] The "cause of failure" refers to the cause of an application not operating normally.

[1312] "Correction method" refers to the specific methods and procedures for solving the discovered cause of failure.

[1313] "Retesting" refers to the process of verifying the modified application again according to the test scenarios to confirm that the improvements have been properly implemented.

[1314] This invention is a system for automating application testing and taking user emotions into account. Specifically, it uses a system that combines a generative artificial intelligence model and an emotion engine.

[1315] First, the server loads a generative AI model. This model generates test scenarios based on user input and operation history. For example, it generates a test scenario for the process of searching for a specific product and adding it to a cart.

[1316] The server then loads the emotion engine, which has the ability to acquire emotion data in real time from the user's facial expressions and voice. The data is collected from devices such as smartphones and tablets.

[1317] When a user provides information about the application under test to the server from their device, the server uses a generative artificial intelligence model to generate a test scenario and sends it to the device. For example, a scenario in which a user searches for a product in a virtual store and adds it to a cart is considered.

[1318] The device operates the application according to the test scenario sent from the server. During the test, the emotion engine collects emotional data in real time from the user's facial expressions and voice. The device records these test results and emotional data in a log and sends them to the server.

[1319] The server analyzes the received log data to identify the cause of the application failure and how to fix it. For example, if the user expresses "confusion" or "dissatisfaction," the server will suggest improvements to the UI / UX based on that emotion data.

[1320] The modified application is then retested, again using the emotion engine to verify whether user satisfaction has improved. To ensure the modified application behaves as expected, retest scenarios are also generated using the generative artificial intelligence model.

[1321] This system allows for comprehensive application testing that takes into account user emotions, rather than simply fixing technical bugs, and allows for application improvements that satisfy users.

[1322] Examples:

[1323] For example, if a user expresses "confusion" in the process of searching for a specific product in a virtual store app and adding it to the cart, a log of this emotion can be collected and suggestions for improving the UI / UX can be generated.An example of a prompt to input to the generative AI model could be, "Generate a test scenario for the virtual store application where the user searches for a specific product and adds it to the shopping cart. Include various steps and possible user inputs, including both successful and erroneous entries."

[1324] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1325] Step 1:

[1326] The server loads the generative AI model. The input is the path of the generative AI model, and the output is the loaded model object. This model object is used to prepare for generating test scenarios.

[1327] Step 2:

[1328] The server loads the emotion engine. The input is the path to the emotion engine, and the output is the loaded engine object. This engine object is used to prepare for collecting user emotion data in real time. Specifically, the parameters of the emotion recognition model are set.

[1329] Step 3:

[1330] The user provides information about the application under test to the server from their device. The input is the information about the application under test (application path, settings, test scope), and the output is the information object received by the server. The server uses this to input prompts to the generative AI model.

[1331] Step 4:

[1332] The server generates test scenarios using a generative AI model. The input is the application information sent by the user, and the output is the generated test scenario. The generative AI model outputs a specific test scenario using a prompt statement.

[1333] Step 5:

[1334] The server sends the generated test scenario to the terminal. The input is the generated test scenario, and the output is confirmation data of the transmission result. The server verifies whether the test scenario was sent correctly to the terminal.

[1335] Step 6:

[1336] The terminal operates the application according to the test scenario. The input is the test scenario received from the server, and the output is the test result. Specifically, the terminal automatically executes the specified actions and records the operation log.

[1337] Step 7:

[1338] The device collects the user's emotional data in real time. The input is real-time data from the camera and microphone, and the output is emotional data. The device inputs this data into the emotion engine and obtains the emotion recognition results.

[1339] Step 8:

[1340] The device sends a log containing the test results and emotional data to the server. The input is the log containing the test results and emotional data, and the output is a transmission confirmation data sent to the server. The server performs analysis based on this.

[1341] Step 9:

[1342] The server analyzes the received log data and identifies the cause of the application failure and how to fix it. The input is the log data sent from the device, and the output is the analysis results including how to fix it. This makes it clear what specific points need to be fixed.

[1343] Step 10:

[1344] The server generates a new test scenario for retesting the modified application and sends it to the terminal. The input is the modified application information, and the output is the new test scenario. The same process is repeated to check the user's satisfaction.

[1345] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1346] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1347] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1348] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1349] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1350] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1351] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1352] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1353] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1354] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1355] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1356] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1357] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1358] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1359] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1360] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1361] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1362] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1363] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1364] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1365] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1366] The following is further disclosed regarding the above embodiment.

[1367] (Claim 1)

[1368] means for loading a generative artificial intelligence model;

[1369] means for generating test scenarios using a generative artificial intelligence model;

[1370] A means for transmitting a test scenario to a terminal;

[1371] a means for the terminal to operate the application according to the test scenario;

[1372] means for collecting and transmitting test results to a server;

[1373] A means for analyzing the test results on the server and providing the cause of failure and a method for correcting the failure;

[1374] a means of retesting the modified application; and

[1375] A system including:

[1376] (Claim 2)

[1377] The system of claim 1 , wherein the test scenario includes a user interface operation.

[1378] (Claim 3)

[1379] 10. The system of claim 1, further comprising means for storing test results received from the terminal in a database.

[1380] "Example 1"

[1381] (Claim 1)

[1382] means for loading the generated artificial intelligence model using the generating means;

[1383] a means for a user to input application information to be tested;

[1384] means for generating test scenarios using a generative artificial intelligence model;

[1385] A means for transmitting a test scenario to a terminal;

[1386] A means for the terminal to launch and operate the application according to the test scenario;

[1387] a means for testing a variety of input values ​​to test both normal and abnormal operation;

[1388] means for collecting and transmitting test results to a server;

[1389] A means for analyzing the test results on the server and providing the cause of failure and a method for correcting the failure;

[1390] means for retesting the modified application based on the modification information;

[1391] A means of reporting and analyzing the final results through retesting and generating reports;

[1392] A system including:

[1393] (Claim 2)

[1394] The system of claim 1 , wherein the test scenario includes a user interface operation.

[1395] (Claim 3)

[1396] 10. The system of claim 1, further comprising means for storing test results received from the terminal in a database.

[1397] "Application Example 1"

[1398] (Claim 1)

[1399] means for loading a generative artificial intelligence model;

[1400] means for generating test scenarios using a generative artificial intelligence model;

[1401] means for transmitting the test scenario to the operation means;

[1402] an operating means for operating an application according to a test scenario;

[1403] means for collecting and transmitting test results to a server;

[1404] A means for analyzing the test results on the server and providing the cause of failure and a method for correcting the failure;

[1405] a means of retesting the modified application; and

[1406] means for generating a product inspection scenario and causing an in-factory device to execute the scenario;

[1407] A means for collecting and analyzing the test results and providing identification and correction of defects;

[1408] A means for generating and re-executing new test scenarios for retesting;

[1409] A system including:

[1410] (Claim 2)

[1411] The system of claim 1 , wherein the test scenario includes a user interface operation.

[1412] (Claim 3)

[1413] 10. The system of claim 1, further comprising means for storing test results received from the operating means in a database.

[1414] "Example 2: Combining Emotion Engines"

[1415] (Claim 1)

[1416] means for loading a generative artificial intelligence model;

[1417] a means for loading the emotion engine;

[1418] means for generating test scenarios using a generative artificial intelligence model;

[1419] means for transmitting the generated test scenario to a terminal;

[1420] a means for causing the terminal to run an application according to a test scenario;

[1421] means for acquiring user emotion data using an emotion engine;

[1422] means for collecting test results and emotion data and transmitting them to a server;

[1423] A means for analyzing the test results and emotion data on the server and providing the cause of failure and a method for correcting the failure;

[1424] A means to retest the modified application; and

[1425] A system including:

[1426] (Claim 2)

[1427] The system of claim 1 , wherein the test scenario includes a user interface operation.

[1428] (Claim 3)

[1429] 10. The system of claim 1, further comprising means for storing test results and emotion data received from the terminal in a database.

[1430] "Application example 2 when combining emotion engines"

[1431] (Claim 1)

[1432] means for loading a generative artificial intelligence model;

[1433] means for generating test scenarios using a generative artificial intelligence model;

[1434] A means for transmitting a test scenario to a terminal;

[1435] a means for the terminal to operate the application according to the test scenario;

[1436] a means for collecting user emotion data in real time;

[1437] means for transmitting a log containing test results and emotion data to a server;

[1438] A means for analyzing the test results and emotion data on the server and providing the cause of failure and a method for correcting the failure;

[1439] a means of retesting the modified application; and

[1440] A system including:

[1441] (Claim 2)

[1442] The system of claim 1 , wherein the test scenario includes a user interface operation.

[1443] (Claim 3)

[1444] 10. The system of claim 1, further comprising means for storing test results and emotion data received from the terminal in a database. [Explanation of symbols]

[1445] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for loading a generative artificial intelligence model; means for generating test scenarios using a generative artificial intelligence model; A means for transmitting a test scenario to a terminal; a means for the terminal to operate the application according to the test scenario; means for collecting and transmitting test results to a server; A means for analyzing the test results on the server and providing the cause of failure and a method for correcting the failure; a means of retesting the modified application; and A system including:

2. The system of claim 1 , wherein the test scenario includes a user interface operation.

3. 2. The system of claim 1, further comprising means for storing test results received from the terminal in a database.

Images