System

Abstract

A system is provided.SOLUTION: This system includes a means for inputting a specification, a means for analyzing the specification and extracting requirements, a means for generating a test item on the basis of the extracted requirements, a means for executing a test according to the generated test item, a means for recording and analyzing a test result, and a means for generating a correction plan when a failure is detected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In the current software development process, designing and implementing a complete test plan is a very labor-intensive process. In particular, understanding the contents of the specifications and setting appropriate test items requires specialized knowledge, placing a heavy burden on developers and personnel with little testing experience. Furthermore, when defects are discovered, the process of correcting them is also time-consuming, increasing development man-hours and labor costs. As a result, the quality of testing can decline, leading to insufficient software quality assurance. The present invention aims to solve these problems and improve the efficiency and quality of testing operations. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for inputting specifications, a means for analyzing specifications and extracting requirements, a means for generating test cases based on the extracted requirements, a means for executing tests according to the generated test cases, a means for recording and analyzing test results, and a means for generating correction proposals when defects are detected. This enables the automation of the entire process from test design to implementation and defect correction, thereby improving the efficiency and quality of testing operations. The system also includes a rule set for automatically generating test cases based on the extracted requirements and a means for collecting test results and classifying test cases into successful and failed. This prevents omissions in testing and reduces labor hours and personnel costs.

[0006] A "specification" is a document that describes in detail the functions, requirements, design, etc. of software or a system.

[0007] "Requirements" indicate the conditions such as functions, performance, and constraints that software or systems must achieve.

[0008] "Test items" refer to the specific content of tests to be performed on specific functions or requirements of software or a system.

[0009] A "test case" is a scenario that includes specific inputs, environments, procedures, expected results, and other specific developments that are set based on test items.

[0010] "Means for inputting specifications" refers to the interface or method by which a user uploads or inputs specifications into the system.

[0011] "Means for analyzing specifications and extracting requirements" refers to techniques or methods that allow a system to analyze the contents of specifications and automatically extract requirements from them.

[0012] "Means for generating test cases" refers to techniques or methods for automatically generating specific test cases based on extracted requirements.

[0013] "Means for executing tests" refers to techniques and methods for automatically executing tests on software or systems according to the generated test items.

[0014] "Means for recording and analyzing test results" means the techniques and methods for recording the results of tests performed and analyzing the data.

[0015] A "means for generating a correction proposal" is a technique or method for automatically generating a correction proposal when a defect is detected based on the results of the test.

[0016] A "rule set" is a set of rules and guidelines for automatically generating test cases based on extracted requirements.

[0017] A "means for classifying test cases as successful and unsuccessful" is a technique or method for classifying test cases as successful or unsuccessful based on the results of the tests executed. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] System configuration

[0040] The system of the present invention automates a series of processes, from inputting specifications to analysis, generating test cases, executing tests, recording results, and proposing fixes for defects. The system consists of the following main components:

[0041] 1. Means of inputting specifications

[0042] 2. A method for analyzing specifications and extracting requirements

[0043] 3. Methods for generating test cases

[0044] 4. How to run the test

[0045] 5. Means for recording and analyzing test results

[0046] 6. Means of generating revisions

[0047] Specific Examples

[0048] Specification input and analysis

[0049] The server provides an interface to receive specifications from users. Users upload specifications in PDF or other text format. The server uses a text analysis engine to analyze the content of the specifications and extract requirements from them.

[0050] As a concrete example, suppose a user uploads a UI test specification for an Internet service. The server automatically extracts functional requirements and test scope from the specification. In the process, it uses natural language processing technology to understand the context of the specification and identify important requirements.

[0051] Test case generation

[0052] Based on the extracted requirements, the server automatically generates test cases, each of which includes specific test scenarios and test steps for a specific function or requirement.

[0053] For example, if requirements related to user registration functionality are extracted, the server generates test items such as "user registration success test" and "user registration failure test," which include specific steps such as filling in a form, clicking the submit button, and checking the success message.

[0054] Running the tests

[0055] The server executes the tests according to the generated test items. The server uses an automated test tool (e.g., Selenium) to operate the web browser and executes the tests. The execution results of each step are recorded and compared with the expected results.

[0056] For example, it automatically enters a name, email address, and password into a form, clicks the submit button, and checks whether a success message is displayed correctly. During this process, it records detailed logs for later reference.

[0057] Recording and analyzing test results

[0058] The server collects the results of the executed tests and classifies them as successes and failures. Based on the test results, the server analyzes the success rate of the tests, the type and frequency of failures, etc.

[0059] For example, if the test for user registration function is successful, the result is recorded as "success" and reflected in the analysis data. If the test fails, the detailed reason for the failure (e.g. error message, thrown exception, etc.) is recorded.

[0060] Generate correction suggestions

[0061] If a defect is detected during testing, the server generates a fix for the defect, analyzes the cause of the defect, and proposes specific fixes and code changes.

[0062] For example, if there is a problem with the validation of input data in the user registration function, the server will generate a correction suggestion called "Strengthen validation of user input data" and provide specific code examples and guidelines.

[0063] summary

[0064] By integrating the above components and processing steps, the system of the present invention automates all processes, from inputting specifications to analysis, generating test cases, executing tests, recording results, and proposing defect fixes. This system improves the efficiency of testing operations and strengthens quality assurance. It is an extremely useful tool, especially for developers and engineers with little testing experience.

[0065] The processing flow will be explained below.

[0066] Step 1:

[0067] The user uploads the specification.

[0068] Through the interface, users upload specifications for their Internet services in PDF or text format, which are then sent to a specific endpoint on the server.

[0069] Step 2:

[0070] The server analyzes the specification and extracts the requirements.

[0071] The server analyzes the received specifications using a text analysis engine (e.g., Textract). From the analysis results, it understands the content of the text and extracts requirements such as software functional requirements, user requirements, and terms of use.

[0072] Step 3:

[0073] The server generates test cases based on the extracted requirements.

[0074] The server automatically generates specific test items based on the requirements extracted from the specifications. For example, if the requirements for a user registration function are included, multiple test scenarios will be generated, such as a registration success test and an error test.

[0075] Step 4:

[0076] The server runs the test items.

[0077] The server uses an automated testing tool (e.g., Selenium) to run tests according to the generated test items, operating the web browser and performing specific operations such as filling out forms and clicking buttons, and recording the results of each operation and comparing them with the expected results.

[0078] Step 5:

[0079] The server records and analyzes the test results.

[0080] The server collects and records the results of the executed tests, categorizes successful and failed test cases, and generates detailed logs and analysis data, such as success rates, types of errors, and frequency of failures.

[0081] Step 6:

[0082] The server generates suggested fixes when defects are detected.

[0083] If a defect is detected, the server analyzes the cause and automatically generates a fix, generating a document with specific suggestions and instructions for code changes to help resolve the problem.

[0084] In this way, a system is realized that, through cooperation between the server and the user, handles everything from inputting specifications to conducting tests and correcting defects.

[0085] Example 1

[0086] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] Traditional software testing processes require a lot of time and effort, including manual requirements extraction, test case creation, and manual test execution. This reduces testing efficiency and increases the likelihood of human error. It also makes it difficult to quickly detect defects and propose fixes. Therefore, to solve these problems, a new system is needed to automate the entire process, from analyzing specifications to running tests, analyzing results, and proposing fixes.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0089] In this invention, the server includes means for inputting specifications as data, means for analyzing the specifications and automatically extracting requirements, means for automatically generating test items based on the extracted requirements, means for executing tests using an automated test tool in accordance with the generated test items, means for recording test results and classifying and analyzing successes and failures, and means for generating and providing correction suggestions when defects are detected. This makes it possible to automate the entire software testing process and significantly improve efficiency and reliability.

[0090] A "specification" is a document that describes in detail the requirements, functions, design, etc. of software or a system.

[0091] "Means of data input" refers to the methods and techniques used to input data in the form of digital files into the system.

[0092] "Analysis" is the process of examining given data in detail and clarifying its structure and content.

[0093] "Requirements" refer to the conditions or specifications that software or a system must meet.

[0094] "Automatic generation" refers to the system generating test items and data based on certain algorithms and rules without manual intervention.

[0095] A "test item" is a specific test case or scenario that verifies a particular function or requirement of software or a system.

[0096] An "automated testing tool" is software that uses scripts and programs to run tests without human intervention.

[0097] "Recording" means saving the results of the executed tests and information about each step.

[0098] "Classification" is the act of grouping collected data based on specific criteria.

[0099] "Analysis" is the process of using collected data to interpret their meaning and draw conclusions.

[0100] A "defect" is a problem or defect that prevents software or a system from functioning properly.

[0101] A "proposed fix" is a specific method or proposal for solving a detected defect.

[0102] MODE FOR CARRYING OUT THE INVENTION

[0103] System configuration

[0104] The system of the present invention automates a series of processes, from inputting specifications to analysis, generating test items, executing tests, recording results, and proposing defect corrections. This system is composed of multiple components, including a server, terminals, and users.

[0105] Specification input and analysis

[0106] The server provides an interface for receiving specifications from users. Users upload specifications in PDF or other text format. After receiving the specifications, the server uses a text analysis engine (e.g., NLTK or spaCy) to analyze the content of the specifications and extract requirements from them. As a specific example, if a user uploads a specification for UI testing of an Internet service, the server analyzes it and automatically extracts functional requirements and test scope. During this process, natural language processing technology is used to understand the context of the specifications and identify important requirements.

[0107] Test case generation

[0108] Based on the extracted requirements, the server automatically generates test items. Each test item includes specific test scenarios and test steps for specific functions and requirements. For example, if requirements related to user registration functions are extracted, the server generates test items such as "User Registration Success Test" and "User Registration Failure Test." These test items include specific steps such as filling out a form, clicking a submit button, and checking for a success message.

[0109] Running the tests

[0110] The server executes the tests according to the generated test items. Using an automated testing tool (for example, Selenium), the tests are executed while operating the web browser. The results of each step are recorded and compared with the expected results. For example, the server can automatically enter a name, email address, and password into a form, click the submit button, and check whether a success message is displayed correctly. During this process, the server records detailed logs that can be referenced later.

[0111] Recording and analyzing test results

[0112] The server collects the results of the executed tests and classifies them as successes or failures. Based on the test results, the server analyzes the success rate, type of failure, frequency, etc. For example, if the test of the user registration function is successful, the result is recorded as "success" and reflected in the analysis data. If the test fails, the detailed reason for the failure (e.g., error message, exception occurred, etc.) is recorded.

[0113] Generate correction suggestions

[0114] If a defect is detected during testing, the server generates a correction proposal to address the defect. The server analyzes the cause of the defect and proposes specific correction methods and code changes. For example, if there is a problem with the input data validation in the user registration function, the server generates a correction proposal called "Strengthen validation of user input data" and provides specific code examples and guidelines.

[0115] Specific prompt examples

[0116] When a user uploads a specification saying they want to "test the new user registration function," the following prompt is input to the generative AI model:

[0117] "For the new user registration function, please generate test cases based on the following requirements:

[0118] 1. Enter your username, email address, and password, then click the Register button to display a success message.

[0119] 2. Display an error message if the user name is not entered.

[0120] 3. If the email address is not in the correct format, display an error message.

[0121] By inputting this prompt into a generative AI model, the model will automatically generate test items based on the requirements.

[0122] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0123] Step 1:

[0124] The user uploads the specification as data. Specifically, the user opens a web browser, accesses the specified form, selects the specification file (e.g., PDF), and clicks the upload button. This input process causes the server to obtain the specification file.

[0125] Step 2:

[0126] The server receives the uploaded specification file and temporarily stores it. Next, it uses a text analysis engine (e.g., NLTK or spaCy) to convert the content of the specification into text format and begins analyzing it. The server tokenizes each sentence and extracts target keywords and structured information. Based on the results of this analysis, a list of important requirements is extracted from the specification.

[0127] Step 3:

[0128] The server automatically generates test cases based on the extracted requirements. It uses a generative AI model or algorithm to analyze the input requirements data and generate specific test scenarios and test steps. For example, if requirements related to user registration functionality are extracted, test cases such as "User Registration Success Test" and "User Registration Failure Test" are generated. The output includes test scenarios and corresponding detailed steps.

[0129] Step 4:

[0130] The server executes the tests according to the generated test items. Specifically, the server uses an automated testing tool (e.g., Selenium) to execute test scenarios while operating a web browser. At each test step, the server automatically inputs forms and clicks buttons, and compares the expected results with the actual results. The test execution results based on this input and output are recorded as a log.

[0131] Step 5:

[0132] The server records and analyzes the results of the executed tests. The server logs the results of each test step and classifies them as success or failure. For example, if user registration is completed successfully, it is recorded as a "success," and if an error occurs, it is recorded as a "failure." This analysis data includes the success rate, the frequency of failures, details of error messages, etc.

[0133] Step 6:

[0134] If a defect is detected during testing, the server generates a correction proposal to address the defect. The server analyzes the cause of the defect and automatically generates specific correction methods and code change suggestions. For example, if there is a problem with input data validation, the server generates a correction proposal called "Strengthen input data validation" and provides specific code examples and guidelines. This correction proposal is provided to the user along with the test results.

[0135] (Application example 1)

[0136] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0137] Because software for autonomous vehicles requires safety and precision, rigorous testing is necessary when updating software or adding new features. However, traditional manual testing is burdensome and inefficient. Furthermore, the wide scope of testing can lead to human error or insufficient testing. This can reduce the reliability of the software and potentially cause safety issues. This invention was developed to solve these problems, and aims to fully automate the software testing process for autonomous vehicles, improving efficiency and accuracy.

[0138] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0139] In this invention, the server includes means for inputting specifications, means for analyzing specifications and extracting requirements, means for generating test items based on the extracted requirements, means for executing tests in accordance with the generated test items, means for recording and analyzing test results, means for generating a correction plan when a defect is detected, means for uploading software specifications for autonomous vehicles, means for automatically extracting requirements from specifications, means for automatically generating and executing test items, and means for analyzing the results in detail and, if necessary, proposing a correction plan for the defect. This automates the entire process of extracting requirements from specifications, generating test items, executing them, analyzing the results, and proposing a correction plan for the defect, thereby enabling the efficiency and accuracy of software testing for autonomous vehicles to be improved.

[0140] A "specification" is a document that describes in detail the functions and requirements of software or a system.

[0141] "Requirements" are the conditions and functions that a system or software must satisfy, extracted from the specifications.

[0142] "Test items" are specific test scenarios or steps that are generated based on extracted requirements and are used to verify specific functions or conditions.

[0143] An "autonomous vehicle" is a vehicle that completely automates driving operations without human intervention, and automatically makes decisions and operates in accordance with the surrounding situation.

[0144] "Upload" refers to the operation of transferring files or data from a local environment to a server.

[0145] "Extraction" refers to the process of selecting specific information or requirements from a document such as a specification.

[0146] "Generation" means creating new test items or scenarios from given data and conditions.

[0147] "Execution" refers to the process of automatically performing tests according to the generated test items and obtaining the results.

[0148] "Recording and analysis" means saving the test results and analyzing them in detail to identify the causes and patterns of success and failure.

[0149] A "fix" is a specific improvement plan or code modification to resolve the cause of a test failure or defect.

[0150] A "proposal" is a way of presenting a solution or improvement to a problem and encouraging its implementation.

[0151] The system of this invention can automatically perform operations from inputting specifications to analyzing them, generating test items, executing tests, recording results, and proposing defect fixes. It can be applied to software updates and adding new functions to autonomous vehicles. This system can improve the reliability and safety of software. The detailed configuration and operation of the system are described below.

[0152] System configuration

[0153] The system mainly consists of the following components:

[0154] 1. Component for inputting specifications

[0155] 2. A component that analyzes specifications and extracts requirements

[0156] 3. A component that generates test cases based on the extracted requirements

[0157] 4. Components that execute tests according to the test items

[0158] 5. Component for recording and analyzing test results

[0159] 6. A component that generates suggested fixes when defects are detected

[0160] Specification input and analysis

[0161] The server provides an interface to receive specifications from users. Users upload specifications in PDF or other text format. The server extracts the text from the specifications using pdfminer and parses and extracts requirements using a natural language processing library. This requirements extraction utilizes AI models and generative AI models.

[0162] Test case generation

[0163] Based on the extracted requirements, the server automatically generates test cases. The generated test cases are highly accurate because they create specific test scenarios for specific functions and operations. In this process, test cases are generated using the generated rule set.

[0164] Running the tests

[0165] The server runs automated tests using Selenium according to the test items. The server simulates each operation and compares the expected results with the actual results. During this process, the execution results of each step are recorded and detailed logs are collected.

[0166] Recording and analyzing test results

[0167] The server collects the executed test results and classifies the test cases as successful and failed. The recorded results are analyzed to determine the success rate and causes of failures. This analysis is done automatically using a generative AI model.

[0168] Generate correction suggestions

[0169] If a defect is detected, the server generates a fix for the defect, suggesting specific fixes and code changes and providing necessary feedback. These fix suggestions are also generated using natural language processing and AI models.

[0170] Specific examples

[0171] For example, in the case of a new navigation system update for a self-driving vehicle, the system might receive and act on the following prompt:

[0172] Example prompt sentence:

[0173] The specification states:

[0174] The new navigation system automatically calculates and displays the optimal route when the user enters their destination.

[0175] It uses cameras and sensors to perceive the surrounding terrain in real time and dynamically update the route.

[0176] It provides location information for charging stations and suggests the best charging point based on the remaining charge.

[0177] Based on this specification, please implement a program that generates test cases and automates each test.

[0178] This system will improve the efficiency of software testing for autonomous vehicles and enable highly accurate testing. Cloud servers (such as Amazon EC2) are suitable for the hardware used.

[0179] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0180] Step 1:

[0181] The user uploads software specifications to the server in PDF or text format. To do this, the user selects and submits the file using a specified interface. The input data is the software specifications, and the server receives and stores this file. This prepares the specification data for analysis by the system.

[0182] Step 2:

[0183] The server extracts text from the uploaded PDF using the pdfminer library. This text is the input data, and the extracted text is analyzed using a natural language processing library. The text data to be analyzed is the content of the specification, and preprocessing is performed to extract requirements from it. This allows the system to obtain text data from the specification.

[0184] Step 3:

[0185] The server uses the extracted text data to identify the requirements of the specification using a generative AI model. Specifically, it analyzes the text using a natural language processing library and extracts the functional and performance requirements contained in the specification. The input data is all text within the scope of the analysis, and the output is the data listed as requirements. Through this step, the system lists the requirements of the software specification.

[0186] Step 4:

[0187] Based on the extracted requirements, the server automatically generates test cases, applying a predefined set of rules to generate specific test scenarios. The input data is a list of requirements, and the output is the corresponding specific test cases and scenarios. The server stores the generated test cases and scenarios in a database, allowing the system to prepare various scenarios for test execution.

[0188] Step 5:

[0189] The server executes tests based on test items created using Selenium. Specifically, it executes defined test scenarios sequentially while automatically operating the browser. The input data are test items and test scenarios, and the output is the execution results of each test step. This allows the system to automatically execute software tests on actual devices and collect the results.

[0190] Step 6:

[0191] The server collects the test results executed by Selenium and classifies them as success or failure. The input data is the execution result log, and the output is the classification results of success or failure and detailed analysis data. The server analyzes this data and generates a log that clearly shows the success rate and details of any errors. This allows the system to record and analyze test results in detail.

[0192] Step 7:

[0193] If a defect is detected, the server uses a generative AI model to generate a fix. Specifically, this process involves analyzing failed test cases and logs to identify the cause and solution. The input data is a detailed log of the execution results, and the output is specific fixes and code change suggestions. This allows the system to provide specific actions to fix the software defect.

[0194] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0195] System configuration

[0196] The system of the present invention automates the process from inputting specifications to analyzing them, generating test cases, executing tests, recording results, proposing bug fixes, and recognizing user emotions. The system consists of the following main components:

[0197] 1. Means of inputting specifications

[0198] 2. A method for analyzing specifications and extracting requirements

[0199] 3. A method for generating test cases based on extracted requirements

[0200] 4. How to execute tests according to the generated test items

[0201] 5. Means for recording and analyzing test results

[0202] 6. A means of generating suggested fixes when defects are detected

[0203] 7. Emotion engine for recognizing user emotions

[0204] Specific Examples

[0205] Specification input and analysis

[0206] Users can upload specifications through the interface, and once they are sent to the server, the server uses a text analysis engine to analyze the content of the specifications and automatically extract requirements, leveraging natural language processing technology to identify key requirements and features within the specifications.

[0207] Test case generation

[0208] Based on the extracted requirements, the server automatically generates test items. For example, if a user registration function is included in the requirements, the server automatically generates test scenarios for this function, including tests for successful registration and tests for checking error messages.

[0209] Running the tests

[0210] Based on the generated test items, the server runs the tests using an automated tool that operates the web browser, filling out forms, clicking buttons, etc. The results are compared with the expected results and logged.

[0211] Recording and analyzing test results

[0212] The server collects and records the results of the tests that have been executed, categorizing successful and failed test cases and storing detailed information about each case (e.g., error messages, performance data, etc.).

[0213] Bug fix suggestions

[0214] If a defect is detected during testing, the server analyzes the cause of the defect and automatically generates a fix proposal, which includes specific code fixes and improvements.

[0215] Emotion engine integration

[0216] The server includes an emotion engine that recognizes emotions based on user operations and inputs. The emotion engine analyzes facial expressions, voice, keyboard typing speed, etc. when the user uploads specifications and checks test results.

[0217] For example, when a user checks their test results, the emotion engine analyzes their facial expressions and tone of voice to determine the emotional state they are in. If the user shows signs of dissatisfaction or irritation, the server can customize the content and presentation of the test result report to provide information in a way that is convincing to the user.

[0218] In this way, a system is realized that consistently automates everything from inputting specifications to analyzing them, generating test cases, executing tests, recording results, proposing bug fixes, and even recognizing user emotions. This system not only improves the efficiency and quality assurance of testing operations, but also provides a highly convenient testing environment that takes user emotions into consideration.

[0219] The processing flow will be explained below.

[0220] Step 1:

[0221] The user uploads the specification.

[0222] Through the interface, users upload specifications for their Internet services in PDF or text format, which are then sent to a designated endpoint on the server.

[0223] Step 2:

[0224] The server analyzes the specification and extracts the requirements.

[0225] The server analyzes the received specifications using a text analysis engine (e.g., Textract). From the analysis results, it understands the content of the specifications and extracts requirements such as software functional requirements, user requirements, and terms of use.

[0226] Step 3:

[0227] The server generates test cases based on the extracted requirements.

[0228] The server automatically generates test items based on the requirements extracted from the specifications. Specifically, if the requirements for the user registration function are included, it generates test scenarios such as a registration success test and an error test.

[0229] Step 4:

[0230] The server runs the test items.

[0231] The server uses an automated testing tool (e.g., Selenium) to execute tests according to the generated test items. Specific operations for executing tests include opening a web browser, filling out forms, and clicking buttons. The test results are compared with the expected results and recorded.

[0232] Step 5:

[0233] The server records and analyzes the test results.

[0234] The server collects the results of the executed tests, categorizes successful and failed test cases, and records the details of the results (e.g., error messages, exceptions raised, performance data, etc.) for later analysis.

[0235] Step 6:

[0236] The server generates suggested fixes when defects are detected.

[0237] If a defect is detected, the server analyzes the cause and automatically generates a fix, including specific code fixes and improvements, and notifies the user.

[0238] Step 7:

[0239] The server uses an emotion engine to recognize the user's emotion.

[0240] The server activates the emotion engine based on the user's input and operation, and the emotion engine determines the user's emotional state based on the user's facial expression, tone of voice, keyboard typing speed, etc.

[0241] Step 8:

[0242] The server customizes the test result report based on the user's emotional state.

[0243] The server customizes the content and presentation of the test result report based on the user's emotional state as determined by the emotion engine. For example, if the user expresses dissatisfaction or irritation, the server may include additional explanations and guidance in the result report to make it easier for the user to understand.

[0244] In this way, a system is provided that covers everything from inputting specifications to running tests, fixing defects, and even recognizing user emotions. This system not only improves the efficiency and quality of testing work, but also provides a testing environment that takes user emotions into consideration.

[0245] Example 2

[0246] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0247] In conventional systems, the processes of analyzing specifications, generating test items, executing tests, recording results, and proposing defect fixes were often done manually, which was time-consuming and labor-intensive. Furthermore, the system did not provide feedback that took user emotions into consideration, creating issues with the user experience. This resulted in insufficient efficiency and quality assurance for testing work.

[0248] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0249] In this invention, the server includes means for inputting specifications, means for analyzing specifications and extracting requirements, means for generating test items based on the extracted requirements, means for executing tests in accordance with the generated test items, means for recording and analyzing test results, means for generating correction suggestions when defects are detected, means for recognizing user emotions, and means for including a rule set for automatically generating test items based on the extracted requirements. This not only improves the efficiency of testing operations and quality assurance, but also makes it possible to provide feedback that takes user emotions into consideration.

[0250] A "specification" is a document that describes the functions and requirements of a system or software.

[0251] The "means for inputting specifications" refers to an interface or function that allows a user to upload specifications to the system.

[0252] A "means for analyzing specifications" is a program for analyzing the contents of specifications and extracting requirements and functions.

[0253] "Requirements" are conditions or functions that a system or software must meet.

[0254] A "means for extracting requirements" is a means for identifying and extracting necessary requirements from the specification.

[0255] A "test item" is a specific test case or scenario used to verify the functionality and requirements of a system or software.

[0256] The "means for generating test items" is a program for automatically generating test items based on the extracted requirements.

[0257] "Means for executing tests" refers to tools or programs for automatically executing tests according to the generated test items.

[0258] "Test Results" means the data and information obtained by running a test.

[0259] "Means for recording test results" refers to means for saving the results of test execution.

[0260] "Means for analyzing test results" refers to the means for analyzing recorded test results, determining success or failure, and identifying the cause.

[0261] A "defect" is a phenomenon or bug that causes a system or software to not function as specified.

[0262] A "fix proposal" is a proposal or method for improving or correcting a detected defect.

[0263] The "means for generating a proposed fix" is a program for automatically generating a proposed fix based on the detected defects.

[0264] "Means for recognizing user emotions" refers to a program or engine for analyzing and recognizing emotions from user operations and input data.

[0265] A "rule set" is a set of rules or guidelines for operating under certain criteria or conditions.

[0266] A "means including a rule set" is a setting or program that enables operation based on specific rules.

[0267] The system of the present invention automates the process from inputting specifications to analysis, generating test items, executing tests, recording results, proposing fixes for defects, and recognizing user emotions. Specific examples of each major component and process are described below.

[0268] Specification input and analysis

[0269] A user uploads a specification file through the interface. For example, in a web application, the user clicks a file selection button, selects a file such as "requirements.docx" from the local disk, and presses the upload button. The server receives this file and analyzes the content of the specification using Python's natural language processing libraries, spaCy and NLTK. The server identifies and extracts important requirements and features.

[0270] Test case generation

[0271] Based on the extracted requirements, the server automatically generates test items. For example, if a user authentication function is included in the requirements, the server generates specific test scenarios such as "Can you log in by entering correct information in the login form?" or "If you enter an incorrect password, does an error message appear?". A test generation engine (e.g., Robot Framework) is used for this process.

[0272] Running the tests

[0273] According to the generated test items, the server executes the tests using automation tools (e.g., Selenium WebDriver or Cypress). For example, the server executes each test case by programmatically operating a Google Chrome browser, entering data into a form, and clicking the submit button. The test execution results are recorded in a log file.

[0274] Recording and analyzing test results

[0275] The server collects the results of the executed tests and records them in a database, categorizing successful and failed test cases and saving detailed information for each (e.g., error messages, performance data, screenshots, etc.), allowing for easy future reference.

[0276] Bug fix suggestions

[0277] If a defect is detected during testing, the server analyzes the cause and automatically generates a correction suggestion. For example, if a database connection error occurs, the server identifies the cause and suggests a fix for the SQL query. The suggestion includes specific code examples and improvements.

[0278] Emotion engine integration

[0279] The server uses an emotion engine (e.g., Microsoft Azure Cognitive Services or Google Cloud Natural Language API) that recognizes emotions based on user actions and input. For example, when a user checks their test results, the device's camera captures their facial expressions and collects voice input. This determines the user's emotional state (e.g., "satisfied," "dissatisfied," "irritated," etc.), and if the user indicates dissatisfaction, the way the test results are displayed can be adjusted to improve the user experience.

[0280] Specific examples and prompts for the generative AI model

[0281] 1. Example of specification analysis

[0282] The user uploads the "Online Shopping System Specification."

[0283] The server receives a file called "shopping_system_spec.docx" and analyzes it using a natural language processing engine (e.g., spaCy) to extract requirements such as "user registration function," "product search function," and "cart function."

[0284] 2. Test Case Generation Example

[0285] For the requirement "User registration function", the server generates the following test items:

[0286] "Test to confirm that the registration form can be completed successfully by entering all required information."

[0287] "Test to see if an error message is displayed when an invalid email address is entered"

[0288] 3. Example prompts for the generated AI model

[0289] Prompt: Generate test scenarios for the following requirements:

[0290] Requirement: User registration function

[0291] 1. Did you enter all the required information in the registration form and register successfully?

[0292] 2. Check the error message that appears when you enter an invalid email address

[0293] This invention makes it possible to consistently automate everything from inputting specifications to analyzing them, generating test items, executing tests, recording results, proposing bug fixes, and even recognizing user emotions, thereby improving the efficiency of testing operations and enhancing quality assurance. Furthermore, by providing a highly convenient testing environment that takes user emotions into consideration, an improved user experience can be expected.

[0294] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0295] Step 1:

[0296] The user uploads a specification file (e.g., "requirements.docx") using the file selection interface of the web application, which is then sent to the server via an HTTP request.

[0297] Input: Specifications file (requirements.docx)

[0298] Output: The file sent to the server

[0299] Step 2:

[0300] The server processes the received file. First, it reads the file contents and converts them into text format. Next, it uses a natural language processing library (e.g., spaCy) to parse the specification and extract key requirements and features. This process involves data processing such as text tokenization, sentence structure analysis, and entity recognition.

[0301] Input: Received specification file

[0302] Output: Extracted requirements data (e.g., user authentication function, data storage function)

[0303] Step 3:

[0304] Based on the extracted requirements, the server automatically generates test items. For example, test items for the user authentication function such as "Can you log in by entering correct information in the login form?" and "Does an error message appear when an invalid password is entered?" are generated. An automatic test generation engine (e.g., Robot Framework) is used for this task.

[0305] Input: Extracted requirements data

[0306] Output: Generated test item list

[0307] Step 4:

[0308] The server executes tests based on the generated test items. It uses an automation tool (e.g., Selenium WebDriver) to operate a web browser (e.g., Google Chrome) to simulate form input, button clicks, etc. Test execution is controlled by a script, and results are collected in real time.

[0309] Input: Test item list

[0310] Output: Test execution results (logs, screenshots, etc.)

[0311] Step 5:

[0312] The server collects the results of the executed tests and records them in a database, categorizing successful and failed test cases and storing detailed information about each, including error messages, performance data, screenshots, etc.

[0313] Input: Test execution results

[0314] Output: Recorded test result data

[0315] Step 6:

[0316] If a test detects a defect, the server analyzes the cause and automatically generates a fix suggestion. For example, if a database connection error is detected, the server identifies the cause and generates a SQL query fix suggestion, including specific code examples and improvements.

[0317] Input: Recorded test results (failure cases)

[0318] Output: Auto-generated correction suggestions

[0319] Step 7:

[0320] The server uses an emotion engine (e.g., Microsoft Azure Cognitive Services) to recognize emotions based on user actions and inputs. For example, it captures the user's facial expressions and voice from the device's camera and microphone, and analyzes them with the emotion engine to determine the user's emotional state. If the user expresses dissatisfaction, the server adjusts the way the test results are displayed to provide information in a more convincing format.

[0321] Input: User operation data (facial expressions, voice)

[0322] Output: Analyzed user's emotional state, adjusted display content

[0323] (Application example 2)

[0324] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0325] There is a need for a system that can monitor the operation of robots operating in factories, effectively execute tests according to required specifications, and analyze the results. Furthermore, existing technologies lack the ability to quickly and specifically suggest corrections for any defects discovered, and provide feedback and improvement measures that take into consideration the emotions of the operators operating the robots. The lack of such a system can lead to reduced production efficiency and quality control issues.

[0326] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for inputting specifications, a means for analyzing specifications and extracting requirements, a means for generating test items based on the extracted requirements, a means for executing tests in accordance with the generated test items, a means for recording and analyzing test results, a means for generating correction suggestions when a defect is detected, a means for recognizing operator emotions and using this to improve system operation, a means for automatically testing and monitoring the operation of in-factory robots based on operational requirements extracted from the specifications, and a device installed on the in-factory robots. This enables quality control of robot operation and efficient production activities.

[0327] A "specification" is a document that describes detailed requirements and conditions regarding the operation and functionality of software or a system.

[0328] "Analysis" is the process of extracting necessary information and requirements from documents and data such as specifications and putting them into an understandable form.

[0329] "Requirements" are specific conditions or requirements regarding the functions and performance that a system or software must meet.

[0330] "Test items" are specific test cases or scenarios used to confirm and verify the operation and functionality of a system based on requirements.

[0331] "Test execution" is the process of actually testing the behavior of a system or robot according to the generated test items.

[0332] "Recording and analysis" refers to the process of saving the results obtained after the test is executed and analyzing the factors behind success or failure.

[0333] A "correction proposal" is a proposal for a specific correction method or improvement plan based on the cause when a defect is detected.

[0334] An "operator" is a person in charge of operating and managing robots and systems within a factory.

[0335] "Emotion recognition" is a technology that determines an operator's psychological state and emotions based on their facial expressions, voice, actions, etc.

[0336] "Operational requirements" refer to the specific operating conditions and actions that a robot or system must perform, as described in the specifications.

[0337] A "factory robot" is a mechanical device that operates within a factory to automate manufacturing and processing operations.

[0338] "Apparatus" refers to an entire machine or system designed to perform a particular function or purpose.

[0339] Specific embodiments of the present invention are as follows: This system automates the monitoring and testing of robot operations in a factory, and recognizes the emotions of the operator and provides feedback.

[0340] System configuration

[0341] The system consists of the following main components:

[0342] 1. Method of inputting specifications: Upload the operation specifications of the factory robots from the interface on the server.

[0343] 2. Analyzing specifications and extracting requirements: Analyzing the contents of specifications using natural language processing tools and extracting operational requirements. The main software used is TextBlob and spaCy.

[0344] 3. A method for generating test cases based on extracted requirements: Test scenarios are automatically generated based on the extracted requirements. Node-RED is used for this process.

[0345] 4. A method for executing tests according to the generated test items: The robots in the factory actually operate according to the generated test items. The operation tests are performed using the Robot Operating System (ROS).

[0346] 5. A means to record and analyze test results: Collect the results of the executed tests and classify them into success and failure cases. The data is recorded and analyzed using SQLAlchemy.

[0347] 6. A method to generate correction suggestions when defects are detected: Analyze the causes of defects detected in the test and automatically generate correction suggestions. Here, we use the GPT-4 API.

[0348] 7. Recognizing operator emotions and using them to improve system operation: Analyzing the emotions of the operator in real time and providing feedback. This is done using OpenAI's CLIP.

[0349] Specific Examples

[0350] Specification input and analysis

[0351] Users can upload operational specifications for factory robots to the server through the interface. The server then analyzes the uploaded specifications using TextBlob and spaCy to automatically extract operational requirements. During this process, it uses natural language processing technology to identify important requirements and functions within the specifications.

[0352] Test case generation

[0353] Based on the extracted requirements, the server automatically generates test items using Node-RED. For example, if a requirement includes checking the robot's battery, tests to check the battery status and tests to take action in the event of an abnormality are automatically generated.

[0354] Running the tests

[0355] Based on the generated test items, the server uses ROS to have the robot execute the tests. Specifically, the robot checks the battery status and performs a series of checks to see if there are any problems with its operation.

[0356] Recording and analyzing test results

[0357] The server collects and records the results of the executed tests, categorizing and analyzing successful and failed test cases using SQLAlchemy, thereby storing detailed information about each case (e.g. error messages, performance data, etc.).

[0358] Bug fix suggestions

[0359] If a defect is detected during testing, the server analyzes the cause of the defect and automatically generates correction suggestions using the GPT-4 API, including specific fixes and improvements.

[0360] Emotion engine integration

[0361] The server uses OpenAI's CLIP to analyze the operator's facial expressions and voice to determine their emotions. For example, if the operator expresses dissatisfaction or irritation while reviewing the test results, the server can customize how the feedback is displayed and provide the information in a way that is understandable to the operator.

[0362] Specific prompt examples

[0363] Example prompts for parsing specifications

[0364] What the specs say: The robot will begin its routine inspection every day at 9 a.m. The inspection will include checking the battery status, checking motor operation, and collecting sensor data.

[0365] Example prompts for emotion recognition

[0366] Operator input: I'm not happy with the test results. The robot has stopped working.

[0367] In this way, a system has been realized that consistently automates everything from inputting specifications to analysis, generating test items, running tests, recording results, proposing defect fixes, and even recognizing operator emotions. This system will enable quality control of robot operations in factories and efficient production activities.

[0368] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0369] Step 1:

[0370] The user uploads the specification from the interface on the server. The input is the operation specification of the factory robot. The output is a specification file saved on the server.

[0371] Step 2:

[0372] The server analyzes the uploaded specifications using TextBlob or spaCy to extract operational requirements. The input is the uploaded specification text. Data processing involves natural language analysis to extract requirements. The output is a list of extracted operational requirements.

[0373] Step 3:

[0374] The server uses Node-RED to automatically generate test cases based on the extracted operational requirements. The input is a list of operational requirements. Data calculations involve generating appropriate test scenarios for each requirement. The output is a list of generated test cases.

[0375] Step 4:

[0376] The server uses the generated test items to have the factory robots execute tests using ROS. The input is a list of generated test items. Specific actions performed by the robots include checking the battery status and motor operation. The output is test result data.

[0377] Step 5:

[0378] The server collects the results of the executed tests and uses SQLAlchemy to classify and record success and failure cases. The input is the test result data. Data processing involves classifying the results as success / failure and recording detailed information (e.g. error messages, performance data). The output is a classified test result report.

[0379] Step 6:

[0380] If the server detects a defect in the test results, it uses the GPT-4 API to analyze the cause of the defect and generate a correction proposal. The input is the data of the failed test case. The data calculation involves analyzing the cause of the failure and generating a correction proposal. The output is a proposal including specific correction code and improvements.

[0381] Step 7:

[0382] The server uses OpenAI's CLIP to analyze the emotions of the operator from their facial expressions and voice and provide feedback. The input is the operator's emotional data (e.g., facial expressions, voice tone). Data processing involves emotion analysis and generation of appropriate feedback content. The output is a customized feedback message for the operator.

[0383] Through the above steps, the server executes a series of automated operations, providing a system that can realize quality control of the operations of robots in the factory and emotional management of operators.

[0384] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0385] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0386] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0387] [Second embodiment]

[0388] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0389] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0390] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0391] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0392] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0393] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0394] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0395] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0396] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0397] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0398] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0399] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0400] System configuration

[0401] The system of the present invention automates a series of processes, from inputting specifications to analysis, generating test cases, executing tests, recording results, and proposing fixes for defects. The system consists of the following main components:

[0402] 1. Means of inputting specifications

[0403] 2. A method for analyzing specifications and extracting requirements

[0404] 3. Methods for generating test cases

[0405] 4. How to run the test

[0406] 5. Means for recording and analyzing test results

[0407] 6. Means of generating revisions

[0408] Specific Examples

[0409] Specification input and analysis

[0410] The server provides an interface to receive specifications from users. Users upload specifications in PDF or other text format. The server uses a text analysis engine to analyze the content of the specifications and extract requirements from them.

[0411] As a concrete example, suppose a user uploads a UI test specification for an Internet service. The server automatically extracts functional requirements and test scope from the specification. In the process, it uses natural language processing technology to understand the context of the specification and identify important requirements.

[0412] Test case generation

[0413] Based on the extracted requirements, the server automatically generates test cases, each of which includes specific test scenarios and test steps for a specific function or requirement.

[0414] For example, if requirements related to user registration functionality are extracted, the server generates test items such as "user registration success test" and "user registration failure test," which include specific steps such as filling in a form, clicking the submit button, and checking the success message.

[0415] Running the tests

[0416] The server executes the tests according to the generated test items. The server uses an automated test tool (e.g., Selenium) to operate the web browser and executes the tests. The execution results of each step are recorded and compared with the expected results.

[0417] For example, it automatically enters a name, email address, and password into a form, clicks the submit button, and checks whether a success message is displayed correctly. During this process, it records detailed logs for later reference.

[0418] Recording and analyzing test results

[0419] The server collects the results of the executed tests and classifies them as successes and failures. Based on the test results, the server analyzes the success rate of the tests, the type and frequency of failures, etc.

[0420] For example, if the test for user registration function is successful, the result is recorded as "success" and reflected in the analysis data. If the test fails, the detailed reason for the failure (e.g. error message, thrown exception, etc.) is recorded.

[0421] Generate correction suggestions

[0422] If a defect is detected during testing, the server generates a fix for the defect, analyzes the cause of the defect, and proposes specific fixes and code changes.

[0423] For example, if there is a problem with the validation of input data in the user registration function, the server will generate a correction suggestion called "Strengthen validation of user input data" and provide specific code examples and guidelines.

[0424] summary

[0425] By integrating the above components and processing steps, the system of the present invention automates all processes, from inputting specifications to analysis, generating test cases, executing tests, recording results, and proposing defect fixes. This system improves the efficiency of testing operations and strengthens quality assurance. It is an extremely useful tool, especially for developers and engineers with little testing experience.

[0426] The processing flow will be explained below.

[0427] Step 1:

[0428] The user uploads the specification.

[0429] Through the interface, users upload specifications for their Internet services in PDF or text format, which are then sent to a specific endpoint on the server.

[0430] Step 2:

[0431] The server analyzes the specification and extracts the requirements.

[0432] The server analyzes the received specifications using a text analysis engine (e.g., Textract). From the analysis results, it understands the content of the text and extracts requirements such as software functional requirements, user requirements, and terms of use.

[0433] Step 3:

[0434] The server generates test cases based on the extracted requirements.

[0435] The server automatically generates specific test items based on the requirements extracted from the specifications. For example, if the requirements for a user registration function are included, multiple test scenarios will be generated, such as a registration success test and an error test.

[0436] Step 4:

[0437] The server runs the test items.

[0438] The server uses an automated testing tool (e.g., Selenium) to run tests according to the generated test items, operating the web browser and performing specific operations such as filling out forms and clicking buttons, and recording the results of each operation and comparing them with the expected results.

[0439] Step 5:

[0440] The server records and analyzes the test results.

[0441] The server collects and records the results of the executed tests, categorizes successful and failed test cases, and generates detailed logs and analysis data, such as success rates, types of errors, and frequency of failures.

[0442] Step 6:

[0443] The server generates suggested fixes when defects are detected.

[0444] If a defect is detected, the server analyzes the cause and automatically generates a fix, generating a document with specific suggestions and instructions for code changes to help resolve the problem.

[0445] In this way, a system is realized that, through cooperation between the server and the user, handles everything from inputting specifications to conducting tests and correcting defects.

[0446] Example 1

[0447] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0448] Traditional software testing processes require a lot of time and effort, including manual requirements extraction, test case creation, and manual test execution. This reduces testing efficiency and increases the likelihood of human error. It also makes it difficult to quickly detect defects and propose fixes. Therefore, to solve these problems, a new system is needed to automate the entire process, from analyzing specifications to running tests, analyzing results, and proposing fixes.

[0449] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0450] In this invention, the server includes means for inputting specifications as data, means for analyzing the specifications and automatically extracting requirements, means for automatically generating test items based on the extracted requirements, means for executing tests using an automated test tool in accordance with the generated test items, means for recording test results and classifying and analyzing successes and failures, and means for generating and providing correction suggestions when defects are detected. This makes it possible to automate the entire software testing process and significantly improve efficiency and reliability.

[0451] A "specification" is a document that describes in detail the requirements, functions, design, etc. of software or a system.

[0452] "Means of data input" refers to the methods and techniques used to input data in the form of digital files into the system.

[0453] "Analysis" is the process of examining given data in detail and clarifying its structure and content.

[0454] "Requirements" refer to the conditions or specifications that software or a system must meet.

[0455] "Automatic generation" refers to the system generating test items and data based on certain algorithms and rules without manual intervention.

[0456] A "test item" is a specific test case or scenario that verifies a particular function or requirement of software or a system.

[0457] An "automated testing tool" is software that uses scripts and programs to run tests without human intervention.

[0458] "Recording" means saving the results of the executed tests and information about each step.

[0459] "Classification" is the act of grouping collected data based on specific criteria.

[0460] "Analysis" is the process of using collected data to interpret their meaning and draw conclusions.

[0461] A "defect" is a problem or defect that prevents software or a system from functioning properly.

[0462] A "proposed fix" is a specific method or proposal for solving a detected defect.

[0463] MODE FOR CARRYING OUT THE INVENTION

[0464] System configuration

[0465] The system of the present invention automates a series of processes, from inputting specifications to analysis, generating test items, executing tests, recording results, and proposing defect corrections. This system is composed of multiple components, including a server, terminals, and users.

[0466] Specification input and analysis

[0467] The server provides an interface for receiving specifications from users. Users upload specifications in PDF or other text format. After receiving the specifications, the server uses a text analysis engine (e.g., NLTK or spaCy) to analyze the content of the specifications and extract requirements from them. As a specific example, if a user uploads a specification for UI testing of an Internet service, the server analyzes it and automatically extracts functional requirements and test scope. During this process, natural language processing technology is used to understand the context of the specifications and identify important requirements.

[0468] Test case generation

[0469] Based on the extracted requirements, the server automatically generates test items. Each test item includes specific test scenarios and test steps for specific functions and requirements. For example, if requirements related to user registration functions are extracted, the server generates test items such as "User Registration Success Test" and "User Registration Failure Test." These test items include specific steps such as filling out a form, clicking a submit button, and checking for a success message.

[0470] Running the tests

[0471] The server executes the tests according to the generated test items. Using an automated testing tool (for example, Selenium), the tests are executed while operating the web browser. The results of each step are recorded and compared with the expected results. For example, the server can automatically enter a name, email address, and password into a form, click the submit button, and check whether a success message is displayed correctly. During this process, the server records detailed logs that can be referenced later.

[0472] Recording and analyzing test results

[0473] The server collects the results of the executed tests and classifies them as successes or failures. Based on the test results, the server analyzes the success rate, type of failure, frequency, etc. For example, if the test of the user registration function is successful, the result is recorded as "success" and reflected in the analysis data. If the test fails, the detailed reason for the failure (e.g., error message, exception occurred, etc.) is recorded.

[0474] Generate correction suggestions

[0475] If a defect is detected during testing, the server generates a correction proposal to address the defect. The server analyzes the cause of the defect and proposes specific correction methods and code changes. For example, if there is a problem with the input data validation in the user registration function, the server generates a correction proposal called "Strengthen validation of user input data" and provides specific code examples and guidelines.

[0476] Specific prompt examples

[0477] When a user uploads a specification saying they want to "test the new user registration function," the following prompt is input to the generative AI model:

[0478] "For the new user registration function, please generate test cases based on the following requirements:

[0479] 1. Enter your username, email address, and password, then click the Register button to display a success message.

[0480] 2. Display an error message if the user name is not entered.

[0481] 3. If the email address is not in the correct format, display an error message.

[0482] By inputting this prompt into a generative AI model, the model will automatically generate test items based on the requirements.

[0483] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0484] Step 1:

[0485] The user uploads the specification as data. Specifically, the user opens a web browser, accesses the specified form, selects the specification file (e.g., PDF), and clicks the upload button. This input process causes the server to obtain the specification file.

[0486] Step 2:

[0487] The server receives the uploaded specification file and temporarily stores it. Next, it uses a text analysis engine (e.g., NLTK or spaCy) to convert the content of the specification into text format and begins analyzing it. The server tokenizes each sentence and extracts target keywords and structured information. Based on the results of this analysis, a list of important requirements is extracted from the specification.

[0488] Step 3:

[0489] The server automatically generates test cases based on the extracted requirements. It uses a generative AI model or algorithm to analyze the input requirements data and generate specific test scenarios and test steps. For example, if requirements related to user registration functionality are extracted, test cases such as "User Registration Success Test" and "User Registration Failure Test" are generated. The output includes test scenarios and corresponding detailed steps.

[0490] Step 4:

[0491] The server executes the tests according to the generated test items. Specifically, the server uses an automated testing tool (e.g., Selenium) to execute test scenarios while operating a web browser. At each test step, the server automatically inputs forms and clicks buttons, and compares the expected results with the actual results. The test execution results based on this input and output are recorded as a log.

[0492] Step 5:

[0493] The server records and analyzes the results of the executed tests. The server logs the results of each test step and classifies them as success or failure. For example, if user registration is completed successfully, it is recorded as a "success," and if an error occurs, it is recorded as a "failure." This analysis data includes the success rate, the frequency of failures, details of error messages, etc.

[0494] Step 6:

[0495] If a defect is detected during testing, the server generates a correction proposal to address the defect. The server analyzes the cause of the defect and automatically generates specific correction methods and code change suggestions. For example, if there is a problem with input data validation, the server generates a correction proposal called "Strengthen input data validation" and provides specific code examples and guidelines. This correction proposal is provided to the user along with the test results.

[0496] (Application example 1)

[0497] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0498] Because software for autonomous vehicles requires safety and precision, rigorous testing is necessary when updating software or adding new features. However, traditional manual testing is burdensome and inefficient. Furthermore, the wide scope of testing can lead to human error or insufficient testing. This can reduce the reliability of the software and potentially cause safety issues. This invention was developed to solve these problems, and aims to fully automate the software testing process for autonomous vehicles, improving efficiency and accuracy.

[0499] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0500] In this invention, the server includes means for inputting specifications, means for analyzing specifications and extracting requirements, means for generating test items based on the extracted requirements, means for executing tests in accordance with the generated test items, means for recording and analyzing test results, means for generating a correction plan when a defect is detected, means for uploading software specifications for autonomous vehicles, means for automatically extracting requirements from specifications, means for automatically generating and executing test items, and means for analyzing the results in detail and, if necessary, proposing a correction plan for the defect. This automates the entire process of extracting requirements from specifications, generating test items, executing them, analyzing the results, and proposing a correction plan for the defect, thereby enabling the efficiency and accuracy of software testing for autonomous vehicles to be improved.

[0501] A "specification" is a document that describes in detail the functions and requirements of software or a system.

[0502] "Requirements" are the conditions and functions that a system or software must satisfy, extracted from the specifications.

[0503] "Test items" are specific test scenarios or steps that are generated based on extracted requirements and are used to verify specific functions or conditions.

[0504] An "autonomous vehicle" is a vehicle that completely automates driving operations without human intervention, and automatically makes decisions and operates in accordance with the surrounding situation.

[0505] "Upload" refers to the operation of transferring files or data from a local environment to a server.

[0506] "Extraction" refers to the process of selecting specific information or requirements from a document such as a specification.

[0507] "Generation" means creating new test items or scenarios from given data and conditions.

[0508] "Execution" refers to the process of automatically performing tests according to the generated test items and obtaining the results.

[0509] "Recording and analysis" means saving the test results and analyzing them in detail to identify the causes and patterns of success and failure.

[0510] A "fix" is a specific improvement plan or code modification to resolve the cause of a test failure or defect.

[0511] A "proposal" is a way of presenting a solution or improvement to a problem and encouraging its implementation.

[0512] The system of this invention can automatically perform operations from inputting specifications to analyzing them, generating test items, executing tests, recording results, and proposing defect fixes. It can be applied to software updates and adding new functions to autonomous vehicles. This system can improve the reliability and safety of software. The detailed configuration and operation of the system are described below.

[0513] System configuration

[0514] The system mainly consists of the following components:

[0515] 1. Component for inputting specifications

[0516] 2. A component that analyzes specifications and extracts requirements

[0517] 3. A component that generates test cases based on the extracted requirements

[0518] 4. Components that execute tests according to the test items

[0519] 5. Component for recording and analyzing test results

[0520] 6. A component that generates suggested fixes when defects are detected

[0521] Specification input and analysis

[0522] The server provides an interface to receive specifications from users. Users upload specifications in PDF or other text format. The server extracts the text from the specifications using pdfminer and parses and extracts requirements using a natural language processing library. This requirements extraction utilizes AI models and generative AI models.

[0523] Test case generation

[0524] Based on the extracted requirements, the server automatically generates test cases. The generated test cases are highly accurate because they create specific test scenarios for specific functions and operations. In this process, test cases are generated using the generated rule set.

[0525] Running the tests

[0526] The server runs automated tests using Selenium according to the test items. The server simulates each operation and compares the expected results with the actual results. During this process, the execution results of each step are recorded and detailed logs are collected.

[0527] Recording and analyzing test results

[0528] The server collects the executed test results and classifies the test cases as successful and failed. The recorded results are analyzed to determine the success rate and causes of failures. This analysis is done automatically using a generative AI model.

[0529] Generate correction suggestions

[0530] If a defect is detected, the server generates a fix for the defect, suggesting specific fixes and code changes and providing necessary feedback. These fix suggestions are also generated using natural language processing and AI models.

[0531] Specific examples

[0532] For example, in the case of a new navigation system update for a self-driving vehicle, the system might receive and act on the following prompt:

[0533] Example prompt sentence:

[0534] The specification states:

[0535] The new navigation system automatically calculates and displays the optimal route when the user enters their destination.

[0536] It uses cameras and sensors to perceive the surrounding terrain in real time and dynamically update the route.

[0537] It provides location information for charging stations and suggests the best charging point based on the remaining charge.

[0538] Based on this specification, please implement a program that generates test cases and automates each test.

[0539] This system will improve the efficiency of software testing for autonomous vehicles and enable highly accurate testing. Cloud servers (such as Amazon EC2) are suitable for the hardware used.

[0540] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0541] Step 1:

[0542] The user uploads software specifications to the server in PDF or text format. To do this, the user selects and submits the file using a specified interface. The input data is the software specifications, and the server receives and stores this file. This prepares the specification data for analysis by the system.

[0543] Step 2:

[0544] The server extracts text from the uploaded PDF using the pdfminer library. This text is the input data, and the extracted text is analyzed using a natural language processing library. The text data to be analyzed is the content of the specification, and preprocessing is performed to extract requirements from it. This allows the system to obtain text data from the specification.

[0545] Step 3:

[0546] The server uses the extracted text data to identify the requirements of the specification using a generative AI model. Specifically, it analyzes the text using a natural language processing library and extracts the functional and performance requirements contained in the specification. The input data is all text within the scope of the analysis, and the output is the data listed as requirements. Through this step, the system lists the requirements of the software specification.

[0547] Step 4:

[0548] Based on the extracted requirements, the server automatically generates test cases, applying a predefined set of rules to generate specific test scenarios. The input data is a list of requirements, and the output is the corresponding specific test cases and scenarios. The server stores the generated test cases and scenarios in a database, allowing the system to prepare various scenarios for test execution.

[0549] Step 5:

[0550] The server executes tests based on test items created using Selenium. Specifically, it executes defined test scenarios sequentially while automatically operating the browser. The input data are test items and test scenarios, and the output is the execution results of each test step. This allows the system to automatically execute software tests on actual devices and collect the results.

[0551] Step 6:

[0552] The server collects the test results executed by Selenium and classifies them as success or failure. The input data is the execution result log, and the output is the classification results of success or failure and detailed analysis data. The server analyzes this data and generates a log that clearly shows the success rate and details of any errors. This allows the system to record and analyze test results in detail.

[0553] Step 7:

[0554] If a defect is detected, the server uses a generative AI model to generate a fix. Specifically, this process involves analyzing failed test cases and logs to identify the cause and solution. The input data is a detailed log of the execution results, and the output is specific fixes and code change suggestions. This allows the system to provide specific actions to fix the software defect.

[0555] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0556] System configuration

[0557] The system of the present invention automates the process from inputting specifications to analyzing them, generating test cases, executing tests, recording results, proposing bug fixes, and recognizing user emotions. The system consists of the following main components:

[0558] 1. Means of inputting specifications

[0559] 2. A method for analyzing specifications and extracting requirements

[0560] 3. A method for generating test cases based on extracted requirements

[0561] 4. How to execute tests according to the generated test items

[0562] 5. Means for recording and analyzing test results

[0563] 6. A means of generating suggested fixes when defects are detected

[0564] 7. Emotion engine for recognizing user emotions

[0565] Specific Examples

[0566] Specification input and analysis

[0567] Users can upload specifications through the interface, and once they are sent to the server, the server uses a text analysis engine to analyze the content of the specifications and automatically extract requirements, leveraging natural language processing technology to identify key requirements and features within the specifications.

[0568] Test case generation

[0569] Based on the extracted requirements, the server automatically generates test items. For example, if a user registration function is included in the requirements, the server automatically generates test scenarios for this function, including tests for successful registration and tests for checking error messages.

[0570] Running the tests

[0571] Based on the generated test items, the server runs the tests using an automated tool that operates the web browser, filling out forms, clicking buttons, etc. The results are compared with the expected results and logged.

[0572] Recording and analyzing test results

[0573] The server collects and records the results of the tests that have been executed, categorizing successful and failed test cases and storing detailed information about each case (e.g., error messages, performance data, etc.).

[0574] Bug fix suggestions

[0575] If a defect is detected during testing, the server analyzes the cause of the defect and automatically generates a fix proposal, which includes specific code fixes and improvements.

[0576] Emotion engine integration

[0577] The server includes an emotion engine that recognizes emotions based on user operations and inputs. The emotion engine analyzes facial expressions, voice, keyboard typing speed, etc. when the user uploads specifications and checks test results.

[0578] For example, when a user checks their test results, the emotion engine analyzes their facial expressions and tone of voice to determine the emotional state they are in. If the user shows signs of dissatisfaction or irritation, the server can customize the content and presentation of the test result report to provide information in a way that is convincing to the user.

[0579] In this way, a system is realized that consistently automates everything from inputting specifications to analyzing them, generating test cases, executing tests, recording results, proposing bug fixes, and even recognizing user emotions. This system not only improves the efficiency and quality assurance of testing operations, but also provides a highly convenient testing environment that takes user emotions into consideration.

[0580] The processing flow will be explained below.

[0581] Step 1:

[0582] The user uploads the specification.

[0583] Through the interface, users upload specifications for their Internet services in PDF or text format, which are then sent to a designated endpoint on the server.

[0584] Step 2:

[0585] The server analyzes the specification and extracts the requirements.

[0586] The server analyzes the received specifications using a text analysis engine (e.g., Textract). From the analysis results, it understands the content of the specifications and extracts requirements such as software functional requirements, user requirements, and terms of use.

[0587] Step 3:

[0588] The server generates test cases based on the extracted requirements.

[0589] The server automatically generates test items based on the requirements extracted from the specifications. Specifically, if the requirements for the user registration function are included, it generates test scenarios such as a registration success test and an error test.

[0590] Step 4:

[0591] The server runs the test items.

[0592] The server uses an automated testing tool (e.g., Selenium) to execute tests according to the generated test items. Specific operations for executing tests include opening a web browser, filling out forms, and clicking buttons. The test results are compared with the expected results and recorded.

[0593] Step 5:

[0594] The server records and analyzes the test results.

[0595] The server collects the results of the executed tests, categorizes successful and failed test cases, and records the details of the results (e.g., error messages, exceptions raised, performance data, etc.) for later analysis.

[0596] Step 6:

[0597] The server generates suggested fixes when defects are detected.

[0598] If a defect is detected, the server analyzes the cause and automatically generates a fix, including specific code fixes and improvements, and notifies the user.

[0599] Step 7:

[0600] The server uses an emotion engine to recognize the user's emotion.

[0601] The server activates the emotion engine based on the user's input and operation, and the emotion engine determines the user's emotional state based on the user's facial expression, tone of voice, keyboard typing speed, etc.

[0602] Step 8:

[0603] The server customizes the test result report based on the user's emotional state.

[0604] The server customizes the content and presentation of the test result report based on the user's emotional state as determined by the emotion engine. For example, if the user expresses dissatisfaction or irritation, the server may include additional explanations and guidance in the result report to make it easier for the user to understand.

[0605] In this way, a system is provided that covers everything from inputting specifications to running tests, fixing defects, and even recognizing user emotions. This system not only improves the efficiency and quality of testing work, but also provides a testing environment that takes user emotions into consideration.

[0606] Example 2

[0607] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0608] In conventional systems, the processes of analyzing specifications, generating test items, executing tests, recording results, and proposing defect fixes were often done manually, which was time-consuming and labor-intensive. Furthermore, the system did not provide feedback that took user emotions into consideration, creating issues with the user experience. This resulted in insufficient efficiency and quality assurance for testing work.

[0609] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0610] In this invention, the server includes means for inputting specifications, means for analyzing specifications and extracting requirements, means for generating test items based on the extracted requirements, means for executing tests in accordance with the generated test items, means for recording and analyzing test results, means for generating correction suggestions when defects are detected, means for recognizing user emotions, and means for including a rule set for automatically generating test items based on the extracted requirements. This not only improves the efficiency of testing operations and quality assurance, but also makes it possible to provide feedback that takes user emotions into consideration.

[0611] A "specification" is a document that describes the functions and requirements of a system or software.

[0612] The "means for inputting specifications" refers to an interface or function that allows a user to upload specifications to the system.

[0613] A "means for analyzing specifications" is a program for analyzing the contents of specifications and extracting requirements and functions.

[0614] "Requirements" are conditions or functions that a system or software must meet.

[0615] A "means for extracting requirements" is a means for identifying and extracting necessary requirements from the specification.

[0616] A "test item" is a specific test case or scenario used to verify the functionality and requirements of a system or software.

[0617] The "means for generating test items" is a program for automatically generating test items based on the extracted requirements.

[0618] "Means for executing tests" refers to tools or programs for automatically executing tests according to the generated test items.

[0619] "Test Results" means the data and information obtained by running a test.

[0620] "Means for recording test results" refers to means for saving the results of test execution.

[0621] "Means for analyzing test results" refers to the means for analyzing recorded test results, determining success or failure, and identifying the cause.

[0622] A "defect" is a phenomenon or bug that causes a system or software to not function as specified.

[0623] A "fix proposal" is a proposal or method for improving or correcting a detected defect.

[0624] The "means for generating a proposed fix" is a program for automatically generating a proposed fix based on the detected defects.

[0625] "Means for recognizing user emotions" refers to a program or engine for analyzing and recognizing emotions from user operations and input data.

[0626] A "rule set" is a set of rules or guidelines for operating under certain criteria or conditions.

[0627] A "means including a rule set" is a setting or program that enables operation based on specific rules.

[0628] The system of the present invention automates the process from inputting specifications to analysis, generating test items, executing tests, recording results, proposing fixes for defects, and recognizing user emotions. Specific examples of each major component and process are described below.

[0629] Specification input and analysis

[0630] A user uploads a specification file through the interface. For example, in a web application, the user clicks a file selection button, selects a file such as "requirements.docx" from the local disk, and presses the upload button. The server receives this file and analyzes the content of the specification using Python's natural language processing libraries, spaCy and NLTK. The server identifies and extracts important requirements and features.

[0631] Test case generation

[0632] Based on the extracted requirements, the server automatically generates test items. For example, if a user authentication function is included in the requirements, the server generates specific test scenarios such as "Can you log in by entering correct information in the login form?" or "If you enter an incorrect password, does an error message appear?". A test generation engine (e.g., Robot Framework) is used for this process.

[0633] Running the tests

[0634] According to the generated test items, the server executes the tests using automation tools (e.g., Selenium WebDriver or Cypress). For example, the server executes each test case by programmatically operating a Google Chrome browser, entering data into a form, and clicking the submit button. The test execution results are recorded in a log file.

[0635] Recording and analyzing test results

[0636] The server collects the results of the executed tests and records them in a database, categorizing successful and failed test cases and saving detailed information for each (e.g., error messages, performance data, screenshots, etc.), allowing for easy future reference.

[0637] Bug fix suggestions

[0638] If a defect is detected during testing, the server analyzes the cause and automatically generates a correction suggestion. For example, if a database connection error occurs, the server identifies the cause and suggests a fix for the SQL query. The suggestion includes specific code examples and improvements.

[0639] Emotion engine integration

[0640] The server uses an emotion engine (e.g., Microsoft Azure Cognitive Services or Google Cloud Natural Language API) that recognizes emotions based on user actions and input. For example, when a user checks their test results, the device's camera captures their facial expressions and collects voice input. This determines the user's emotional state (e.g., "satisfied," "dissatisfied," "irritated," etc.), and if the user indicates dissatisfaction, the way the test results are displayed can be adjusted to improve the user experience.

[0641] Specific examples and prompts for the generative AI model

[0642] 1. Example of specification analysis

[0643] The user uploads the "Online Shopping System Specification."

[0644] The server receives a file called "shopping_system_spec.docx" and analyzes it using a natural language processing engine (e.g., spaCy) to extract requirements such as "user registration function," "product search function," and "cart function."

[0645] 2. Test Case Generation Example

[0646] For the requirement "User registration function", the server generates the following test items:

[0647] "Test to confirm that the registration form can be completed successfully by entering all required information."

[0648] "Test to see if an error message is displayed when an invalid email address is entered"

[0649] 3. Example prompts for the generated AI model

[0650] Prompt: Generate test scenarios for the following requirements:

[0651] Requirement: User registration function

[0652] 1. Did you enter all the required information in the registration form and register successfully?

[0653] 2. Check the error message that appears when you enter an invalid email address

[0654] This invention makes it possible to consistently automate everything from inputting specifications to analyzing them, generating test items, executing tests, recording results, proposing bug fixes, and even recognizing user emotions, thereby improving the efficiency of testing operations and enhancing quality assurance. Furthermore, by providing a highly convenient testing environment that takes user emotions into consideration, an improved user experience can be expected.

[0655] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0656] Step 1:

[0657] The user uploads a specification file (e.g., "requirements.docx") using the file selection interface of the web application, which is then sent to the server via an HTTP request.

[0658] Input: Specifications file (requirements.docx)

[0659] Output: The file sent to the server

[0660] Step 2:

[0661] The server processes the received file. First, it reads the file contents and converts them into text format. Next, it uses a natural language processing library (e.g., spaCy) to parse the specification and extract key requirements and features. This process involves data processing such as text tokenization, sentence structure analysis, and entity recognition.

[0662] Input: Received specification file

[0663] Output: Extracted requirements data (e.g., user authentication function, data storage function)

[0664] Step 3:

[0665] Based on the extracted requirements, the server automatically generates test items. For example, test items for the user authentication function such as "Can you log in by entering correct information in the login form?" and "Does an error message appear when an invalid password is entered?" are generated. An automatic test generation engine (e.g., Robot Framework) is used for this task.

[0666] Input: Extracted requirements data

[0667] Output: Generated test item list

[0668] Step 4:

[0669] The server executes tests based on the generated test items. It uses an automation tool (e.g., Selenium WebDriver) to operate a web browser (e.g., Google Chrome) to simulate form input, button clicks, etc. Test execution is controlled by a script, and results are collected in real time.

[0670] Input: Test item list

[0671] Output: Test execution results (logs, screenshots, etc.)

[0672] Step 5:

[0673] The server collects the results of the executed tests and records them in a database, categorizing successful and failed test cases and storing detailed information about each, including error messages, performance data, screenshots, etc.

[0674] Input: Test execution results

[0675] Output: Recorded test result data

[0676] Step 6:

[0677] If a test detects a defect, the server analyzes the cause and automatically generates a fix suggestion. For example, if a database connection error is detected, the server identifies the cause and generates a SQL query fix suggestion, including specific code examples and improvements.

[0678] Input: Recorded test results (failure cases)

[0679] Output: Auto-generated correction suggestions

[0680] Step 7:

[0681] The server uses an emotion engine (e.g., Microsoft Azure Cognitive Services) to recognize emotions based on user actions and inputs. For example, it captures the user's facial expressions and voice from the device's camera and microphone, and analyzes them with the emotion engine to determine the user's emotional state. If the user expresses dissatisfaction, the server adjusts the way the test results are displayed to provide information in a more convincing format.

[0682] Input: User operation data (facial expressions, voice)

[0683] Output: Analyzed user's emotional state, adjusted display content

[0684] (Application example 2)

[0685] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0686] There is a need for a system that can monitor the operation of robots operating in factories, effectively execute tests according to required specifications, and analyze the results. Furthermore, existing technologies lack the ability to quickly and specifically suggest corrections for any defects discovered, and provide feedback and improvement measures that take into consideration the emotions of the operators operating the robots. The lack of such a system can lead to reduced production efficiency and quality control issues.

[0687] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for inputting specifications, a means for analyzing specifications and extracting requirements, a means for generating test items based on the extracted requirements, a means for executing tests in accordance with the generated test items, a means for recording and analyzing test results, a means for generating correction suggestions when a defect is detected, a means for recognizing operator emotions and using this to improve system operation, a means for automatically testing and monitoring the operation of in-factory robots based on operational requirements extracted from the specifications, and a device installed on the in-factory robots. This enables quality control of robot operation and efficient production activities.

[0688] A "specification" is a document that describes detailed requirements and conditions regarding the operation and functionality of software or a system.

[0689] "Analysis" is the process of extracting necessary information and requirements from documents and data such as specifications and putting them into an understandable form.

[0690] "Requirements" are specific conditions or requirements regarding the functions and performance that a system or software must meet.

[0691] "Test items" are specific test cases or scenarios used to confirm and verify the operation and functionality of a system based on requirements.

[0692] "Test execution" is the process of actually testing the behavior of a system or robot according to the generated test items.

[0693] "Recording and analysis" refers to the process of saving the results obtained after the test is executed and analyzing the factors behind success or failure.

[0694] A "correction proposal" is a proposal for a specific correction method or improvement plan based on the cause when a defect is detected.

[0695] An "operator" is a person in charge of operating and managing robots and systems within a factory.

[0696] "Emotion recognition" is a technology that determines an operator's psychological state and emotions based on their facial expressions, voice, actions, etc.

[0697] "Operational requirements" refer to the specific operating conditions and actions that a robot or system must perform, as described in the specifications.

[0698] A "factory robot" is a mechanical device that operates within a factory to automate manufacturing and processing operations.

[0699] "Apparatus" refers to an entire machine or system designed to perform a particular function or purpose.

[0700] Specific embodiments of the present invention are as follows: This system automates the monitoring and testing of robot operations in a factory, and recognizes the emotions of the operator and provides feedback.

[0701] System configuration

[0702] The system consists of the following main components:

[0703] 1. Method of inputting specifications: Upload the operation specifications of the factory robots from the interface on the server.

[0704] 2. Analyzing specifications and extracting requirements: Analyzing the contents of specifications using natural language processing tools and extracting operational requirements. The main software used is TextBlob and spaCy.

[0705] 3. A method for generating test cases based on extracted requirements: Test scenarios are automatically generated based on the extracted requirements. Node-RED is used for this process.

[0706] 4. A method for executing tests according to the generated test items: The robots in the factory actually operate according to the generated test items. The operation tests are performed using the Robot Operating System (ROS).

[0707] 5. A means to record and analyze test results: Collect the results of the executed tests and classify them into success and failure cases. The data is recorded and analyzed using SQLAlchemy.

[0708] 6. A method to generate correction suggestions when defects are detected: Analyze the causes of defects detected in the test and automatically generate correction suggestions. Here, we use the GPT-4 API.

[0709] 7. Recognizing operator emotions and using them to improve system operation: Analyzing the emotions of the operator in real time and providing feedback. This is done using OpenAI's CLIP.

[0710] Specific Examples

[0711] Specification input and analysis

[0712] Users can upload operational specifications for factory robots to the server through the interface. The server then analyzes the uploaded specifications using TextBlob and spaCy to automatically extract operational requirements. During this process, it uses natural language processing technology to identify important requirements and functions within the specifications.

[0713] Test case generation

[0714] Based on the extracted requirements, the server automatically generates test items using Node-RED. For example, if a requirement includes checking the robot's battery, tests to check the battery status and tests to take action in the event of an abnormality are automatically generated.

[0715] Running the tests

[0716] Based on the generated test items, the server uses ROS to have the robot execute the tests. Specifically, the robot checks the battery status and performs a series of checks to see if there are any problems with its operation.

[0717] Recording and analyzing test results

[0718] The server collects and records the results of the executed tests, categorizing and analyzing successful and failed test cases using SQLAlchemy, thereby storing detailed information about each case (e.g. error messages, performance data, etc.).

[0719] Bug fix suggestions

[0720] If a defect is detected during testing, the server analyzes the cause of the defect and automatically generates correction suggestions using the GPT-4 API, including specific fixes and improvements.

[0721] Emotion engine integration

[0722] The server uses OpenAI's CLIP to analyze the operator's facial expressions and voice to determine their emotions. For example, if the operator expresses dissatisfaction or irritation while reviewing the test results, the server can customize how the feedback is displayed and provide the information in a way that is understandable to the operator.

[0723] Specific prompt examples

[0724] Example prompts for parsing specifications

[0725] What the specs say: The robot will begin its routine inspection every day at 9 a.m. The inspection will include checking the battery status, checking motor operation, and collecting sensor data.

[0726] Example prompts for emotion recognition

[0727] Operator input: I'm not happy with the test results. The robot has stopped working.

[0728] In this way, a system has been realized that consistently automates everything from inputting specifications to analysis, generating test items, running tests, recording results, proposing defect fixes, and even recognizing operator emotions. This system will enable quality control of robot operations in factories and efficient production activities.

[0729] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0730] Step 1:

[0731] The user uploads the specification from the interface on the server. The input is the operation specification of the factory robot. The output is a specification file saved on the server.

[0732] Step 2:

[0733] The server analyzes the uploaded specifications using TextBlob or spaCy to extract operational requirements. The input is the uploaded specification text. Data processing involves natural language analysis to extract requirements. The output is a list of extracted operational requirements.

[0734] Step 3:

[0735] The server uses Node-RED to automatically generate test cases based on the extracted operational requirements. The input is a list of operational requirements. Data calculations involve generating appropriate test scenarios for each requirement. The output is a list of generated test cases.

[0736] Step 4:

[0737] The server uses the generated test items to have the factory robots execute tests using ROS. The input is a list of generated test items. Specific actions performed by the robots include checking the battery status and motor operation. The output is test result data.

[0738] Step 5:

[0739] The server collects the results of the executed tests and uses SQLAlchemy to classify and record success and failure cases. The input is the test result data. Data processing involves classifying the results as success / failure and recording detailed information (e.g. error messages, performance data). The output is a classified test result report.

[0740] Step 6:

[0741] If the server detects a defect in the test results, it uses the GPT-4 API to analyze the cause of the defect and generate a correction proposal. The input is the data of the failed test case. The data calculation involves analyzing the cause of the failure and generating a correction proposal. The output is a proposal including specific correction code and improvements.

[0742] Step 7:

[0743] The server uses OpenAI's CLIP to analyze the emotions of the operator from their facial expressions and voice and provide feedback. The input is the operator's emotional data (e.g., facial expressions, voice tone). Data processing involves emotion analysis and generation of appropriate feedback content. The output is a customized feedback message for the operator.

[0744] Through the above steps, the server executes a series of automated operations, providing a system that can realize quality control of the operations of robots in the factory and emotional management of operators.

[0745] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0746] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0747] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0748] [Third embodiment]

[0749] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0750] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0751] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0752] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0753] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0754] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0755] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0756] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0757] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0758] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0759] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0760] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0761] System configuration

[0762] The system of the present invention automates a series of processes, from inputting specifications to analysis, generating test cases, executing tests, recording results, and proposing fixes for defects. The system consists of the following main components:

[0763] 1. Means of inputting specifications

[0764] 2. A method for analyzing specifications and extracting requirements

[0765] 3. Methods for generating test cases

[0766] 4. How to run the test

[0767] 5. Means for recording and analyzing test results

[0768] 6. Means of generating revisions

[0769] Specific Examples

[0770] Specification input and analysis

[0771] The server provides an interface to receive specifications from users. Users upload specifications in PDF or other text format. The server uses a text analysis engine to analyze the content of the specifications and extract requirements from them.

[0772] As a concrete example, suppose a user uploads a UI test specification for an Internet service. The server automatically extracts functional requirements and test scope from the specification. In the process, it uses natural language processing technology to understand the context of the specification and identify important requirements.

[0773] Test case generation

[0774] Based on the extracted requirements, the server automatically generates test cases, each of which includes specific test scenarios and test steps for a specific function or requirement.

[0775] For example, if requirements related to user registration functionality are extracted, the server generates test items such as "user registration success test" and "user registration failure test," which include specific steps such as filling in a form, clicking the submit button, and checking the success message.

[0776] Running the tests

[0777] The server executes the tests according to the generated test items. The server uses an automated test tool (e.g., Selenium) to operate the web browser and executes the tests. The execution results of each step are recorded and compared with the expected results.

[0778] For example, it automatically enters a name, email address, and password into a form, clicks the submit button, and checks whether a success message is displayed correctly. During this process, it records detailed logs for later reference.

[0779] Recording and analyzing test results

[0780] The server collects the results of the executed tests and classifies them as successes and failures. Based on the test results, the server analyzes the success rate of the tests, the type and frequency of failures, etc.

[0781] For example, if the test for user registration function is successful, the result is recorded as "success" and reflected in the analysis data. If the test fails, the detailed reason for the failure (e.g. error message, thrown exception, etc.) is recorded.

[0782] Generate correction suggestions

[0783] If a defect is detected during testing, the server generates a fix for the defect, analyzes the cause of the defect, and proposes specific fixes and code changes.

[0784] For example, if there is a problem with the validation of input data in the user registration function, the server will generate a correction suggestion called "Strengthen validation of user input data" and provide specific code examples and guidelines.

[0785] summary

[0786] By integrating the above components and processing steps, the system of the present invention automates all processes, from inputting specifications to analysis, generating test cases, executing tests, recording results, and proposing defect fixes. This system improves the efficiency of testing operations and strengthens quality assurance. It is an extremely useful tool, especially for developers and engineers with little testing experience.

[0787] The processing flow will be explained below.

[0788] Step 1:

[0789] The user uploads the specification.

[0790] Through the interface, users upload specifications for their Internet services in PDF or text format, which are then sent to a specific endpoint on the server.

[0791] Step 2:

[0792] The server analyzes the specification and extracts the requirements.

[0793] The server analyzes the received specifications using a text analysis engine (e.g., Textract). From the analysis results, it understands the content of the text and extracts requirements such as software functional requirements, user requirements, and terms of use.

[0794] Step 3:

[0795] The server generates test cases based on the extracted requirements.

[0796] The server automatically generates specific test items based on the requirements extracted from the specifications. For example, if the requirements for a user registration function are included, multiple test scenarios will be generated, such as a registration success test and an error test.

[0797] Step 4:

[0798] The server runs the test items.

[0799] The server uses an automated testing tool (e.g., Selenium) to run tests according to the generated test items, operating the web browser and performing specific operations such as filling out forms and clicking buttons, and recording the results of each operation and comparing them with the expected results.

[0800] Step 5:

[0801] The server records and analyzes the test results.

[0802] The server collects and records the results of the executed tests, categorizes successful and failed test cases, and generates detailed logs and analysis data, such as success rates, types of errors, and frequency of failures.

[0803] Step 6:

[0804] The server generates suggested fixes when defects are detected.

[0805] If a defect is detected, the server analyzes the cause and automatically generates a fix, generating a document with specific suggestions and instructions for code changes to help resolve the problem.

[0806] In this way, a system is realized that, through cooperation between the server and the user, handles everything from inputting specifications to conducting tests and correcting defects.

[0807] Example 1

[0808] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0809] Traditional software testing processes require a lot of time and effort, including manual requirements extraction, test case creation, and manual test execution. This reduces testing efficiency and increases the likelihood of human error. It also makes it difficult to quickly detect defects and propose fixes. Therefore, to solve these problems, a new system is needed to automate the entire process, from analyzing specifications to running tests, analyzing results, and proposing fixes.

[0810] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0811] In this invention, the server includes means for inputting specifications as data, means for analyzing the specifications and automatically extracting requirements, means for automatically generating test items based on the extracted requirements, means for executing tests using an automated test tool in accordance with the generated test items, means for recording test results and classifying and analyzing successes and failures, and means for generating and providing correction suggestions when defects are detected. This makes it possible to automate the entire software testing process and significantly improve efficiency and reliability.

[0812] A "specification" is a document that describes in detail the requirements, functions, design, etc. of software or a system.

[0813] "Means of data input" refers to the methods and techniques used to input data in the form of digital files into the system.

[0814] "Analysis" is the process of examining given data in detail and clarifying its structure and content.

[0815] "Requirements" refer to the conditions or specifications that software or a system must meet.

[0816] "Automatic generation" refers to the system generating test items and data based on certain algorithms and rules without manual intervention.

[0817] A "test item" is a specific test case or scenario that verifies a particular function or requirement of software or a system.

[0818] An "automated testing tool" is software that uses scripts and programs to run tests without human intervention.

[0819] "Recording" means saving the results of the executed tests and information about each step.

[0820] "Classification" is the act of grouping collected data based on specific criteria.

[0821] "Analysis" is the process of using collected data to interpret their meaning and draw conclusions.

[0822] A "defect" is a problem or defect that prevents software or a system from functioning properly.

[0823] A "proposed fix" is a specific method or proposal for solving a detected defect.

[0824] MODE FOR CARRYING OUT THE INVENTION

[0825] System configuration

[0826] The system of the present invention automates a series of processes, from inputting specifications to analysis, generating test items, executing tests, recording results, and proposing defect corrections. This system is composed of multiple components, including a server, terminals, and users.

[0827] Specification input and analysis

[0828] The server provides an interface for receiving specifications from users. Users upload specifications in PDF or other text format. After receiving the specifications, the server uses a text analysis engine (e.g., NLTK or spaCy) to analyze the content of the specifications and extract requirements from them. As a specific example, if a user uploads a specification for UI testing of an Internet service, the server analyzes it and automatically extracts functional requirements and test scope. During this process, natural language processing technology is used to understand the context of the specifications and identify important requirements.

[0829] Test case generation

[0830] Based on the extracted requirements, the server automatically generates test items. Each test item includes specific test scenarios and test steps for specific functions and requirements. For example, if requirements related to user registration functions are extracted, the server generates test items such as "User Registration Success Test" and "User Registration Failure Test." These test items include specific steps such as filling out a form, clicking a submit button, and checking for a success message.

[0831] Running the tests

[0832] The server executes the tests according to the generated test items. Using an automated testing tool (for example, Selenium), the tests are executed while operating the web browser. The results of each step are recorded and compared with the expected results. For example, the server can automatically enter a name, email address, and password into a form, click the submit button, and check whether a success message is displayed correctly. During this process, the server records detailed logs that can be referenced later.

[0833] Recording and analyzing test results

[0834] The server collects the results of the executed tests and classifies them as successes or failures. Based on the test results, the server analyzes the success rate, type of failure, frequency, etc. For example, if the test of the user registration function is successful, the result is recorded as "success" and reflected in the analysis data. If the test fails, the detailed reason for the failure (e.g., error message, exception occurred, etc.) is recorded.

[0835] Generate correction suggestions

[0836] If a defect is detected during testing, the server generates a correction proposal to address the defect. The server analyzes the cause of the defect and proposes specific correction methods and code changes. For example, if there is a problem with the input data validation in the user registration function, the server generates a correction proposal called "Strengthen validation of user input data" and provides specific code examples and guidelines.

[0837] Specific prompt examples

[0838] When a user uploads a specification saying they want to "test the new user registration function," the following prompt is input to the generative AI model:

[0839] "For the new user registration function, please generate test cases based on the following requirements:

[0840] 1. Enter your username, email address, and password, then click the Register button to display a success message.

[0841] 2. Display an error message if the user name is not entered.

[0842] 3. If the email address is not in the correct format, display an error message.

[0843] By inputting this prompt into a generative AI model, the model will automatically generate test items based on the requirements.

[0844] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0845] Step 1:

[0846] The user uploads the specification as data. Specifically, the user opens a web browser, accesses the specified form, selects the specification file (e.g., PDF), and clicks the upload button. This input process causes the server to obtain the specification file.

[0847] Step 2:

[0848] The server receives the uploaded specification file and temporarily stores it. Next, it uses a text analysis engine (e.g., NLTK or spaCy) to convert the content of the specification into text format and begins analyzing it. The server tokenizes each sentence and extracts target keywords and structured information. Based on the results of this analysis, a list of important requirements is extracted from the specification.

[0849] Step 3:

[0850] The server automatically generates test cases based on the extracted requirements. It uses a generative AI model or algorithm to analyze the input requirements data and generate specific test scenarios and test steps. For example, if requirements related to user registration functionality are extracted, test cases such as "User Registration Success Test" and "User Registration Failure Test" are generated. The output includes test scenarios and corresponding detailed steps.

[0851] Step 4:

[0852] The server executes the tests according to the generated test items. Specifically, the server uses an automated testing tool (e.g., Selenium) to execute test scenarios while operating a web browser. At each test step, the server automatically inputs forms and clicks buttons, and compares the expected results with the actual results. The test execution results based on this input and output are recorded as a log.

[0853] Step 5:

[0854] The server records and analyzes the results of the executed tests. The server logs the results of each test step and classifies them as success or failure. For example, if user registration is completed successfully, it is recorded as a "success," and if an error occurs, it is recorded as a "failure." This analysis data includes the success rate, the frequency of failures, details of error messages, etc.

[0855] Step 6:

[0856] If a defect is detected during testing, the server generates a correction proposal to address the defect. The server analyzes the cause of the defect and automatically generates specific correction methods and code change suggestions. For example, if there is a problem with input data validation, the server generates a correction proposal called "Strengthen input data validation" and provides specific code examples and guidelines. This correction proposal is provided to the user along with the test results.

[0857] (Application example 1)

[0858] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0859] Because software for autonomous vehicles requires safety and precision, rigorous testing is necessary when updating software or adding new features. However, traditional manual testing is burdensome and inefficient. Furthermore, the wide scope of testing can lead to human error or insufficient testing. This can reduce the reliability of the software and potentially cause safety issues. This invention was developed to solve these problems, and aims to fully automate the software testing process for autonomous vehicles, improving efficiency and accuracy.

[0860] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0861] In this invention, the server includes means for inputting specifications, means for analyzing specifications and extracting requirements, means for generating test items based on the extracted requirements, means for executing tests in accordance with the generated test items, means for recording and analyzing test results, means for generating a correction plan when a defect is detected, means for uploading software specifications for autonomous vehicles, means for automatically extracting requirements from specifications, means for automatically generating and executing test items, and means for analyzing the results in detail and, if necessary, proposing a correction plan for the defect. This automates the entire process of extracting requirements from specifications, generating test items, executing them, analyzing the results, and proposing a correction plan for the defect, thereby enabling the efficiency and accuracy of software testing for autonomous vehicles to be improved.

[0862] A "specification" is a document that describes in detail the functions and requirements of software or a system.

[0863] "Requirements" are the conditions and functions that a system or software must satisfy, extracted from the specifications.

[0864] "Test items" are specific test scenarios or steps that are generated based on extracted requirements and are used to verify specific functions or conditions.

[0865] An "autonomous vehicle" is a vehicle that completely automates driving operations without human intervention, and automatically makes decisions and operates in accordance with the surrounding situation.

[0866] "Upload" refers to the operation of transferring files or data from a local environment to a server.

[0867] "Extraction" refers to the process of selecting specific information or requirements from a document such as a specification.

[0868] "Generation" means creating new test items or scenarios from given data and conditions.

[0869] "Execution" refers to the process of automatically performing tests according to the generated test items and obtaining the results.

[0870] "Recording and analysis" means saving the test results and analyzing them in detail to identify the causes and patterns of success and failure.

[0871] A "fix" is a specific improvement plan or code modification to resolve the cause of a test failure or defect.

[0872] A "proposal" is a way of presenting a solution or improvement to a problem and encouraging its implementation.

[0873] The system of this invention can automatically perform operations from inputting specifications to analyzing them, generating test items, executing tests, recording results, and proposing defect fixes. It can be applied to software updates and adding new functions to autonomous vehicles. This system can improve the reliability and safety of software. The detailed configuration and operation of the system are described below.

[0874] System configuration

[0875] The system mainly consists of the following components:

[0876] 1. Component for inputting specifications

[0877] 2. A component that analyzes specifications and extracts requirements

[0878] 3. A component that generates test cases based on the extracted requirements

[0879] 4. Components that execute tests according to the test items

[0880] 5. Component for recording and analyzing test results

[0881] 6. A component that generates suggested fixes when defects are detected

[0882] Specification input and analysis

[0883] The server provides an interface to receive specifications from users. Users upload specifications in PDF or other text format. The server extracts the text from the specifications using pdfminer and parses and extracts requirements using a natural language processing library. This requirements extraction utilizes AI models and generative AI models.

[0884] Test case generation

[0885] Based on the extracted requirements, the server automatically generates test cases. The generated test cases are highly accurate because they create specific test scenarios for specific functions and operations. In this process, test cases are generated using the generated rule set.

[0886] Running the tests

[0887] The server runs automated tests using Selenium according to the test items. The server simulates each operation and compares the expected results with the actual results. During this process, the execution results of each step are recorded and detailed logs are collected.

[0888] Recording and analyzing test results

[0889] The server collects the executed test results and classifies the test cases as successful and failed. The recorded results are analyzed to determine the success rate and causes of failures. This analysis is done automatically using a generative AI model.

[0890] Generate correction suggestions

[0891] If a defect is detected, the server generates a fix for the defect, suggesting specific fixes and code changes and providing necessary feedback. These fix suggestions are also generated using natural language processing and AI models.

[0892] Specific examples

[0893] For example, in the case of a new navigation system update for a self-driving vehicle, the system might receive and act on the following prompt:

[0894] Example prompt sentence:

[0895] The specification states:

[0896] The new navigation system automatically calculates and displays the optimal route when the user enters their destination.

[0897] It uses cameras and sensors to perceive the surrounding terrain in real time and dynamically update the route.

[0898] It provides location information for charging stations and suggests the best charging point based on the remaining charge.

[0899] Based on this specification, please implement a program that generates test cases and automates each test.

[0900] This system will improve the efficiency of software testing for autonomous vehicles and enable highly accurate testing. Cloud servers (such as Amazon EC2) are suitable for the hardware used.

[0901] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0902] Step 1:

[0903] The user uploads software specifications to the server in PDF or text format. To do this, the user selects and submits the file using a specified interface. The input data is the software specifications, and the server receives and stores this file. This prepares the specification data for analysis by the system.

[0904] Step 2:

[0905] The server extracts text from the uploaded PDF using the pdfminer library. This text is the input data, and the extracted text is analyzed using a natural language processing library. The text data to be analyzed is the content of the specification, and preprocessing is performed to extract requirements from it. This allows the system to obtain text data from the specification.

[0906] Step 3:

[0907] The server uses the extracted text data to identify the requirements of the specification using a generative AI model. Specifically, it analyzes the text using a natural language processing library and extracts the functional and performance requirements contained in the specification. The input data is all text within the scope of the analysis, and the output is the data listed as requirements. Through this step, the system lists the requirements of the software specification.

[0908] Step 4:

[0909] Based on the extracted requirements, the server automatically generates test cases, applying a predefined set of rules to generate specific test scenarios. The input data is a list of requirements, and the output is the corresponding specific test cases and scenarios. The server stores the generated test cases and scenarios in a database, allowing the system to prepare various scenarios for test execution.

[0910] Step 5:

[0911] The server executes tests based on test items created using Selenium. Specifically, it executes defined test scenarios sequentially while automatically operating the browser. The input data are test items and test scenarios, and the output is the execution results of each test step. This allows the system to automatically execute software tests on actual devices and collect the results.

[0912] Step 6:

[0913] The server collects the test results executed by Selenium and classifies them as success or failure. The input data is the execution result log, and the output is the classification results of success or failure and detailed analysis data. The server analyzes this data and generates a log that clearly shows the success rate and details of any errors. This allows the system to record and analyze test results in detail.

[0914] Step 7:

[0915] If a defect is detected, the server uses a generative AI model to generate a fix. Specifically, this process involves analyzing failed test cases and logs to identify the cause and solution. The input data is a detailed log of the execution results, and the output is specific fixes and code change suggestions. This allows the system to provide specific actions to fix the software defect.

[0916] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0917] System configuration

[0918] The system of the present invention automates the process from inputting specifications to analyzing them, generating test cases, executing tests, recording results, proposing bug fixes, and recognizing user emotions. The system consists of the following main components:

[0919] 1. Means of inputting specifications

[0920] 2. A method for analyzing specifications and extracting requirements

[0921] 3. A method for generating test cases based on extracted requirements

[0922] 4. How to execute tests according to the generated test items

[0923] 5. Means for recording and analyzing test results

[0924] 6. A means of generating suggested fixes when defects are detected

[0925] 7. Emotion engine for recognizing user emotions

[0926] Specific Examples

[0927] Specification input and analysis

[0928] Users can upload specifications through the interface, and once they are sent to the server, the server uses a text analysis engine to analyze the content of the specifications and automatically extract requirements, leveraging natural language processing technology to identify key requirements and features within the specifications.

[0929] Test case generation

[0930] Based on the extracted requirements, the server automatically generates test items. For example, if a user registration function is included in the requirements, the server automatically generates test scenarios for this function, including tests for successful registration and tests for checking error messages.

[0931] Running the tests

[0932] Based on the generated test items, the server runs the tests using an automated tool that operates the web browser, filling out forms, clicking buttons, etc. The results are compared with the expected results and logged.

[0933] Recording and analyzing test results

[0934] The server collects and records the results of the tests that have been executed, categorizing successful and failed test cases and storing detailed information about each case (e.g., error messages, performance data, etc.).

[0935] Bug fix suggestions

[0936] If a defect is detected during testing, the server analyzes the cause of the defect and automatically generates a fix proposal, which includes specific code fixes and improvements.

[0937] Emotion engine integration

[0938] The server includes an emotion engine that recognizes emotions based on user operations and inputs. The emotion engine analyzes facial expressions, voice, keyboard typing speed, etc. when the user uploads specifications and checks test results.

[0939] For example, when a user checks their test results, the emotion engine analyzes their facial expressions and tone of voice to determine the emotional state they are in. If the user shows signs of dissatisfaction or irritation, the server can customize the content and presentation of the test result report to provide information in a way that is convincing to the user.

[0940] In this way, a system is realized that consistently automates everything from inputting specifications to analyzing them, generating test cases, executing tests, recording results, proposing bug fixes, and even recognizing user emotions. This system not only improves the efficiency and quality assurance of testing operations, but also provides a highly convenient testing environment that takes user emotions into consideration.

[0941] The processing flow will be explained below.

[0942] Step 1:

[0943] The user uploads the specification.

[0944] Through the interface, users upload specifications for their Internet services in PDF or text format, which are then sent to a designated endpoint on the server.

[0945] Step 2:

[0946] The server analyzes the specification and extracts the requirements.

[0947] The server analyzes the received specifications using a text analysis engine (e.g., Textract). From the analysis results, it understands the content of the specifications and extracts requirements such as software functional requirements, user requirements, and terms of use.

[0948] Step 3:

[0949] The server generates test cases based on the extracted requirements.

[0950] The server automatically generates test items based on the requirements extracted from the specifications. Specifically, if the requirements for the user registration function are included, it generates test scenarios such as a registration success test and an error test.

[0951] Step 4:

[0952] The server runs the test items.

[0953] The server uses an automated testing tool (e.g., Selenium) to execute tests according to the generated test items. Specific operations for executing tests include opening a web browser, filling out forms, and clicking buttons. The test results are compared with the expected results and recorded.

[0954] Step 5:

[0955] The server records and analyzes the test results.

[0956] The server collects the results of the executed tests, categorizes successful and failed test cases, and records the details of the results (e.g., error messages, exceptions raised, performance data, etc.) for later analysis.

[0957] Step 6:

[0958] The server generates suggested fixes when defects are detected.

[0959] If a defect is detected, the server analyzes the cause and automatically generates a fix, including specific code fixes and improvements, and notifies the user.

[0960] Step 7:

[0961] The server uses an emotion engine to recognize the user's emotion.

[0962] The server activates the emotion engine based on the user's input and operation, and the emotion engine determines the user's emotional state based on the user's facial expression, tone of voice, keyboard typing speed, etc.

[0963] Step 8:

[0964] The server customizes the test result report based on the user's emotional state.

[0965] The server customizes the content and presentation of the test result report based on the user's emotional state as determined by the emotion engine. For example, if the user expresses dissatisfaction or irritation, the server may include additional explanations and guidance in the result report to make it easier for the user to understand.

[0966] In this way, a system is provided that covers everything from inputting specifications to running tests, fixing defects, and even recognizing user emotions. This system not only improves the efficiency and quality of testing work, but also provides a testing environment that takes user emotions into consideration.

[0967] Example 2

[0968] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0969] In conventional systems, the processes of analyzing specifications, generating test items, executing tests, recording results, and proposing defect fixes were often done manually, which was time-consuming and labor-intensive. Furthermore, the system did not provide feedback that took user emotions into consideration, creating issues with the user experience. This resulted in insufficient efficiency and quality assurance for testing work.

[0970] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0971] In this invention, the server includes means for inputting specifications, means for analyzing specifications and extracting requirements, means for generating test items based on the extracted requirements, means for executing tests in accordance with the generated test items, means for recording and analyzing test results, means for generating correction suggestions when defects are detected, means for recognizing user emotions, and means for including a rule set for automatically generating test items based on the extracted requirements. This not only improves the efficiency of testing operations and quality assurance, but also makes it possible to provide feedback that takes user emotions into consideration.

[0972] A "specification" is a document that describes the functions and requirements of a system or software.

[0973] The "means for inputting specifications" refers to an interface or function that allows a user to upload specifications to the system.

[0974] A "means for analyzing specifications" is a program for analyzing the contents of specifications and extracting requirements and functions.

[0975] "Requirements" are conditions or functions that a system or software must meet.

[0976] A "means for extracting requirements" is a means for identifying and extracting necessary requirements from the specification.

[0977] A "test item" is a specific test case or scenario used to verify the functionality and requirements of a system or software.

[0978] The "means for generating test items" is a program for automatically generating test items based on the extracted requirements.

[0979] "Means for executing tests" refers to tools or programs for automatically executing tests according to the generated test items.

[0980] "Test Results" means the data and information obtained by running a test.

[0981] "Means for recording test results" refers to means for saving the results of test execution.

[0982] "Means for analyzing test results" refers to the means for analyzing recorded test results, determining success or failure, and identifying the cause.

[0983] A "defect" is a phenomenon or bug that causes a system or software to not function as specified.

[0984] A "fix proposal" is a proposal or method for improving or correcting a detected defect.

[0985] The "means for generating a proposed fix" is a program for automatically generating a proposed fix based on the detected defects.

[0986] "Means for recognizing user emotions" refers to a program or engine for analyzing and recognizing emotions from user operations and input data.

[0987] A "rule set" is a set of rules or guidelines for operating under certain criteria or conditions.

[0988] A "means including a rule set" is a setting or program that enables operation based on specific rules.

[0989] The system of the present invention automates the process from inputting specifications to analysis, generating test items, executing tests, recording results, proposing fixes for defects, and recognizing user emotions. Specific examples of each major component and process are described below.

[0990] Specification input and analysis

[0991] A user uploads a specification file through the interface. For example, in a web application, the user clicks a file selection button, selects a file such as "requirements.docx" from the local disk, and presses the upload button. The server receives this file and analyzes the content of the specification using Python's natural language processing libraries, spaCy and NLTK. The server identifies and extracts important requirements and features.

[0992] Test case generation

[0993] Based on the extracted requirements, the server automatically generates test items. For example, if a user authentication function is included in the requirements, the server generates specific test scenarios such as "Can you log in by entering correct information in the login form?" or "If you enter an incorrect password, does an error message appear?". A test generation engine (e.g., Robot Framework) is used for this process.

[0994] Running the tests

[0995] According to the generated test items, the server executes the tests using automation tools (e.g., Selenium WebDriver or Cypress). For example, the server executes each test case by programmatically operating a Google Chrome browser, entering data into a form, and clicking the submit button. The test execution results are recorded in a log file.

[0996] Recording and analyzing test results

[0997] The server collects the results of the executed tests and records them in a database, categorizing successful and failed test cases and saving detailed information for each (e.g., error messages, performance data, screenshots, etc.), allowing for easy future reference.

[0998] Bug fix suggestions

[0999] If a defect is detected during testing, the server analyzes the cause and automatically generates a correction suggestion. For example, if a database connection error occurs, the server identifies the cause and suggests a fix for the SQL query. The suggestion includes specific code examples and improvements.

[1000] Emotion engine integration

[1001] The server uses an emotion engine (e.g., Microsoft Azure Cognitive Services or Google Cloud Natural Language API) that recognizes emotions based on user actions and input. For example, when a user checks their test results, the device's camera captures their facial expressions and collects voice input. This determines the user's emotional state (e.g., "satisfied," "dissatisfied," "irritated," etc.), and if the user indicates dissatisfaction, the way the test results are displayed can be adjusted to improve the user experience.

[1002] Specific examples and prompts for the generative AI model

[1003] 1. Example of specification analysis

[1004] The user uploads the "Online Shopping System Specification."

[1005] The server receives a file called "shopping_system_spec.docx" and analyzes it using a natural language processing engine (e.g., spaCy) to extract requirements such as "user registration function," "product search function," and "cart function."

[1006] 2. Test Case Generation Example

[1007] For the requirement "User registration function", the server generates the following test items:

[1008] "Test to confirm that the registration form can be completed successfully by entering all required information."

[1009] "Test to see if an error message is displayed when an invalid email address is entered"

[1010] 3. Example prompts for the generated AI model

[1011] Prompt: Generate test scenarios for the following requirements:

[1012] Requirement: User registration function

[1013] 1. Did you enter all the required information in the registration form and register successfully?

[1014] 2. Check the error message that appears when you enter an invalid email address

[1015] This invention makes it possible to consistently automate everything from inputting specifications to analyzing them, generating test items, executing tests, recording results, proposing bug fixes, and even recognizing user emotions, thereby improving the efficiency of testing operations and enhancing quality assurance. Furthermore, by providing a highly convenient testing environment that takes user emotions into consideration, an improved user experience can be expected.

[1016] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1017] Step 1:

[1018] The user uploads a specification file (e.g., "requirements.docx") using the file selection interface of the web application, which is then sent to the server via an HTTP request.

[1019] Input: Specifications file (requirements.docx)

[1020] Output: The file sent to the server

[1021] Step 2:

[1022] The server processes the received file. First, it reads the file contents and converts them into text format. Next, it uses a natural language processing library (e.g., spaCy) to parse the specification and extract key requirements and features. This process involves data processing such as text tokenization, sentence structure analysis, and entity recognition.

[1023] Input: Received specification file

[1024] Output: Extracted requirements data (e.g., user authentication function, data storage function)

[1025] Step 3:

[1026] Based on the extracted requirements, the server automatically generates test items. For example, test items for the user authentication function such as "Can you log in by entering correct information in the login form?" and "Does an error message appear when an invalid password is entered?" are generated. An automatic test generation engine (e.g., Robot Framework) is used for this task.

[1027] Input: Extracted requirements data

[1028] Output: Generated test item list

[1029] Step 4:

[1030] The server executes tests based on the generated test items. It uses an automation tool (e.g., Selenium WebDriver) to operate a web browser (e.g., Google Chrome) to simulate form input, button clicks, etc. Test execution is controlled by a script, and results are collected in real time.

[1031] Input: Test item list

[1032] Output: Test execution results (logs, screenshots, etc.)

[1033] Step 5:

[1034] The server collects the results of the executed tests and records them in a database, categorizing successful and failed test cases and storing detailed information about each, including error messages, performance data, screenshots, etc.

[1035] Input: Test execution results

[1036] Output: Recorded test result data

[1037] Step 6:

[1038] If a test detects a defect, the server analyzes the cause and automatically generates a fix suggestion. For example, if a database connection error is detected, the server identifies the cause and generates a SQL query fix suggestion, including specific code examples and improvements.

[1039] Input: Recorded test results (failure cases)

[1040] Output: Auto-generated correction suggestions

[1041] Step 7:

[1042] The server uses an emotion engine (e.g., Microsoft Azure Cognitive Services) to recognize emotions based on user actions and inputs. For example, it captures the user's facial expressions and voice from the device's camera and microphone, and analyzes them with the emotion engine to determine the user's emotional state. If the user expresses dissatisfaction, the server adjusts the way the test results are displayed to provide information in a more convincing format.

[1043] Input: User operation data (facial expressions, voice)

[1044] Output: Analyzed user's emotional state, adjusted display content

[1045] (Application example 2)

[1046] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1047] There is a need for a system that can monitor the operation of robots operating in factories, effectively execute tests according to required specifications, and analyze the results. Furthermore, existing technologies lack the ability to quickly and specifically suggest corrections for any defects discovered, and provide feedback and improvement measures that take into consideration the emotions of the operators operating the robots. The lack of such a system can lead to reduced production efficiency and quality control issues.

[1048] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for inputting specifications, a means for analyzing specifications and extracting requirements, a means for generating test items based on the extracted requirements, a means for executing tests in accordance with the generated test items, a means for recording and analyzing test results, a means for generating correction suggestions when a defect is detected, a means for recognizing operator emotions and using this to improve system operation, a means for automatically testing and monitoring the operation of in-factory robots based on operational requirements extracted from the specifications, and a device installed on the in-factory robots. This enables quality control of robot operation and efficient production activities.

[1049] A "specification" is a document that describes detailed requirements and conditions regarding the operation and functionality of software or a system.

[1050] "Analysis" is the process of extracting necessary information and requirements from documents and data such as specifications and putting them into an understandable form.

[1051] "Requirements" are specific conditions or requirements regarding the functions and performance that a system or software must meet.

[1052] "Test items" are specific test cases or scenarios used to confirm and verify the operation and functionality of a system based on requirements.

[1053] "Test execution" is the process of actually testing the behavior of a system or robot according to the generated test items.

[1054] "Recording and analysis" refers to the process of saving the results obtained after the test is executed and analyzing the factors behind success or failure.

[1055] A "correction proposal" is a proposal for a specific correction method or improvement plan based on the cause when a defect is detected.

[1056] An "operator" is a person in charge of operating and managing robots and systems within a factory.

[1057] "Emotion recognition" is a technology that determines an operator's psychological state and emotions based on their facial expressions, voice, actions, etc.

[1058] "Operational requirements" refer to the specific operating conditions and actions that a robot or system must perform, as described in the specifications.

[1059] A "factory robot" is a mechanical device that operates within a factory to automate manufacturing and processing operations.

[1060] "Apparatus" refers to an entire machine or system designed to perform a particular function or purpose.

[1061] Specific embodiments of the present invention are as follows: This system automates the monitoring and testing of robot operations in a factory, and recognizes the emotions of the operator and provides feedback.

[1062] System configuration

[1063] The system consists of the following main components:

[1064] 1. Method of inputting specifications: Upload the operation specifications of the factory robots from the interface on the server.

[1065] 2. Analyzing specifications and extracting requirements: Analyzing the contents of specifications using natural language processing tools and extracting operational requirements. The main software used is TextBlob and spaCy.

[1066] 3. A method for generating test cases based on extracted requirements: Test scenarios are automatically generated based on the extracted requirements. Node-RED is used for this process.

[1067] 4. A method for executing tests according to the generated test items: The robots in the factory actually operate according to the generated test items. The operation tests are performed using the Robot Operating System (ROS).

[1068] 5. A means to record and analyze test results: Collect the results of the executed tests and classify them into success and failure cases. The data is recorded and analyzed using SQLAlchemy.

[1069] 6. A method to generate correction suggestions when defects are detected: Analyze the causes of defects detected in the test and automatically generate correction suggestions. Here, we use the GPT-4 API.

[1070] 7. Recognizing operator emotions and using them to improve system operation: Analyzing the emotions of the operator in real time and providing feedback. This is done using OpenAI's CLIP.

[1071] Specific Examples

[1072] Specification input and analysis

[1073] Users can upload operational specifications for factory robots to the server through the interface. The server then analyzes the uploaded specifications using TextBlob and spaCy to automatically extract operational requirements. During this process, it uses natural language processing technology to identify important requirements and functions within the specifications.

[1074] Test case generation

[1075] Based on the extracted requirements, the server automatically generates test items using Node-RED. For example, if a requirement includes checking the robot's battery, tests to check the battery status and tests to take action in the event of an abnormality are automatically generated.

[1076] Running the tests

[1077] Based on the generated test items, the server uses ROS to have the robot execute the tests. Specifically, the robot checks the battery status and performs a series of checks to see if there are any problems with its operation.

[1078] Recording and analyzing test results

[1079] The server collects and records the results of the executed tests, categorizing and analyzing successful and failed test cases using SQLAlchemy, thereby storing detailed information about each case (e.g. error messages, performance data, etc.).

[1080] Bug fix suggestions

[1081] If a defect is detected during testing, the server analyzes the cause of the defect and automatically generates correction suggestions using the GPT-4 API, including specific fixes and improvements.

[1082] Emotion engine integration

[1083] The server uses OpenAI's CLIP to analyze the operator's facial expressions and voice to determine their emotions. For example, if the operator expresses dissatisfaction or irritation while reviewing the test results, the server can customize how the feedback is displayed and provide the information in a way that is understandable to the operator.

[1084] Specific prompt examples

[1085] Example prompts for parsing specifications

[1086] What the specs say: The robot will begin its routine inspection every day at 9 a.m. The inspection will include checking the battery status, checking motor operation, and collecting sensor data.

[1087] Example prompts for emotion recognition

[1088] Operator input: I'm not happy with the test results. The robot has stopped working.

[1089] In this way, a system has been realized that consistently automates everything from inputting specifications to analysis, generating test items, running tests, recording results, proposing defect fixes, and even recognizing operator emotions. This system will enable quality control of robot operations in factories and efficient production activities.

[1090] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1091] Step 1:

[1092] The user uploads the specification from the interface on the server. The input is the operation specification of the factory robot. The output is a specification file saved on the server.

[1093] Step 2:

[1094] The server analyzes the uploaded specifications using TextBlob or spaCy to extract operational requirements. The input is the uploaded specification text. Data processing involves natural language analysis to extract requirements. The output is a list of extracted operational requirements.

[1095] Step 3:

[1096] The server uses Node-RED to automatically generate test cases based on the extracted operational requirements. The input is a list of operational requirements. Data calculations involve generating appropriate test scenarios for each requirement. The output is a list of generated test cases.

[1097] Step 4:

[1098] The server uses the generated test items to have the factory robots execute tests using ROS. The input is a list of generated test items. Specific actions performed by the robots include checking the battery status and motor operation. The output is test result data.

[1099] Step 5:

[1100] The server collects the results of the executed tests and uses SQLAlchemy to classify and record success and failure cases. The input is the test result data. Data processing involves classifying the results as success / failure and recording detailed information (e.g. error messages, performance data). The output is a classified test result report.

[1101] Step 6:

[1102] If the server detects a defect in the test results, it uses the GPT-4 API to analyze the cause of the defect and generate a correction proposal. The input is the data of the failed test case. The data calculation involves analyzing the cause of the failure and generating a correction proposal. The output is a proposal including specific correction code and improvements.

[1103] Step 7:

[1104] The server uses OpenAI's CLIP to analyze the emotions of the operator from their facial expressions and voice and provide feedback. The input is the operator's emotional data (e.g., facial expressions, voice tone). Data processing involves emotion analysis and generation of appropriate feedback content. The output is a customized feedback message for the operator.

[1105] Through the above steps, the server executes a series of automated operations, providing a system that can realize quality control of the operations of robots in the factory and emotional management of operators.

[1106] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1107] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1108] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1109] [Fourth embodiment]

[1110] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1111] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1112] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1113] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1114] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1115] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1116] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1117] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1118] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1119] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1120] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1121] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1122] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1123] System configuration

[1124] The system of the present invention automates a series of processes, from inputting specifications to analysis, generating test cases, executing tests, recording results, and proposing fixes for defects. The system consists of the following main components:

[1125] 1. Means of inputting specifications

[1126] 2. A method for analyzing specifications and extracting requirements

[1127] 3. Methods for generating test cases

[1128] 4. How to run the test

[1129] 5. Means for recording and analyzing test results

[1130] 6. Means of generating revisions

[1131] Specific Examples

[1132] Specification input and analysis

[1133] The server provides an interface to receive specifications from users. Users upload specifications in PDF or other text format. The server uses a text analysis engine to analyze the content of the specifications and extract requirements from them.

[1134] As a concrete example, suppose a user uploads a UI test specification for an Internet service. The server automatically extracts functional requirements and test scope from the specification. In the process, it uses natural language processing technology to understand the context of the specification and identify important requirements.

[1135] Test case generation

[1136] Based on the extracted requirements, the server automatically generates test cases, each of which includes specific test scenarios and test steps for a specific function or requirement.

[1137] For example, if requirements related to user registration functionality are extracted, the server generates test items such as "user registration success test" and "user registration failure test," which include specific steps such as filling in a form, clicking the submit button, and checking the success message.

[1138] Running the tests

[1139] The server executes the tests according to the generated test items. The server uses an automated test tool (e.g., Selenium) to operate the web browser and executes the tests. The execution results of each step are recorded and compared with the expected results.

[1140] For example, it automatically enters a name, email address, and password into a form, clicks the submit button, and checks whether a success message is displayed correctly. During this process, it records detailed logs for later reference.

[1141] Recording and analyzing test results

[1142] The server collects the results of the executed tests and classifies them as successes and failures. Based on the test results, the server analyzes the success rate of the tests, the type and frequency of failures, etc.

[1143] For example, if the test for user registration function is successful, the result is recorded as "success" and reflected in the analysis data. If the test fails, the detailed reason for the failure (e.g. error message, thrown exception, etc.) is recorded.

[1144] Generate correction suggestions

[1145] If a defect is detected during testing, the server generates a fix for the defect, analyzes the cause of the defect, and proposes specific fixes and code changes.

[1146] For example, if there is a problem with the validation of input data in the user registration function, the server will generate a correction suggestion called "Strengthen validation of user input data" and provide specific code examples and guidelines.

[1147] summary

[1148] By integrating the above components and processing steps, the system of the present invention automates all processes, from inputting specifications to analysis, generating test cases, executing tests, recording results, and proposing defect fixes. This system improves the efficiency of testing operations and strengthens quality assurance. It is an extremely useful tool, especially for developers and engineers with little testing experience.

[1149] The processing flow will be explained below.

[1150] Step 1:

[1151] The user uploads the specification.

[1152] Through the interface, users upload specifications for their Internet services in PDF or text format, which are then sent to a specific endpoint on the server.

[1153] Step 2:

[1154] The server analyzes the specification and extracts the requirements.

[1155] The server analyzes the received specifications using a text analysis engine (e.g., Textract). From the analysis results, it understands the content of the text and extracts requirements such as software functional requirements, user requirements, and terms of use.

[1156] Step 3:

[1157] The server generates test cases based on the extracted requirements.

[1158] The server automatically generates specific test items based on the requirements extracted from the specifications. For example, if the requirements for a user registration function are included, multiple test scenarios will be generated, such as a registration success test and an error test.

[1159] Step 4:

[1160] The server runs the test items.

[1161] The server uses an automated testing tool (e.g., Selenium) to run tests according to the generated test items, operating the web browser and performing specific operations such as filling out forms and clicking buttons, and recording the results of each operation and comparing them with the expected results.

[1162] Step 5:

[1163] The server records and analyzes the test results.

[1164] The server collects and records the results of the executed tests, categorizes successful and failed test cases, and generates detailed logs and analysis data, such as success rates, types of errors, and frequency of failures.

[1165] Step 6:

[1166] The server generates suggested fixes when defects are detected.

[1167] If a defect is detected, the server analyzes the cause and automatically generates a fix, generating a document with specific suggestions and instructions for code changes to help resolve the problem.

[1168] In this way, a system is realized that, through cooperation between the server and the user, handles everything from inputting specifications to conducting tests and correcting defects.

[1169] Example 1

[1170] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1171] Traditional software testing processes require a lot of time and effort, including manual requirements extraction, test case creation, and manual test execution. This reduces testing efficiency and increases the likelihood of human error. It also makes it difficult to quickly detect defects and propose fixes. Therefore, to solve these problems, a new system is needed to automate the entire process, from analyzing specifications to running tests, analyzing results, and proposing fixes.

[1172] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1173] In this invention, the server includes means for inputting specifications as data, means for analyzing the specifications and automatically extracting requirements, means for automatically generating test items based on the extracted requirements, means for executing tests using an automated test tool in accordance with the generated test items, means for recording test results and classifying and analyzing successes and failures, and means for generating and providing correction suggestions when defects are detected. This makes it possible to automate the entire software testing process and significantly improve efficiency and reliability.

[1174] A "specification" is a document that describes in detail the requirements, functions, design, etc. of software or a system.

[1175] "Means of data input" refers to the methods and techniques used to input data in the form of digital files into the system.

[1176] "Analysis" is the process of examining given data in detail and clarifying its structure and content.

[1177] "Requirements" refer to the conditions or specifications that software or a system must meet.

[1178] "Automatic generation" refers to the system generating test items and data based on certain algorithms and rules without manual intervention.

[1179] A "test item" is a specific test case or scenario that verifies a particular function or requirement of software or a system.

[1180] An "automated testing tool" is software that uses scripts and programs to run tests without human intervention.

[1181] "Recording" means saving the results of the executed tests and information about each step.

[1182] "Classification" is the act of grouping collected data based on specific criteria.

[1183] "Analysis" is the process of using collected data to interpret their meaning and draw conclusions.

[1184] A "defect" is a problem or defect that prevents software or a system from functioning properly.

[1185] A "proposed fix" is a specific method or proposal for solving a detected defect.

[1186] MODE FOR CARRYING OUT THE INVENTION

[1187] System configuration

[1188] The system of the present invention automates a series of processes, from inputting specifications to analysis, generating test items, executing tests, recording results, and proposing defect corrections. This system is composed of multiple components, including a server, terminals, and users.

[1189] Specification input and analysis

[1190] The server provides an interface for receiving specifications from users. Users upload specifications in PDF or other text format. After receiving the specifications, the server uses a text analysis engine (e.g., NLTK or spaCy) to analyze the content of the specifications and extract requirements from them. As a specific example, if a user uploads a specification for UI testing of an Internet service, the server analyzes it and automatically extracts functional requirements and test scope. During this process, natural language processing technology is used to understand the context of the specifications and identify important requirements.

[1191] Test case generation

[1192] Based on the extracted requirements, the server automatically generates test items. Each test item includes specific test scenarios and test steps for specific functions and requirements. For example, if requirements related to user registration functions are extracted, the server generates test items such as "User Registration Success Test" and "User Registration Failure Test." These test items include specific steps such as filling out a form, clicking a submit button, and checking for a success message.

[1193] Running the tests

[1194] The server executes the tests according to the generated test items. Using an automated testing tool (for example, Selenium), the tests are executed while operating the web browser. The results of each step are recorded and compared with the expected results. For example, the server can automatically enter a name, email address, and password into a form, click the submit button, and check whether a success message is displayed correctly. During this process, the server records detailed logs that can be referenced later.

[1195] Recording and analyzing test results

[1196] The server collects the results of the executed tests and classifies them as successes or failures. Based on the test results, the server analyzes the success rate, type of failure, frequency, etc. For example, if the test of the user registration function is successful, the result is recorded as "success" and reflected in the analysis data. If the test fails, the detailed reason for the failure (e.g., error message, exception occurred, etc.) is recorded.

[1197] Generate correction suggestions

[1198] If a defect is detected during testing, the server generates a correction proposal to address the defect. The server analyzes the cause of the defect and proposes specific correction methods and code changes. For example, if there is a problem with the input data validation in the user registration function, the server generates a correction proposal called "Strengthen validation of user input data" and provides specific code examples and guidelines.

[1199] Specific prompt examples

[1200] When a user uploads a specification saying they want to "test the new user registration function," the following prompt is input to the generative AI model:

[1201] "For the new user registration function, please generate test cases based on the following requirements:

[1202] 1. Enter your username, email address, and password, then click the Register button to display a success message.

[1203] 2. Display an error message if the user name is not entered.

[1204] 3. If the email address is not in the correct format, display an error message.

[1205] By inputting this prompt into a generative AI model, the model will automatically generate test items based on the requirements.

[1206] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1207] Step 1:

[1208] The user uploads the specification as data. Specifically, the user opens a web browser, accesses the specified form, selects the specification file (e.g., PDF), and clicks the upload button. This input process causes the server to obtain the specification file.

[1209] Step 2:

[1210] The server receives the uploaded specification file and temporarily stores it. Next, it uses a text analysis engine (e.g., NLTK or spaCy) to convert the content of the specification into text format and begins analyzing it. The server tokenizes each sentence and extracts target keywords and structured information. Based on the results of this analysis, a list of important requirements is extracted from the specification.

[1211] Step 3:

[1212] The server automatically generates test cases based on the extracted requirements. It uses a generative AI model or algorithm to analyze the input requirements data and generate specific test scenarios and test steps. For example, if requirements related to user registration functionality are extracted, test cases such as "User Registration Success Test" and "User Registration Failure Test" are generated. The output includes test scenarios and corresponding detailed steps.

[1213] Step 4:

[1214] The server executes the tests according to the generated test items. Specifically, the server uses an automated testing tool (e.g., Selenium) to execute test scenarios while operating a web browser. At each test step, the server automatically inputs forms and clicks buttons, and compares the expected results with the actual results. The test execution results based on this input and output are recorded as a log.

[1215] Step 5:

[1216] The server records and analyzes the results of the executed tests. The server logs the results of each test step and classifies them as success or failure. For example, if user registration is completed successfully, it is recorded as a "success," and if an error occurs, it is recorded as a "failure." This analysis data includes the success rate, the frequency of failures, details of error messages, etc.

[1217] Step 6:

[1218] If a defect is detected during testing, the server generates a correction proposal to address the defect. The server analyzes the cause of the defect and automatically generates specific correction methods and code change suggestions. For example, if there is a problem with input data validation, the server generates a correction proposal called "Strengthen input data validation" and provides specific code examples and guidelines. This correction proposal is provided to the user along with the test results.

[1219] (Application example 1)

[1220] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1221] Because software for autonomous vehicles requires safety and precision, rigorous testing is necessary when updating software or adding new features. However, traditional manual testing is burdensome and inefficient. Furthermore, the wide scope of testing can lead to human error or insufficient testing. This can reduce the reliability of the software and potentially cause safety issues. This invention was developed to solve these problems, and aims to fully automate the software testing process for autonomous vehicles, improving efficiency and accuracy.

[1222] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1223] In this invention, the server includes means for inputting specifications, means for analyzing specifications and extracting requirements, means for generating test items based on the extracted requirements, means for executing tests in accordance with the generated test items, means for recording and analyzing test results, means for generating a correction plan when a defect is detected, means for uploading software specifications for autonomous vehicles, means for automatically extracting requirements from specifications, means for automatically generating and executing test items, and means for analyzing the results in detail and, if necessary, proposing a correction plan for the defect. This automates the entire process of extracting requirements from specifications, generating test items, executing them, analyzing the results, and proposing a correction plan for the defect, thereby enabling the efficiency and accuracy of software testing for autonomous vehicles to be improved.

[1224] A "specification" is a document that describes in detail the functions and requirements of software or a system.

[1225] "Requirements" are the conditions and functions that a system or software must satisfy, extracted from the specifications.

[1226] "Test items" are specific test scenarios or steps that are generated based on extracted requirements and are used to verify specific functions or conditions.

[1227] An "autonomous vehicle" is a vehicle that completely automates driving operations without human intervention, and automatically makes decisions and operates in accordance with the surrounding situation.

[1228] "Upload" refers to the operation of transferring files or data from a local environment to a server.

[1229] "Extraction" refers to the process of selecting specific information or requirements from a document such as a specification.

[1230] "Generation" means creating new test items or scenarios from given data and conditions.

[1231] "Execution" refers to the process of automatically performing tests according to the generated test items and obtaining the results.

[1232] "Recording and analysis" means saving the test results and analyzing them in detail to identify the causes and patterns of success and failure.

[1233] A "fix" is a specific improvement plan or code modification to resolve the cause of a test failure or defect.

[1234] A "proposal" is a way of presenting a solution or improvement to a problem and encouraging its implementation.

[1235] The system of this invention can automatically perform operations from inputting specifications to analyzing them, generating test items, executing tests, recording results, and proposing defect fixes. It can be applied to software updates and adding new functions to autonomous vehicles. This system can improve the reliability and safety of software. The detailed configuration and operation of the system are described below.

[1236] System configuration

[1237] The system mainly consists of the following components:

[1238] 1. Component for inputting specifications

[1239] 2. A component that analyzes specifications and extracts requirements

[1240] 3. A component that generates test cases based on the extracted requirements

[1241] 4. Components that execute tests according to the test items

[1242] 5. Component for recording and analyzing test results

[1243] 6. A component that generates suggested fixes when defects are detected

[1244] Specification input and analysis

[1245] The server provides an interface to receive specifications from users. Users upload specifications in PDF or other text format. The server extracts the text from the specifications using pdfminer and parses and extracts requirements using a natural language processing library. This requirements extraction utilizes AI models and generative AI models.

[1246] Test case generation

[1247] Based on the extracted requirements, the server automatically generates test cases. The generated test cases are highly accurate because they create specific test scenarios for specific functions and operations. In this process, test cases are generated using the generated rule set.

[1248] Running the tests

[1249] The server runs automated tests using Selenium according to the test items. The server simulates each operation and compares the expected results with the actual results. During this process, the execution results of each step are recorded and detailed logs are collected.

[1250] Recording and analyzing test results

[1251] The server collects the executed test results and classifies the test cases as successful and failed. The recorded results are analyzed to determine the success rate and causes of failures. This analysis is done automatically using a generative AI model.

[1252] Generate correction suggestions

[1253] If a defect is detected, the server generates a fix for the defect, suggesting specific fixes and code changes and providing necessary feedback. These fix suggestions are also generated using natural language processing and AI models.

[1254] Specific examples

[1255] For example, in the case of a new navigation system update for a self-driving vehicle, the system might receive and act on the following prompt:

[1256] Example prompt sentence:

[1257] The specification states:

[1258] The new navigation system automatically calculates and displays the optimal route when the user enters their destination.

[1259] It uses cameras and sensors to perceive the surrounding terrain in real time and dynamically update the route.

[1260] It provides location information for charging stations and suggests the best charging point based on the remaining charge.

[1261] Based on this specification, please implement a program that generates test cases and automates each test.

[1262] This system will improve the efficiency of software testing for autonomous vehicles and enable highly accurate testing. Cloud servers (such as Amazon EC2) are suitable for the hardware used.

[1263] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1264] Step 1:

[1265] The user uploads software specifications to the server in PDF or text format. To do this, the user selects and submits the file using a specified interface. The input data is the software specifications, and the server receives and stores this file. This prepares the specification data for analysis by the system.

[1266] Step 2:

[1267] The server extracts text from the uploaded PDF using the pdfminer library. This text is the input data, and the extracted text is analyzed using a natural language processing library. The text data to be analyzed is the content of the specification, and preprocessing is performed to extract requirements from it. This allows the system to obtain text data from the specification.

[1268] Step 3:

[1269] The server uses the extracted text data to identify the requirements of the specification using a generative AI model. Specifically, it analyzes the text using a natural language processing library and extracts the functional and performance requirements contained in the specification. The input data is all text within the scope of the analysis, and the output is the data listed as requirements. Through this step, the system lists the requirements of the software specification.

[1270] Step 4:

[1271] Based on the extracted requirements, the server automatically generates test cases, applying a predefined set of rules to generate specific test scenarios. The input data is a list of requirements, and the output is the corresponding specific test cases and scenarios. The server stores the generated test cases and scenarios in a database, allowing the system to prepare various scenarios for test execution.

[1272] Step 5:

[1273] The server executes tests based on test items created using Selenium. Specifically, it executes defined test scenarios sequentially while automatically operating the browser. The input data are test items and test scenarios, and the output is the execution results of each test step. This allows the system to automatically execute software tests on actual devices and collect the results.

[1274] Step 6:

[1275] The server collects the test results executed by Selenium and classifies them as success or failure. The input data is the execution result log, and the output is the classification results of success or failure and detailed analysis data. The server analyzes this data and generates a log that clearly shows the success rate and details of any errors. This allows the system to record and analyze test results in detail.

[1276] Step 7:

[1277] If a defect is detected, the server uses a generative AI model to generate a fix. Specifically, this process involves analyzing failed test cases and logs to identify the cause and solution. The input data is a detailed log of the execution results, and the output is specific fixes and code change suggestions. This allows the system to provide specific actions to fix the software defect.

[1278] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1279] System configuration

[1280] The system of the present invention automates the process from inputting specifications to analyzing them, generating test cases, executing tests, recording results, proposing bug fixes, and recognizing user emotions. The system consists of the following main components:

[1281] 1. Means of inputting specifications

[1282] 2. A method for analyzing specifications and extracting requirements

[1283] 3. A method for generating test cases based on extracted requirements

[1284] 4. How to execute tests according to the generated test items

[1285] 5. Means for recording and analyzing test results

[1286] 6. A means of generating suggested fixes when defects are detected

[1287] 7. Emotion engine for recognizing user emotions

[1288] Specific Examples

[1289] Specification input and analysis

[1290] Users can upload specifications through the interface, and once they are sent to the server, the server uses a text analysis engine to analyze the content of the specifications and automatically extract requirements, leveraging natural language processing technology to identify key requirements and features within the specifications.

[1291] Test case generation

[1292] Based on the extracted requirements, the server automatically generates test items. For example, if a user registration function is included in the requirements, the server automatically generates test scenarios for this function, including tests for successful registration and tests for checking error messages.

[1293] Running the tests

[1294] Based on the generated test items, the server runs the tests using an automated tool that operates the web browser, filling out forms, clicking buttons, etc. The results are compared with the expected results and logged.

[1295] Recording and analyzing test results

[1296] The server collects and records the results of the tests that have been executed, categorizing successful and failed test cases and storing detailed information about each case (e.g., error messages, performance data, etc.).

[1297] Bug fix suggestions

[1298] If a defect is detected during testing, the server analyzes the cause of the defect and automatically generates a fix proposal, which includes specific code fixes and improvements.

[1299] Emotion engine integration

[1300] The server includes an emotion engine that recognizes emotions based on user operations and inputs. The emotion engine analyzes facial expressions, voice, keyboard typing speed, etc. when the user uploads specifications and checks test results.

[1301] For example, when a user checks their test results, the emotion engine analyzes their facial expressions and tone of voice to determine the emotional state they are in. If the user shows signs of dissatisfaction or irritation, the server can customize the content and presentation of the test result report to provide information in a way that is convincing to the user.

[1302] In this way, a system is realized that consistently automates everything from inputting specifications to analyzing them, generating test cases, executing tests, recording results, proposing bug fixes, and even recognizing user emotions. This system not only improves the efficiency and quality assurance of testing operations, but also provides a highly convenient testing environment that takes user emotions into consideration.

[1303] The processing flow will be explained below.

[1304] Step 1:

[1305] The user uploads the specification.

[1306] Through the interface, users upload specifications for their Internet services in PDF or text format, which are then sent to a designated endpoint on the server.

[1307] Step 2:

[1308] The server analyzes the specification and extracts the requirements.

[1309] The server analyzes the received specifications using a text analysis engine (e.g., Textract). From the analysis results, it understands the content of the specifications and extracts requirements such as software functional requirements, user requirements, and terms of use.

[1310] Step 3:

[1311] The server generates test cases based on the extracted requirements.

[1312] The server automatically generates test items based on the requirements extracted from the specifications. Specifically, if the requirements for the user registration function are included, it generates test scenarios such as a registration success test and an error test.

[1313] Step 4:

[1314] The server runs the test items.

[1315] The server uses an automated testing tool (e.g., Selenium) to execute tests according to the generated test items. Specific operations for executing tests include opening a web browser, filling out forms, and clicking buttons. The test results are compared with the expected results and recorded.

[1316] Step 5:

[1317] The server records and analyzes the test results.

[1318] The server collects the results of the executed tests, categorizes successful and failed test cases, and records the details of the results (e.g., error messages, exceptions raised, performance data, etc.) for later analysis.

[1319] Step 6:

[1320] The server generates suggested fixes when defects are detected.

[1321] If a defect is detected, the server analyzes the cause and automatically generates a fix, including specific code fixes and improvements, and notifies the user.

[1322] Step 7:

[1323] The server uses an emotion engine to recognize the user's emotion.

[1324] The server activates the emotion engine based on the user's input and operation, and the emotion engine determines the user's emotional state based on the user's facial expression, tone of voice, keyboard typing speed, etc.

[1325] Step 8:

[1326] The server customizes the test result report based on the user's emotional state.

[1327] The server customizes the content and presentation of the test result report based on the user's emotional state as determined by the emotion engine. For example, if the user expresses dissatisfaction or irritation, the server may include additional explanations and guidance in the result report to make it easier for the user to understand.

[1328] In this way, a system is provided that covers everything from inputting specifications to running tests, fixing defects, and even recognizing user emotions. This system not only improves the efficiency and quality of testing work, but also provides a testing environment that takes user emotions into consideration.

[1329] Example 2

[1330] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1331] In conventional systems, the processes of analyzing specifications, generating test items, executing tests, recording results, and proposing defect fixes were often done manually, which was time-consuming and labor-intensive. Furthermore, the system did not provide feedback that took user emotions into consideration, creating issues with the user experience. This resulted in insufficient efficiency and quality assurance for testing work.

[1332] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1333] In this invention, the server includes means for inputting specifications, means for analyzing specifications and extracting requirements, means for generating test items based on the extracted requirements, means for executing tests in accordance with the generated test items, means for recording and analyzing test results, means for generating correction suggestions when defects are detected, means for recognizing user emotions, and means for including a rule set for automatically generating test items based on the extracted requirements. This not only improves the efficiency of testing operations and quality assurance, but also makes it possible to provide feedback that takes user emotions into consideration.

[1334] A "specification" is a document that describes the functions and requirements of a system or software.

[1335] The "means for inputting specifications" refers to an interface or function that allows a user to upload specifications to the system.

[1336] A "means for analyzing specifications" is a program for analyzing the contents of specifications and extracting requirements and functions.

[1337] "Requirements" are conditions or functions that a system or software must meet.

[1338] A "means for extracting requirements" is a means for identifying and extracting necessary requirements from the specification.

[1339] A "test item" is a specific test case or scenario used to verify the functionality and requirements of a system or software.

[1340] The "means for generating test items" is a program for automatically generating test items based on the extracted requirements.

[1341] "Means for executing tests" refers to tools or programs for automatically executing tests according to the generated test items.

[1342] "Test Results" means the data and information obtained by running a test.

[1343] "Means for recording test results" refers to means for saving the results of test execution.

[1344] "Means for analyzing test results" refers to the means for analyzing recorded test results, determining success or failure, and identifying the cause.

[1345] A "defect" is a phenomenon or bug that causes a system or software to not function as specified.

[1346] A "fix proposal" is a proposal or method for improving or correcting a detected defect.

[1347] The "means for generating a proposed fix" is a program for automatically generating a proposed fix based on the detected defects.

[1348] "Means for recognizing user emotions" refers to a program or engine for analyzing and recognizing emotions from user operations and input data.

[1349] A "rule set" is a set of rules or guidelines for operating under certain criteria or conditions.

[1350] A "means including a rule set" is a setting or program that enables operation based on specific rules.

[1351] The system of the present invention automates the process from inputting specifications to analysis, generating test items, executing tests, recording results, proposing fixes for defects, and recognizing user emotions. Specific examples of each major component and process are described below.

[1352] Specification input and analysis

[1353] A user uploads a specification file through the interface. For example, in a web application, the user clicks a file selection button, selects a file such as "requirements.docx" from the local disk, and presses the upload button. The server receives this file and analyzes the content of the specification using Python's natural language processing libraries, spaCy and NLTK. The server identifies and extracts important requirements and features.

[1354] Test case generation

[1355] Based on the extracted requirements, the server automatically generates test items. For example, if a user authentication function is included in the requirements, the server generates specific test scenarios such as "Can you log in by entering correct information in the login form?" or "If you enter an incorrect password, does an error message appear?". A test generation engine (e.g., Robot Framework) is used for this process.

[1356] Running the tests

[1357] According to the generated test items, the server executes the tests using automation tools (e.g., Selenium WebDriver or Cypress). For example, the server executes each test case by programmatically operating a Google Chrome browser, entering data into a form, and clicking the submit button. The test execution results are recorded in a log file.

[1358] Recording and analyzing test results

[1359] The server collects the results of the executed tests and records them in a database, categorizing successful and failed test cases and saving detailed information for each (e.g., error messages, performance data, screenshots, etc.), allowing for easy future reference.

[1360] Bug fix suggestions

[1361] If a defect is detected during testing, the server analyzes the cause and automatically generates a correction suggestion. For example, if a database connection error occurs, the server identifies the cause and suggests a fix for the SQL query. The suggestion includes specific code examples and improvements.

[1362] Emotion engine integration

[1363] The server uses an emotion engine (e.g., Microsoft Azure Cognitive Services or Google Cloud Natural Language API) that recognizes emotions based on user actions and input. For example, when a user checks their test results, the device's camera captures their facial expressions and collects voice input. This determines the user's emotional state (e.g., "satisfied," "dissatisfied," "irritated," etc.), and if the user indicates dissatisfaction, the way the test results are displayed can be adjusted to improve the user experience.

[1364] Specific examples and prompts for the generative AI model

[1365] 1. Example of specification analysis

[1366] The user uploads the "Online Shopping System Specification."

[1367] The server receives a file called "shopping_system_spec.docx" and analyzes it using a natural language processing engine (e.g., spaCy) to extract requirements such as "user registration function," "product search function," and "cart function."

[1368] 2. Test Case Generation Example

[1369] For the requirement "User registration function", the server generates the following test items:

[1370] "Test to confirm that the registration form can be completed successfully by entering all required information."

[1371] "Test to see if an error message is displayed when an invalid email address is entered"

[1372] 3. Example prompts for the generated AI model

[1373] Prompt: Generate test scenarios for the following requirements:

[1374] Requirement: User registration function

[1375] 1. Did you enter all the required information in the registration form and register successfully?

[1376] 2. Check the error message that appears when you enter an invalid email address

[1377] This invention makes it possible to consistently automate everything from inputting specifications to analyzing them, generating test items, executing tests, recording results, proposing bug fixes, and even recognizing user emotions, thereby improving the efficiency of testing operations and enhancing quality assurance. Furthermore, by providing a highly convenient testing environment that takes user emotions into consideration, an improved user experience can be expected.

[1378] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1379] Step 1:

[1380] The user uploads a specification file (e.g., "requirements.docx") using the file selection interface of the web application, which is then sent to the server via an HTTP request.

[1381] Input: Specifications file (requirements.docx)

[1382] Output: The file sent to the server

[1383] Step 2:

[1384] The server processes the received file. First, it reads the file contents and converts them into text format. Next, it uses a natural language processing library (e.g., spaCy) to parse the specification and extract key requirements and features. This process involves data processing such as text tokenization, sentence structure analysis, and entity recognition.

[1385] Input: Received specification file

[1386] Output: Extracted requirements data (e.g., user authentication function, data storage function)

[1387] Step 3:

[1388] Based on the extracted requirements, the server automatically generates test items. For example, test items for the user authentication function such as "Can you log in by entering correct information in the login form?" and "Does an error message appear when an invalid password is entered?" are generated. An automatic test generation engine (e.g., Robot Framework) is used for this task.

[1389] Input: Extracted requirements data

[1390] Output: Generated test item list

[1391] Step 4:

[1392] The server executes tests based on the generated test items. It uses an automation tool (e.g., Selenium WebDriver) to operate a web browser (e.g., Google Chrome) to simulate form input, button clicks, etc. Test execution is controlled by a script, and results are collected in real time.

[1393] Input: Test item list

[1394] Output: Test execution results (logs, screenshots, etc.)

[1395] Step 5:

[1396] The server collects the results of the executed tests and records them in a database, categorizing successful and failed test cases and storing detailed information about each, including error messages, performance data, screenshots, etc.

[1397] Input: Test execution results

[1398] Output: Recorded test result data

[1399] Step 6:

[1400] If a test detects a defect, the server analyzes the cause and automatically generates a fix suggestion. For example, if a database connection error is detected, the server identifies the cause and generates a SQL query fix suggestion, including specific code examples and improvements.

[1401] Input: Recorded test results (failure cases)

[1402] Output: Auto-generated correction suggestions

[1403] Step 7:

[1404] The server uses an emotion engine (e.g., Microsoft Azure Cognitive Services) to recognize emotions based on user actions and inputs. For example, it captures the user's facial expressions and voice from the device's camera and microphone, and analyzes them with the emotion engine to determine the user's emotional state. If the user expresses dissatisfaction, the server adjusts the way the test results are displayed to provide information in a more convincing format.

[1405] Input: User operation data (facial expressions, voice)

[1406] Output: Analyzed user's emotional state, adjusted display content

[1407] (Application example 2)

[1408] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1409] There is a need for a system that can monitor the operation of robots operating in factories, effectively execute tests according to required specifications, and analyze the results. Furthermore, existing technologies lack the ability to quickly and specifically suggest corrections for any defects discovered, and provide feedback and improvement measures that take into consideration the emotions of the operators operating the robots. The lack of such a system can lead to reduced production efficiency and quality control issues.

[1410] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for inputting specifications, a means for analyzing specifications and extracting requirements, a means for generating test items based on the extracted requirements, a means for executing tests in accordance with the generated test items, a means for recording and analyzing test results, a means for generating correction suggestions when a defect is detected, a means for recognizing operator emotions and using this to improve system operation, a means for automatically testing and monitoring the operation of in-factory robots based on operational requirements extracted from the specifications, and a device installed on the in-factory robots. This enables quality control of robot operation and efficient production activities.

[1411] A "specification" is a document that describes detailed requirements and conditions regarding the operation and functionality of software or a system.

[1412] "Analysis" is the process of extracting necessary information and requirements from documents and data such as specifications and putting them into an understandable form.

[1413] "Requirements" are specific conditions or requirements regarding the functions and performance that a system or software must meet.

[1414] "Test items" are specific test cases or scenarios used to confirm and verify the operation and functionality of a system based on requirements.

[1415] "Test execution" is the process of actually testing the behavior of a system or robot according to the generated test items.

[1416] "Recording and analysis" refers to the process of saving the results obtained after the test is executed and analyzing the factors behind success or failure.

[1417] A "correction proposal" is a proposal for a specific correction method or improvement plan based on the cause when a defect is detected.

[1418] An "operator" is a person in charge of operating and managing robots and systems within a factory.

[1419] "Emotion recognition" is a technology that determines an operator's psychological state and emotions based on their facial expressions, voice, actions, etc.

[1420] "Operational requirements" refer to the specific operating conditions and actions that a robot or system must perform, as described in the specifications.

[1421] A "factory robot" is a mechanical device that operates within a factory to automate manufacturing and processing operations.

[1422] "Apparatus" refers to an entire machine or system designed to perform a particular function or purpose.

[1423] Specific embodiments of the present invention are as follows: This system automates the monitoring and testing of robot operations in a factory, and recognizes the emotions of the operator and provides feedback.

[1424] System configuration

[1425] The system consists of the following main components:

[1426] 1. Method of inputting specifications: Upload the operation specifications of the factory robots from the interface on the server.

[1427] 2. Analyzing specifications and extracting requirements: Analyzing the contents of specifications using natural language processing tools and extracting operational requirements. The main software used is TextBlob and spaCy.

[1428] 3. A method for generating test cases based on extracted requirements: Test scenarios are automatically generated based on the extracted requirements. Node-RED is used for this process.

[1429] 4. A method for executing tests according to the generated test items: The robots in the factory actually operate according to the generated test items. The operation tests are performed using the Robot Operating System (ROS).

[1430] 5. A means to record and analyze test results: Collect the results of the executed tests and classify them into success and failure cases. The data is recorded and analyzed using SQLAlchemy.

[1431] 6. A method to generate correction suggestions when defects are detected: Analyze the causes of defects detected in the test and automatically generate correction suggestions. Here, we use the GPT-4 API.

[1432] 7. Recognizing operator emotions and using them to improve system operation: Analyzing the emotions of the operator in real time and providing feedback. This is done using OpenAI's CLIP.

[1433] Specific Examples

[1434] Specification input and analysis

[1435] Users can upload operational specifications for factory robots to the server through the interface. The server then analyzes the uploaded specifications using TextBlob and spaCy to automatically extract operational requirements. During this process, it uses natural language processing technology to identify important requirements and functions within the specifications.

[1436] Test case generation

[1437] Based on the extracted requirements, the server automatically generates test items using Node-RED. For example, if a requirement includes checking the robot's battery, tests to check the battery status and tests to take action in the event of an abnormality are automatically generated.

[1438] Running the tests

[1439] Based on the generated test items, the server uses ROS to have the robot execute the tests. Specifically, the robot checks the battery status and performs a series of checks to see if there are any problems with its operation.

[1440] Recording and analyzing test results

[1441] The server collects and records the results of the executed tests, categorizing and analyzing successful and failed test cases using SQLAlchemy, thereby storing detailed information about each case (e.g. error messages, performance data, etc.).

[1442] Bug fix suggestions

[1443] If a defect is detected during testing, the server analyzes the cause of the defect and automatically generates correction suggestions using the GPT-4 API, including specific fixes and improvements.

[1444] Emotion engine integration

[1445] The server uses OpenAI's CLIP to analyze the operator's facial expressions and voice to determine their emotions. For example, if the operator expresses dissatisfaction or irritation while reviewing the test results, the server can customize how the feedback is displayed and provide the information in a way that is understandable to the operator.

[1446] Specific prompt examples

[1447] Example prompts for parsing specifications

[1448] What the specs say: The robot will begin its routine inspection every day at 9 a.m. The inspection will include checking the battery status, checking motor operation, and collecting sensor data.

[1449] Example prompts for emotion recognition

[1450] Operator input: I'm not happy with the test results. The robot has stopped working.

[1451] In this way, a system has been realized that consistently automates everything from inputting specifications to analysis, generating test items, running tests, recording results, proposing defect fixes, and even recognizing operator emotions. This system will enable quality control of robot operations in factories and efficient production activities.

[1452] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1453] Step 1:

[1454] The user uploads the specification from the interface on the server. The input is the operation specification of the factory robot. The output is a specification file saved on the server.

[1455] Step 2:

[1456] The server analyzes the uploaded specifications using TextBlob or spaCy to extract operational requirements. The input is the uploaded specification text. Data processing involves natural language analysis to extract requirements. The output is a list of extracted operational requirements.

[1457] Step 3:

[1458] The server uses Node-RED to automatically generate test cases based on the extracted operational requirements. The input is a list of operational requirements. Data calculations involve generating appropriate test scenarios for each requirement. The output is a list of generated test cases.

[1459] Step 4:

[1460] The server uses the generated test items to have the factory robots execute tests using ROS. The input is a list of generated test items. Specific actions performed by the robots include checking the battery status and motor operation. The output is test result data.

[1461] Step 5:

[1462] The server collects the results of the executed tests and uses SQLAlchemy to classify and record success and failure cases. The input is the test result data. Data processing involves classifying the results as success / failure and recording detailed information (e.g. error messages, performance data). The output is a classified test result report.

[1463] Step 6:

[1464] If the server detects a defect in the test results, it uses the GPT-4 API to analyze the cause of the defect and generate a correction proposal. The input is the data of the failed test case. The data calculation involves analyzing the cause of the failure and generating a correction proposal. The output is a proposal including specific correction code and improvements.

[1465] Step 7:

[1466] The server uses OpenAI's CLIP to analyze the emotions of the operator from their facial expressions and voice and provide feedback. The input is the operator's emotional data (e.g., facial expressions, voice tone). Data processing involves emotion analysis and generation of appropriate feedback content. The output is a customized feedback message for the operator.

[1467] Through the above steps, the server executes a series of automated operations, providing a system that can realize quality control of the operations of robots in the factory and emotional management of operators.

[1468] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1469] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1470] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1471] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1472] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1473] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1474] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1475] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1476] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1477] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1478] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1479] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1480] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1481] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1482] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1483] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1484] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1485] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1486] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1487] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1488] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1489] The following is further disclosed regarding the above embodiment.

[1490] (Claim 1)

[1491] a means for inputting specifications;

[1492] A means of analyzing specifications and extracting requirements;

[1493] A means for generating test cases based on the extracted requirements;

[1494] A means for executing tests according to the generated test items;

[1495] A means of recording and analyzing test results;

[1496] means for generating suggested fixes when defects are detected;

[1497] A system including:

[1498] (Claim 2)

[1499] 10. The system of claim 1, further comprising a rule set for automatically generating test cases based on the extracted requirements.

[1500] (Claim 3)

[1501] 10. The system of claim 1, further comprising means for collecting test results and categorizing successful and failed test cases.

[1502] "Example 1"

[1503] (Claim 1)

[1504] a means for inputting the specification as data;

[1505] A means to analyze specifications and automatically extract requirements;

[1506] A means for automatically generating test cases based on the extracted requirements;

[1507] A means for executing tests using an automated test tool according to the generated test items;

[1508] A means of recording test results and categorizing and analyzing successes and failures;

[1509] means for generating and providing suggested fixes when defects are detected;

[1510] A system including:

[1511] (Claim 2)

[1512] 10. The system of claim 1, further comprising an algorithm for automatically generating test cases based on the extracted requirements.

[1513] (Claim 3)

[1514] 10. The system of claim 1, including means for recording test results in a database and for providing detailed analysis of successes and failures.

[1515] "Application Example 1"

[1516] (Claim 1)

[1517] a means for inputting specifications;

[1518] A means of analyzing specifications and extracting requirements;

[1519] A means for generating test cases based on the extracted requirements;

[1520] A means for executing tests according to the generated test items;

[1521] A means of recording and analyzing test results;

[1522] means for generating suggested fixes when defects are detected;

[1523] a means for uploading software specifications for the autonomous vehicle;

[1524] A means of automatically extracting requirements from specifications;

[1525] A means of automatically generating and executing test cases,

[1526] A means to analyze the results in detail and, if necessary, propose solutions to fix the problems.

[1527] A system including:

[1528] (Claim 2)

[1529] 10. The system of claim 1, further comprising a rule set for automatically generating test cases based on the extracted requirements.

[1530] (Claim 3)

[1531] 10. The system of claim 1, further comprising means for collecting test results, categorizing successful and failed test cases, and analyzing the results to generate suggested modifications.

[1532] "Example 2: Combining Emotion Engines"

[1533] (Claim 1)

[1534] a means for inputting specifications;

[1535] A means of analyzing specifications and extracting requirements;

[1536] A means for generating test cases based on the extracted requirements;

[1537] A means for executing tests according to the generated test items;

[1538] A means of recording and analyzing test results;

[1539] means for generating suggested fixes when defects are detected;

[1540] means for recognizing a user's emotion;

[1541] a means including a rule set for automatically generating test cases based on the extracted requirements;

[1542] A system including:

[1543] (Claim 2)

[1544] 10. The system of claim 1, further comprising means for collecting test results and categorizing successful and failed test cases.

[1545] (Claim 3)

[1546] 10. The system of claim 1, further comprising means for automatically adjusting the display of test results based on user emotions.

[1547] "Application example 2 when combining emotion engines"

[1548] (Claim 1)

[1549] a means for inputting specifications;

[1550] A means of analyzing specifications and extracting requirements;

[1551] A means for generating test cases based on the extracted requirements;

[1552] A means for executing tests according to the generated test items;

[1553] A means of recording and analyzing test results;

[1554] means for generating suggested fixes when defects are detected;

[1555] A means of recognizing operator emotions and using them to improve system operation;

[1556] a means for automatically testing and monitoring the operation of a factory robot based on operational requirements extracted from the specification;

[1557] Equipment installed on robots in factories,

[1558] A system including:

[1559] (Claim 2)

[1560] 10. The system of claim 1, further comprising a rule set for automatically generating test cases based on the extracted requirements.

[1561] (Claim 3)

[1562] 10. The system of claim 1, further comprising means for collecting test results and categorizing successful and failed test cases. [Explanation of symbols]

[1563] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for inputting specifications; A means of analyzing specifications and extracting requirements; A means for generating test cases based on the extracted requirements; A means for executing tests according to the generated test items; a means of recording and analyzing test results; means for generating suggested fixes when defects are detected; A system including:

2. The system according to claim 1 , further comprising a rule set for automatically generating test cases based on the extracted requirements.

3. 10. The system of claim 1, further comprising means for collecting test results and categorizing successful and failed test cases.

Images