System

Abstract

To provide a system.SOLUTION: The system includes: means for acquiring voice data; means for converting the voice data into texts by using a voice recognition technique; analyzing the text data by using a natural language processing technique and extracting a summary; and means for automatically generating minutes on the basis of the extracted requirement; and providing the generated minutes to a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, development managers at system integrators and other organizations face the challenges of enormous manual work and quality control. Specific tasks include generating meeting minutes and extracting requirements from interviews, creating RFPs and specifications, creating UIs and incorporating customer feedback, coding, and creating tests. These tasks consume a large amount of time and effort, and also carry the risk of quality degradation due to human error. [Means for solving the problem]

[0005] This invention provides a means to solve the above problems by automatically generating minutes and extracting requirements from interview content, automatically creating RFPs and specifications, automatically creating UIs and incorporating customer feedback, and automatically coding and creating tests. This significantly reduces the workload of development managers and improves the efficiency and quality of system development. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50.

[0019] 0 are connected to the bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0021] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0026] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0027] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0028] "Example 1"

[0029] In one embodiment of the present invention, the system automatically generates meeting minutes and extracts requirements from the contents of interviews. Specifically, the system converts the interview contents into text using speech recognition technology and extracts requirements from the text using natural language processing technology. For example, the system converts the audio data during the interview into text in real time, and extracts keywords and phrases from the text to create requirements.

[0030] "Example 2"

[0031] The system also automatically creates RFPs and specifications. Specifically, it generates RFPs and specifications using templates based on the extracted requirements. For example, it automatically fills in each item in the template based on the requirements to generate the RFP and specifications.

[0032] "Example 3"

[0033] Furthermore, the system can automatically create a UI and incorporate customer feedback. Specifically, the system automatically generates a UI based on requirements and then updates the UI based on customer feedback. For example, the system can generate an initial UI design based on requirements and then refine the UI based on customer feedback.

[0034] "Example 4"

[0035] The system also enables automatic coding and test creation. Specifically, it uses AI technology to generate code and create test cases. For example, AI generates code based on requirements and simultaneously creates test cases. This allows developers to significantly reduce the effort required for coding and test creation.

[0036] The processing flow of each embodiment will be described below.

[0037] "Example 1"

[0038] Step 1: The system converts the audio data being heard into text in real time using voice recognition technology.

[0039] Step 2: The system then uses natural language processing techniques to extract keywords and phrases from the text, which automatically extracts requirements.

[0040] Step 3: Finally, minutes are generated based on the extracted requirements. This automatically generates minutes and extracts requirements from the interview content.

[0041] "Example 2"

[0042] Step 1: The system selects an RFP or specification template based on the extracted requirements. Step 2: Next, each item in the selected template is automatically filled in based on the requirements. Step 3: Finally, the filled-in template is output as an RFP or specification. This allows the RFP or specification to be automatically created.

[0043] "Example 3"

[0044] Step 1: The system automatically generates a UI based on the requirements, using AI technology.

[0045] Step 2: Next, the system receives feedback from the customer.

[0046] Step 3: Finally, update the UI based on the feedback received, which automates the creation of the UI and incorporates customer feedback.

[0047] "Example 4"

[0048] Step 1: The system generates code based on the requirements. This is done using AI technology.

[0049] Step 2: At the same time, the system generates test cases, also using AI technology.

[0050] Step 3: Finally, the generated code and test cases are output, which allows for automated coding and test creation.

[0051] Example 1

[0052] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0053] In traditional meetings and hearings, minutes and requirements extraction were often done manually, which was time-consuming and labor-intensive. Manual minutes creation could lead to information leaks and misunderstandings, hindering efficient project management. Furthermore, the lack of automation in requirements extraction and minutes creation made it difficult to organize information in the early stages of a project.

[0054] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0055] In this invention, the server includes means for acquiring voice data, means for transmitting the acquired voice data to the server, means for converting the voice data into text using voice recognition technology, means for analyzing the text data and extracting requirements using natural language processing technology, means for automatically generating minutes based on the extracted requirements, and means for providing the generated minutes to a user. This makes it possible to automatically convert the contents of meetings and hearings into text, extract requirements, and generate minutes. This eliminates the need for manual minutes creation, prevents information leaks and misunderstandings, and enables efficient project management.

[0056] "Audio data" refers to data in which audio information from meetings, hearings, etc. is recorded in digital format.

[0057] A "server" is a computer system that processes and stores data over a network.

[0058] "Voice recognition technology" is a technology that analyzes voice data and converts it into text data.

[0059] "Text data" is data expressed as character information.

[0060] "Natural language processing technology" is a technology that analyzes text data, understands its meaning, and extracts information.

[0061] "Requirements" are the conditions and specifications required for a project or system.

[0062] "Minutes" are documents that record the contents of meetings or hearings.

[0063] "Automatic generation" refers to the system automatically creating data or documents without manual intervention.

[0064] "User" refers to a person or organization that uses the system.

[0065] "RFP" stands for Request for Proposal, a document that requests proposals for projects or services.

[0066] A "specification" is a document that lists the detailed specifications of a system or project.

[0067] "UI" stands for user interface, and refers to the screen and operation methods that allow users to interact with the system.

[0068] "Customer feedback" refers to opinions and requests provided by customers.

[0069] "Auto-coding" is a technique for automatically generating program code.

[0070] "Test creation" refers to creating test cases to check the operation of a system or program.

[0071] "Artificial intelligence technology" is a technology that allows machines to imitate human intelligence and learn and reason.

[0072] The present invention is a system that automatically converts the contents of a meeting or hearing into text, extracts requirements, and generates minutes. A specific embodiment of this system will be described below.

[0073] Hardware and Software Configuration

[0074] 1. Acquiring audio data

[0075] Users acquire audio data from meetings and hearings using devices such as laptops and smartphones. The devices are equipped with microphones, and audio data is recorded by launching a recording application.

[0076] 2. Sending audio data

[0077] The device sends the acquired audio data to the server. Specifically, the device application uploads the audio data to the server as an HTTP request. At this time, the audio data is sent in an appropriate format, such as WAV or MP3.

[0078] 3. Converting audio data into text

[0079] The server processes the received voice data in real time. Specifically, it converts the voice data into text data using voice recognition technology. For example, it uses the Google® Cloud Speech-to-Text API. The server sends the voice data to the API and stores the returned text data.

[0080] 4. Text Data Analysis

[0081] The server analyzes the text data using natural language processing (NLP) techniques. Specifically, it uses the Python libraries spaCy and NLTK to extract keywords and important phrases from the text. The server passes the text data to the analysis module and obtains the analysis results.

[0082] 5. Requirements Extraction

[0083] The server extracts requirements from the analysis results. Specifically, it lists important requirements from the conversation based on the extracted keywords and phrases. For example, requirements such as "project start date," "budget," and "person in charge" are extracted.

[0084] 6. Generate minutes

[0085] The server automatically generates minutes based on the extracted requirements. Specifically, it organizes the main points of the conversation and decisions made, and creates text-based minutes. The server uses templates to format the minutes.

[0086] 7. Providing Results

[0087] The server provides the generated minutes to the user. Specifically, it generates the minutes in PDF or Word format and sends a download link to the user's device. The user can click the link to download the minutes and check their contents.

[0088] Specific examples

[0089] For example, consider a scenario where a user is hearing requirements for a new project. The user uses their laptop to record the conversation, and the audio data is sent to a server. The server converts the audio into text using the Google Cloud Speech-to-Text API, and then analyzes the text using spaCy. As a result of the analysis, requirements such as "project start date," "budget," and "person in charge" are extracted. Finally, minutes containing these requirements are automatically generated and provided to the user.

[0090] Prompt Sentence Examples

[0091] Examples of prompts to be input to a generative AI model include:

[0092] "Please transcribe the following audio and extract the key requirements. The audio contains requirements for a new project."

[0093] Using this prompt, the generative AI model can properly process the voice data and extract the necessary information.

[0094] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0095] Step 1: Acquire audio data

[0096] Users use devices such as laptops and smartphones to capture audio data from meetings and hearings. Specifically, they start up the device's recording application and press the record button to begin capturing audio data. The input is the audio from the meeting or hearing, and the output is digital audio data.

[0097] Step 2: Sending audio data

[0098] The device sends the acquired audio data to the server. Specifically, the device application uploads the audio data to the server as an HTTP request. The audio data is sent in an appropriate format, such as WAV or MP3. The input is digital audio data, and the output is the audio data sent to the server.

[0099] Step 3: Converting audio data to text

[0100] The server processes the received voice data in real time. Specifically, it uses voice recognition technology to convert the voice data into text data. For example, it uses the Google Cloud Speech-to-Text API. The server sends the voice data to the API and saves the returned text data. The input is the voice data sent to the server, and the output is text data.

[0101] Step 4: Analyzing the text data

[0102] The server analyzes the text data using natural language processing (NLP) techniques. Specifically, it uses the Python libraries spaCy and NLTK to extract keywords and important phrases from the text. The server passes the text data to the analysis module and obtains the analysis results. The input is the text data, and the output is the keywords and important phrases that are the analysis results.

[0103] Step 5: Eliciting requirements

[0104] The server extracts requirements from the analysis results. Specifically, it lists important requirements from the conversation based on the extracted keywords and phrases. For example, requirements such as "project start date," "budget," and "person in charge" are extracted. The input is the keywords and important phrases from the analysis results, and the output is the extracted requirements.

[0105] Step 6: Generate the transcript

[0106] The server automatically generates minutes based on the extracted requirements. Specifically, it organizes the main points of the conversation and decisions and creates text-based minutes. The server uses a template to format the minutes. The input is the extracted requirements, and the output is the generated minutes.

[0107] Step 7: Delivering results

[0108] The server provides the generated minutes to the user. Specifically, it generates the minutes in PDF or Word format and sends a download link to the user's device. The user can click the link to download the minutes and check their contents. The input is the generated minutes, and the output is the minutes provided to the user.

[0109] (Application example 1)

[0110] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0111] In autonomous vehicles, smooth communication between passengers and vehicle systems is required, and passenger instructions and requests must be accurately reflected. However, with current systems, it is difficult to convert passenger voice instructions into text in real time, extract requirements, and reflect them in the vehicle system. In addition, since minutes are generated and requirements are extracted manually, this is inefficient and lacks accuracy. Technology is needed to solve these problems.

[0112] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0113] In this invention, the server includes means for automatically generating minutes and extracting requirements from the contents of the interview, means for automatically creating RFPs and specifications, means for automatically creating a UI and reflecting customer feedback, means for realizing automatic coding and test creation, means for converting the contents of the interview into text using speech recognition technology and extracting requirements from the text using natural language processing technology, means for converting conversations inside the autonomous vehicle into text in real time, generating minutes, and extracting requirements, and means for reflecting the extracted requirements in the vehicle system. This makes it possible to accurately reflect passenger instructions and requests in real time.

[0114] "Hearing content" refers to audio information collected from users and customers.

[0115] "Minutes generation" is the process of recording the contents of a meeting or conversation as a document based on the contents of the hearing.

[0116] "Requirements extraction" is the process of extracting important keywords and phrases from the interview content and organizing them as system or project requirements.

[0117] "RFP" stands for Request for Proposal, a document requesting proposals for the provision of projects or services.

[0118] A "specification" is a document that describes the detailed requirements and functions of a system or project.

[0119] "UI" stands for user interface and refers to the screens and methods of operation that allow users to interact with systems and applications.

[0120] "Customer feedback" refers to information such as opinions, requests, and evaluations provided by customers.

[0121] "Autocoding" is the process of automatically generating program code.

[0122] "Test creation" is the process of creating test cases to verify the operation of a system or application.

[0123] "Speech recognition technology" is a technology that converts speech into text.

[0124] "Natural language processing technology" is a technology for analyzing text data and understanding its meaning and structure.

[0125] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[0126] "Real-time" means processing or reacting immediately, without delay.

[0127] A "vehicle system" is a system for controlling the operation and functions of an autonomous vehicle.

[0128] The following system configuration is used as an embodiment of this invention. The server includes means for automatically generating minutes and extracting requirements from the contents of interviews, means for automatically creating RFPs and specifications, means for automatically creating UIs and reflecting customer feedback, means for realizing automatic coding and test creation, means for converting the contents of interviews into text using speech recognition technology and extracting requirements from the text using natural language processing technology, means for converting conversations inside an autonomous vehicle into text in real time, generating minutes, and extracting requirements, and means for reflecting the extracted requirements in the vehicle system.

[0129] Program processing explanation

[0130] The server captures voice data using the smartphone's microphone and converts it to text using the Google Speech Recognition API. The converted text is then analyzed using natural language processing models in the Transformers library to extract important keywords and phrases, ensuring passengers' instructions and requests are accurately reflected in real time.

[0131] Specifically, the system converts the interview content into text using speech recognition technology and then extracts requirements from the text using natural language processing technology. For example, if a passenger says, "Turn right at the next intersection," the system extracts the requirements "next intersection" and "turn right" and reflects them in the autonomous vehicle's system.

[0132] Hardware and software used

[0133] Hardware: Smartphone microphone

[0134] Software: Python, Google Speech Recognition API, Transformers library

[0135] Specific examples

[0136] For example, if a passenger says, "Turn right at the next intersection," the system extracts the requirements "next intersection" and "turn right" and reflects them in the autonomous vehicle's system.

[0137] Prompt Sentence Examples

[0138] Passenger: "Turn right at the next intersection."

[0139] System: Extracted the requirements "next intersection" and "turn right."

[0140] In this way, passenger instructions can be recognized in real time and reflected in the operation of the autonomous vehicle.

[0141] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0142] Step 1:

[0143] The server acquires voice data using the smartphone's microphone. The user's voice is input, and voice data is collected through the microphone. This voice data is then sent to the server in real time.

[0144] Step 2:

[0145] The server converts the acquired voice data into text using the Google Speech Recognition API. The input is voice data and the output is text data. Using voice recognition technology, the voice data is analyzed and converted into the corresponding text.

[0146] Step 3:

[0147] The server analyzes the converted text data using a natural language processing model from the Transformers library. The input is text data, and the output is extracted keywords and phrases. Natural language processing techniques are used to extract important keywords and phrases from the text data.

[0148] Step 4:

[0149] The server organizes the extracted keywords and phrases as requirements and reflects them in the autonomous vehicle system. The input is the extracted keywords and phrases, and the output is the requirements reflected in the vehicle system. The requirements are organized and data processing is performed to apply them to the vehicle system.

[0150] Step 5:

[0151] The server confirms that the requirements have been accurately reflected and provides feedback to the user. The input is the requirements reflected in the vehicle system, and the output is a feedback message to the user. A process is performed to generate the feedback message and notify the user.

[0152] Example 2

[0153] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0154] In the conventional system, tasks such as generating minutes from interviews, extracting requirements, and creating RFPs and specifications based on the requirements were all done manually, which was time-consuming and labor-intensive. There was also a high possibility of human error occurring in the extraction of requirements and document creation. Furthermore, there was a lack of efficient means for creating UIs, incorporating customer feedback, and automated coding and test creation. To solve these issues, a more efficient and accurate system is needed.

[0155] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0156] In this invention, the server includes a means for automatically generating minutes and extracting requirements from the interview content, a means for receiving requirements documents and extracting requirements using a generative AI model, and a means for automatically creating RFPs and specifications using templates based on the extracted requirements. This automates the process from requirements extraction to creating RFPs and specifications, significantly reducing time and labor. It also reduces human error and enables more accurate document creation.

[0157] "Hearing contents" refers to information including the requirements and wishes of the user regarding the project or system provided.

[0158] "Generating minutes" is the process of recording and documenting the contents of a meeting based on the hearing content.

[0159] "Requirements extraction" is the process of extracting necessary information from interview content and requirements documents and organizing it as structured data.

[0160] A "generative AI model" is a model that uses artificial intelligence technology to generate text and extract requirements.

[0161] A "prompt sentence" is an input sentence that gives specific instructions to a generative AI model.

[0162] A "template" is a predefined document template used when creating RFPs and specifications.

[0163] "RFP" stands for Request for Proposal, a document used to request proposals for projects or services.

[0164] A "specification" is a document that lists the detailed requirements and specifications for a system or project.

[0165] "UI" stands for user interface and refers to the screen and operation methods that users use to interact with a system.

[0166] "Customer feedback" refers to opinions and requests provided by customers, and is information that can be used to improve and adjust the system.

[0167] "Autocoding" is the process of automatically generating program code.

[0168] "Test creation" is the process of creating test cases to verify the operation of a system or program.

[0169] This invention relates to a system that automatically generates minutes and extracts requirements from interview content, receives requirements documents, extracts requirements using a generative AI model, and automatically creates RFPs and specifications using templates based on the extracted requirements.

[0170] The server receives the requirements document provided by the user and extracts requirements using a generative AI model. Specifically, the server performs the following steps:

[0171] First, the user uploads a requirements document through the system interface. The requirements document can be provided in the form of a text file, PDF, Word, etc. The server receives the uploaded requirements document and analyzes its contents.

[0172] Next, the server generates a prompt based on the requirements document. A prompt is an input sentence that provides specific instructions to the generative AI model. For example, the following prompt might be generated:

[0173] Generate an RFP based on the requirements below.

[0174] Project Objective: Development of a new customer management system

[0175] Goal: Efficiently manage and analyze customer data

[0176] Functional requirements: Ability to register, search, update, and delete customer information

[0177] Non-functional requirements: high security, scalability

[0178] The server inputs this prompt into a generative AI model to extract requirements. For example, OpenAI's GPT-4 (registered trademark) is used as the generative AI model. The generative AI model extracts necessary information from the requirements document based on the prompt and saves it as structured data.

[0179] The server then selects a pre-prepared RFP or specification template and automatically fills in each item in the template with the extracted requirements. For example, the extracted "project objectives" and "goals" are filled in the "Project Overview" section of the template.

[0180] Finally, the server generates the final RFP and specifications based on the requirements embedded in the template. The generated documents are output in PDF or Word format and made available for download by users. Users can download and review the generated RFP and specifications from the system interface.

[0181] This system automates the process from requirements extraction to the creation of RFPs and specifications, significantly reducing time and effort, and also reducing human error, resulting in more accurate document creation.

[0182] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0183] Step 1:

[0184] A user uploads a requirements document.

[0185] The user logs in to the system's web interface and clicks the "Upload requirements document" button. A file selection dialog appears, and the user selects and uploads the requirements document. The input is the requirements document, and the output is the requirements document saved on the server.

[0186] Step 2:

[0187] The server receives the requirements document and parses the contents.

[0188] The server receives the uploaded requirements document and analyzes its contents. Specifically, it reads the text of the requirements document and identifies each section. The input is the uploaded requirements document, and the output is the parsed text data of the requirements document.

[0189] Step 3:

[0190] The server generates a prompt.

[0191] The server generates a prompt sentence based on the content of the parsed requirements document. For example, if the requirements document states "Project Objective: Develop a new customer management system," the prompt sentence will contain "Project Objective: Develop a new customer management system." The input is the text data of the parsed requirements document, and the output is the generated prompt sentence.

[0192] Step 4:

[0193] The server uses a generative AI model to extract requirements.

[0194] The server inputs the generated prompt sentence into a generative AI model to extract requirements. Examples of generative AI models that can be used include OpenAI's GPT-4. The generative AI model extracts necessary information from the requirements document based on the prompt sentence and saves it as structured data. The input is the prompt sentence, and the output is structured data of the extracted requirements.

[0195] Step 5:

[0196] The server selects a template and fills it with the extracted requirements.

[0197] The server selects a pre-prepared RFP or specification template. Then, it automatically embeds the extracted requirements into each section of the template. For example, the extracted "Project Objectives" and "Goals" are embedded in the "Project Overview" section. The input is structured data of the extracted requirements, and the output is a template with the embedded requirements.

[0198] Step 6:

[0199] The server generates the final RFP and specifications and provides them to the user.

[0200] The server generates the final RFP or specification based on the requirements embedded in the template. The generated document is output in PDF or Word format and made available for download by the user. The user can download and review the generated RFP or specification from the system interface. The input is the template with the embedded requirements, and the output is the generated RFP or specification.

[0201] (Application example 2)

[0202] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0203] In the conventional system, processes such as generating minutes from interviews, extracting requirements, creating RFPs and specifications, automatically creating UI, incorporating customer feedback, and automatically coding and creating tests were all done manually, which was time-consuming and labor-intensive. Additionally, creating specifications for machinery used in factories was also done manually, which was inefficient and prone to errors. To solve these issues, a system was needed that could automate these processes and perform them efficiently and accurately.

[0204] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0205] In this invention, the server includes a means for automatically generating minutes and extracting requirements from interview content, a means for automatically creating RFPs and specifications, a means for automatically creating UIs and incorporating customer feedback, a means for realizing automatic coding and test creation, and a means for automatically generating specifications for machines used in factories. This automates these processes, making them efficient and accurate.

[0206] "Hearing content" refers to oral information collected through meetings, interviews, etc.

[0207] "Generating minutes" means creating a document that records the contents of a meeting based on the contents of the hearing.

[0208] "Requirements elicitation" is the process of identifying and extracting necessary requirements from interviews and other sources of information.

[0209] "RFP" stands for Request for Proposal, a document that solicits proposals for a specific project or service.

[0210] A "specification" is a document that details the functions, performance, design, etc. of a product or system.

[0211] "UI" stands for user interface, the means by which a user interacts with a system or application.

[0212] "Customer feedback" refers to information such as opinions, requests, and evaluations provided by customers.

[0213] "Auto-coding" means automatically generating program code.

[0214] "Test creation" means creating test cases to verify the operation of software or systems.

[0215] "Machinery used in a factory" refers to equipment and facilities used for production and processing in the manufacturing industry.

[0216] "Automatic specification generation" means automatically creating specifications based on specific requirements and conditions.

[0217] A system for carrying out this invention is configured as follows: The server includes means for automatically generating minutes and extracting requirements from interview content, means for automatically creating RFPs and specifications, means for automatically creating UIs and incorporating customer feedback, means for realizing automatic coding and test creation, and means for automatically generating specifications for machines used in factories.

[0218] The server converts the interview content into text using speech recognition technology and extracts requirements from the text using natural language processing technology. This makes it possible to automatically generate meeting minutes and extract requirements from the interview content. Furthermore, it uses a generative AI model to generate prompt sentences based on the requirements and automatically create RFPs and specifications.

[0219] Specifically, the server performs the following process. First, the user inputs the interview content by voice, and the server converts it into text using speech recognition technology. Next, it uses natural language processing technology to extract requirements from the text. The extracted requirements are input into a generative AI model, which generates prompt text. Based on this prompt text, an RFP and specifications are automatically created.

[0220] The server also includes a means for automatically generating specifications for machines used in the factory. When a user inputs requirements for machines in the factory, the server analyzes them and automatically generates specifications using an appropriate template. For example, when requirements for the installation of a new manufacturing line are input, the server generates a prompt sentence such as the following:

[0221] Example prompt sentence:

[0222] Generate a specification based on the following requirements:

[0223] 1. Installation of a new production line

[0224] 2. Machine layout and connection

[0225] 3. Implementation of safety measures

[0226] 4. Regular maintenance schedule

[0227] Template:

[0228] 1. Overview

[0229] 2. Detailed requirements

[0230] 3. Implementation Plan

[0231] 4. Maintenance Plan

[0232] Generated specification:

[0233] Based on this prompt, the server uses a generative AI model to automatically generate specifications, enabling efficient and accurate creation of specifications for machines used in factories.

[0234] The hardware used includes factory robots (e.g., ABB's IRB series) and servers. The software used includes the OpenAI GPT-3 (registered trademark) model and Python programs. The combination of these technologies enables the achievement of the objectives of the invention.

[0235] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0236] Step 1:

[0237] The user inputs what he or she has heard by voice.

[0238] Input: The user inputs what they heard by voice.

[0239] How it works: A user uses a microphone to dictate a meeting or interview.

[0240] Output: The audio data is sent to the server.

[0241] Step 2:

[0242] The server uses voice recognition technology to convert the voice data into text.

[0243] Input: Audio data

[0244] How it works: The server uses speech recognition software (e.g., Google Speech-to-Text API) to convert the audio data into text.

[0245] Output: Text data

[0246] Step 3:

[0247] The server uses natural language processing technology to extract requirements from the text data.

[0248] Input: Text data

[0249] How it works: The server uses a natural language processing library (e.g., spaCy) to extract requirements from text data, specifically by extracting keywords and analyzing context.

[0250] Output: Requirement list

[0251] Step 4:

[0252] The server generates a prompt using a generative AI model.

[0253] Input: Requirement List

[0254] How it works: The server feeds a generative AI model (e.g., OpenAI GPT-3) with a list of requirements and generates a prompt.

[0255] Output: prompt statement

[0256] Step 5:

[0257] The server automatically generates RFPs and specifications based on the prompt text.

[0258] Input: prompt statement

[0259] How it works: The server uses a generative AI model to generate RFPs and specifications based on the prompt text, specifically by embedding requirements in templates to create documents.

[0260] Output: RFP and specifications

[0261] Step 6:

[0262] The server automatically generates specifications for the machines used in the factory.

[0263] Input: Factory machinery requirements

[0264] How it works: The server analyzes the requirements for machines in a factory and automatically generates specifications using appropriate templates.

[0265] Output: Machine specifications

[0266] Step 7:

[0267] The server provides the generated RFP and specifications to the user.

[0268] Input: Generated RFP or specification

[0269] What happens: The server sends the generated document to the user, either via email or by making it available for download within the system.

[0270] Output: User-accessible RFP and specifications

[0271] Example 3

[0272] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0273] In the traditional system development process, each phase, from requirements definition to design, coding, and testing, requires a lot of manual work, which is time-consuming and labor-intensive. Also, incorporating customer feedback requires manual corrections, which is inefficient. Furthermore, there is little automation in code generation and test case creation, placing a heavy burden on developers. To solve these problems, automation and efficiency in each phase are required.

[0274] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0275] In this invention, the server includes a means for automatically generating minutes and extracting requirements from interview content, a means for automatically creating requirements specifications and design documents, a means for automatically generating a user interface based on requirements and incorporating customer feedback, and a means for generating code and creating test cases using a generative AI model based on requirements, thereby automating and streamlining each phase of the system development process.

[0276] "Hearing content" refers to information such as requirements, wishes, and opinions collected from customers and stakeholders during system development.

[0277] "Generating minutes" is the process of recording and documenting the contents of a meeting based on the hearing content.

[0278] "Requirements extraction" is the process of identifying and extracting the requirements necessary for system development from interview content and meeting minutes.

[0279] A "requirements specification" is a document that describes in detail the functions, performance, and constraints of a system or software.

[0280] A "design document" is a document that describes in detail the structure and operation of a system or software.

[0281] "User interface" refers to the screens and methods of operation that allow a user to interact with a system or software.

[0282] "Customer feedback" refers to information such as opinions, evaluations, and areas for improvement provided by customers.

[0283] A "generative AI model" is a model that uses artificial intelligence technology to automatically perform specific tasks.

[0284] "Code generation" is the process of automatically creating program code based on requirements and designs.

[0285] "Test case generation" is the process of automatically creating test scenarios to verify whether the generated code works correctly.

[0286] The present invention is a system for automating and streamlining each phase of a system development process. Specific embodiments of this system will be described below.

[0287] Generate meeting minutes and extract requirements from interview content

[0288] The server uses speech recognition technology to convert the contents of the hearing into text. For example, speech recognition software is used to convert what is said during the meeting into text in real time. It then uses natural language processing technology to extract requirements from the text. Specifically, it uses a text analysis algorithm to identify important keywords and phrases and extract them as requirements.

[0289] Automatic creation of requirements specifications and design documents

[0290] The server automatically creates requirements specifications and design documents based on the extracted requirements, for example, by using a template-based document generation tool to document the requirements in an appropriate format, thereby eliminating the need to manually create the documents.

[0291] Automatically generate user interfaces and incorporate customer feedback

[0292] The server automatically generates a user interface (UI) based on the requirements. Specifically, an initial UI design is created using a UI design tool (e.g., Figma). The user then reviews the generated UI design and provides feedback. The server analyzes the user's feedback and automatically applies the necessary changes to update the UI design.

[0293] Examples:

[0294] Prompt: "If the user gives feedback that they would like to change the position of the buttons on the login screen, move the buttons to the center."

[0295] Code generation and test case creation

[0296] The server generates code using a generative AI model (e.g., OpenAI Codex) based on the requirements. The generated code satisfies the system's functional requirements. At the same time, the server generates test cases using a test automation tool (e.g., Selenium). This automatically creates test scenarios to verify that the generated code works correctly.

[0297] Examples:

[0298] Prompt: "Generate the code for the user authentication function and create a test case for user authentication at the same time."

[0299] Reviewing the code and test cases

[0300] The user checks the generated code and test cases and makes any necessary modifications. The terminal then sends the modified code and test cases to the server, completing the final system.

[0301] In this way, this invention can automate and streamline each phase of the system development process. Specific software used includes voice recognition software, natural language processing algorithms, UI design tools (e.g., Figma), generative AI models (e.g., OpenAI Codex), and test automation tools (e.g., Selenium). The flow of specific processing in Example 3 will be described with reference to FIG. 15.

[0302] Step 1:

[0303] The user inputs UI and function requirements into the system. Specifically, the user describes the requirements in detail using forms and text boxes. The terminal then sends the input requirements to the server.

[0304] Input: User-entered requirements (e.g., "Please add username and password input fields to the login screen")

[0305] Output: Requirement data sent to the server

[0306] Step 2:

[0307] The server automatically generates an initial UI design using a UI design tool based on the received requirements. Specifically, it uses a UI design tool (e.g., Figma) to create a UI design based on the requirements. The server then sends the generated UI design to the device.

[0308] Input: Requirement data

[0309] Output: Initial UI design data

[0310] Step 3:

[0311] The user checks the generated UI design and provides feedback, specifically indicating which parts of the UI they would like to change using comments or markup. The device then sends the user's feedback to the server.

[0312] Input: Initial UI design data

[0313] Output: User feedback data (e.g. "Please move the button to the center")

[0314] Step 4:

[0315] The server analyzes the user feedback and automatically applies the necessary changes. Specifically, it analyzes the feedback and updates the design using a UI design tool. The server then sends the updated UI design back to the device.

[0316] Input: User feedback data

[0317] Output: Updated UI design data

[0318] Step 5:

[0319] The server generates code using a generative AI model based on the requirements. Specifically, it uses a generative AI model (e.g., OpenAI Codex) to create program code that meets the requirements. The server then sends the generated code to the device.

[0320] Input: Requirement data

[0321] Output: Generated code data

[0322] Step 6:

[0323] The server generates test cases based on the generated code using a test automation tool. Specifically, it uses a test automation tool (e.g., Selenium) to create test scenarios to verify the code's behavior. The server then sends the generated test cases to the terminal.

[0324] Input: Generated code data

[0325] Output: Generated test case data

[0326] Step 7:

[0327] The user checks the generated code and test cases and makes any necessary modifications. The terminal then sends the modified code and test cases to the server, completing the final system.

[0328] Input: Generated code data, generated test case data

[0329] Output: Modified code data, modified test case data

[0330] (Application example 3)

[0331] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0332] With the previous system, designing and updating the user interface (UI) required a great deal of time and effort, making it difficult to quickly reflect user feedback. Furthermore, adding new features and creating test cases for them had to be done manually, reducing development efficiency. This made it difficult to improve the user experience and quickly add new features, potentially resulting in a loss of competitiveness.

[0333] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0334] In this invention, the server includes means for automatically generating minutes and extracting requirements from interview content, means for automatically creating requirements specifications and design documents, means for automatically creating a UI and incorporating customer feedback, means for automatically coding and creating tests, means for optimizing the user interface in real time, and means for automatically generating code and test cases for new functions. This makes it possible to quickly incorporate user feedback and optimize the UI in real time. Furthermore, the addition of new functions and the creation of their test cases are also automated, significantly improving development efficiency.

[0335] "Hearing content" refers to the content of conversations and interviews conducted to gather requirements and opinions from users and customers.

[0336] "Minutes generation" is the process of automatically creating a record of a meeting based on the contents of the hearing.

[0337] "Requirements extraction" is the process of identifying the functions and specifications required for a system or application from meeting minutes and interview content.

[0338] A "requirements specification" is a document that details the functions and performance that a system or application must meet.

[0339] A "design document" is a technical document that describes in detail the structure and operation of a system or application.

[0340] "UI" stands for user interface and refers to the screens and methods of operation that allow users to interact with systems and applications.

[0341] "Customer feedback" refers to information such as opinions, evaluations, and areas for improvement provided by users and customers.

[0342] "Auto-coding" is the process of using AI technology to automatically generate program code instead of writing it manually.

[0343] "Test creation" is the process of designing and creating test cases to verify the operation of a system or application.

[0344] "User interface optimization" is the process of improving and adjusting the UI to improve user operability and convenience.

[0345] "New function code" is program code for implementing new functions that are added to a system or application.

[0346] A "test case" is a description of specific test scenarios and conditions for verifying the operation of a system or application.

[0347] A system for carrying out this invention includes the following main components: The server includes means for automatically generating minutes and extracting requirements from interview content, means for automatically creating requirement specifications and design documents, means for automatically creating a UI and incorporating customer feedback, means for realizing automatic coding and test creation, means for optimizing the user interface in real time, and means for automatically generating code and test cases for new functions.

[0348] Hardware and software used

[0349] Hardware: Cloud-based servers, user devices (smartphones, tablets, PCs, etc.)

[0350] Software: Python, AI model (e.g., GPT-3, BERT, etc.), database (MySQL (registered trademark), MongoDB, etc.)

[0351] Data processing and calculation

[0352] Processing of hearing contents

[0353] The server uses speech recognition technology to convert the interview content into text and natural language processing technology to extract requirements from the text, automatically generating minutes and clarifying the requirements.

[0354] Automatic creation of requirements specifications and design documents

[0355] The server automatically generates requirements specifications and design documents based on the extracted requirements, thereby streamlining the development process.

[0356] Automatic UI generation and feedback reflection

[0357] The server automatically generates an initial UI design based on requirements and collects real-time user feedback to update the UI, resulting in a better user experience.

[0358] Automated coding and test creation

[0359] The server uses AI technology to automatically generate code and test cases for new features, significantly reducing the amount of work required by developers.

[0360] User interface optimization

[0361] The server optimizes the user interface in real time based on user operation data and feedback, thereby improving user convenience.

[0362] Specific examples

[0363] For example, if a user provides feedback saying "I would like a wishlist feature," the server would do the following:

[0364] 1. Collect feedback and feed it into the AI ​​model.

[0365] 2. AI model generates new UI design and adds wishlist feature.

[0366] 3. Automatically generate code and test cases for new features.

[0367] Prompt Sentence Examples

[0368] User feedback: "I want a wishlist feature"

[0369] Requirement: {"feature": "wishlist", "description": "A feature that allows users to add products to a wishlist"}

[0370] By inputting this prompt into the AI ​​model, new UI designs, code, and test cases are automatically generated.

[0371] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0372] Step 1:

[0373] The user provides feedback.

[0374] The user enters feedback, such as "I want a wishlist function," into the feedback form of the application. The input feedback is sent to the server.

[0375] Step 2:

[0376] The server collects the feedback and inputs it into the AI ​​model.

[0377] The server receives the feedback sent by the user and inputs it into the AI ​​model. Specifically, the feedback content is passed to the AI ​​model in text format. The input data is in the form of "User feedback: 'I want a wishlist function'".

[0378] Step 3:

[0379] The server uses the AI ​​model to generate a new UI design.

[0380] The server inputs the feedback into the AI ​​model and generates a new UI design. The AI ​​model analyzes the feedback and adds or modifies appropriate UI elements. The output data is a specification for the new UI design.

[0381] Step 4:

[0382] The server automatically generates code for the new feature.

[0383] The server uses the AI ​​model to generate code for new functions. Specifically, it identifies the required functions based on the feedback and generates program code to realize those functions. The output data is the program code for the new functions.

[0384] Step 5:

[0385] The server automatically generates test cases for new features.

[0386] The server uses the AI ​​model to generate test cases for new functions. Specifically, it creates test scenarios based on the generated program code to verify whether the code works correctly. The output data is a test case specification.

[0387] Step 6:

[0388] The server deploys the generated UI design and code.

[0389] The server deploys the generated UI design and program code into the application, integrating new UI elements and functionality into the existing application and making them available to users. The output data is the updated application.

[0390] Step 7:

[0391] Users will experience the new UI and features.

[0392] Users use the updated application to experience the new UI and added features, such as adding products to a wishlist. The output data is the user's operation log and new feedback.

[0393] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0394] "Example 1"

[0395] One embodiment of the present invention is a system that incorporates an emotion engine. This system recognizes emotions from the user's voice and facial expressions, and automatically creates a UI based on those emotions and incorporates customer feedback. Specifically, if the user makes an expression that shows joy or surprise, the system captures that information and automatically changes the UI color and layout to be brighter and more fun. Also, if the user makes an expression that shows anger or dissatisfaction, the system captures that information, identifies the problem area, and automatically generates feedback to encourage improvement in that area.

[0396] "Example 2"

[0397] Another embodiment of the present invention is automatic coding and test creation using an emotion engine. Specifically, if a user makes an expression that indicates confusion or hesitation, the system captures that information and determines that there is a problem with the code generation or test case creation. The system then identifies the problem area and redoes the code generation or test case creation for that area. This enables more appropriate code generation and test case creation that reflects the user's emotions.

[0398] "Example 3"

[0399] One embodiment of the present invention is a system that incorporates an emotion engine. This system recognizes emotions from the user's voice and facial expressions, and automatically creates a UI based on those emotions and incorporates customer feedback. Specifically, if the user makes an expression that shows joy or surprise, the system captures that information and automatically changes the UI color and layout to be brighter and more fun. Also, if the user makes an expression that shows anger or dissatisfaction, the system captures that information, identifies the problem area, and automatically generates feedback to encourage improvement in that area.

[0400] The processing flow of each embodiment will be described below.

[0401] "Example 1"

[0402] Step 1: Capture the user's voice and facial expressions through the camera and microphone.

[0403] Step 2: The acquired voice and facial expression data is input into the emotion engine to recognize the user's emotions.

[0404] Step 3: Automatically change UI color and layout based on the recognized emotion.

[0405] Step 4: If the user makes a facial expression that indicates anger or dissatisfaction, the system will identify the problem area and automatically generate feedback to encourage improvement in that area.

[0406] "Example 2"

[0407] Step 1: Capture the user's voice and facial expressions through the camera and microphone.

[0408] Step 2: The acquired voice and facial expression data is input into the emotion engine to recognize the user's emotions.

[0409] Step 3: If the user looks confused or unsure, it is determined that there is a problem with the code generation or test case creation.

[0410] Step 4: Identify the problem areas and regenerate code and create test cases for those areas.

[0411] Example 1

[0412] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0413] In traditional meetings and interviews, creating minutes and extracting requirements is often done manually, which takes time and effort. Furthermore, it is difficult to automatically create a user interface that reflects user emotions or automatically generate customer feedback, making improving the user experience a challenge. Furthermore, there is a demand for more efficient automated coding and test creation.

[0414] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0415] In this invention, the server includes means for collecting voice data and converting it into text in real time, means for analyzing the text data and extracting requirements, means for recognizing user emotions and automatically changing the user interface, means for automatically generating customer feedback, and means for realizing automatic coding and test creation. This enables the automatic generation of minutes and efficient extraction of requirements, enabling the automatic creation of a user interface that reflects user emotions and the rapid provision of customer feedback. Furthermore, automatic coding and test creation using artificial intelligence technology can improve the efficiency of the development process.

[0416] "Voice data" refers to data in which a user's speech or voice is recorded in digital format.

[0417] "Text data" refers to data obtained by converting voice data into character information.

[0418] "Speech recognition technology" is a technology that analyzes voice data and converts it into corresponding text data.

[0419] "Natural language processing technology" is a technology that analyzes text data and understands its meaning and structure.

[0420] "Emotion recognition technology" is a technology that analyzes and recognizes emotions from a user's facial expressions and voice.

[0421] "User interface" refers to the screen and operation method that allows a user to interact with a system.

[0422] "Customer feedback" refers to improvements and evaluations based on users' usage experiences and opinions.

[0423] "Auto-coding" is a technique for automatically generating program code.

[0424] "Test creation" is the technique of generating test cases to verify the behavior of a program.

[0425] "Artificial intelligence technology" is a technology that allows machines to imitate human intelligence and perform learning and reasoning.

[0426] This invention is a system that includes means for collecting voice data and converting it into text in real time, means for analyzing the text data and extracting requirements, means for recognizing user emotions and automatically changing the user interface, means for automatically generating customer feedback, and means for realizing automatic coding and test creation.

[0427] Collecting and converting audio data into text

[0428] The device captures the user's voice data in real time through a microphone, which is then sent over the network to a server, which then converts the received voice data into text using voice recognition technology such as the Google Cloud Speech-to-Text API.

[0429] Example: When a user says "Let's schedule our next meeting for next Tuesday" during a meeting, the device captures the audio and sends it to the server, which uses a speech recognition API to convert the audio into text: "Let's schedule our next meeting for next Tuesday."

[0430] Text data analysis and requirement extraction

[0431] The server analyzes the text data using natural language processing technologies such as Python's NLTK library and SpaCy, extracts important keywords and phrases from the analysis results, and records them as requirements.

[0432] Example: The server analyzes the text data "Please set the next meeting for next Tuesday" and extracts the keywords "next meeting," "next Tuesday," and "set."

[0433] Emotion Recognition and Automatic User Interface Creation

[0434] The device captures the user's facial expressions and voice in real time through a camera and microphone. This data is then sent to a server via a network. The server then analyzes the received emotional data using emotion recognition technologies such as Microsoft® Azure® Emotion API and OpenCV. Based on the analysis results, the color and layout of the user interface are automatically changed.

[0435] Example: If a user makes a surprised expression, the device captures that expression and sends it to the server, which uses the Emotion API to recognize that the user is surprised and changes the UI background color to bright yellow.

[0436] Automated customer feedback generation

[0437] The server analyzes the user's facial expressions, which indicate anger or dissatisfaction, to identify the problem area, and automatically generates and provides feedback to the user to encourage improvement of the identified problem.

[0438] Example: If a user makes a facial expression that indicates dissatisfaction, the server analyzes that information and generates feedback such as, "The user has expressed dissatisfaction. Please improve the following areas."

[0439] Automated coding and test creation

[0440] The server uses artificial intelligence technology to generate code and create test cases, streamlining the development process.

[0441] Example prompt: "Transcribe audio data from a meeting in real time and extract key requirements. Also, describe a system that recognizes user emotions and automatically changes the UI color and layout."

[0442] This system enables the automatic generation of meeting minutes and efficient extraction of requirements, automatically creating user interfaces that reflect user emotions, and providing rapid customer feedback. Furthermore, the system also streamlines the development process by using artificial intelligence technology for automatic coding and test creation.

[0443] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0444] Step 1:

[0445] A user speaks during a meeting or hearing. The device captures the user's voice data in real time through a microphone. The input is the user's voice, and the output is digital voice data. A specific example of this is when the user says, "Please schedule the next meeting for next Tuesday."

[0446] Step 2:

[0447] The terminal transmits the captured audio data to the server via the network. The input is digital audio data, and the output is the audio data transmitted to the server. Specifically, the terminal divides the audio data into packets and transmits them to the server.

[0448] Step 3:

[0449] The server uses the Google Cloud Speech-to-Text API to convert the received voice data into text data. The input is the voice data sent to the server, and the output is text data. Specifically, the server calls the speech recognition API and generates the text "Please schedule the next meeting for next Tuesday."

[0450] Step 4:

[0451] The server uses Python's NLTK library and SpaCy to analyze text data. The input is text data, and the output is the analysis results. Specifically, the server tokenizes the text data and extracts important keywords and phrases.

[0452] Step 5:

[0453] The server extracts important keywords and phrases from the analysis results and records them as requirements. The input is the analysis results, and the output is the keywords and phrases recorded as requirements. Specifically, the server extracts the keywords "next meeting," "next Tuesday," and "settings" and records them in the database.

[0454] Step 6:

[0455] The device captures the user's facial expressions and voice in real time through a camera and microphone. The input is the user's facial expressions and voice, and the output is digital emotional data. Specifically, if the user makes a surprised expression, the device will capture that expression.

[0456] Step 7:

[0457] The device transmits the captured emotion data to the server via the network. The input is the emotion data in digital format, and the output is the emotion data transmitted to the server. Specifically, the device divides the emotion data into packets and transmits them to the server.

[0458] Step 8:

[0459] The server analyzes the received emotion data using Microsoft Azure's Emotion API and OpenCV. The input is the emotion data sent to the server, and the output is the analysis result. Specifically, the server calls the Emotion API to recognize that the user is surprised.

[0460] Step 9:

[0461] The server automatically changes the color and layout of the user interface based on the analysis results. The input is the analysis results, and the output is the modified user interface. Specifically, the server changes the background color of the UI to bright yellow.

[0462] Step 10:

[0463] The server analyzes the user's facial expressions, which indicate anger or dissatisfaction, and identifies the problem area. The input is the analysis result, and the output is the identified problem. Specifically, the server generates feedback such as, "The user has expressed dissatisfaction. Please improve the following areas."

[0464] Step 11:

[0465] The server automatically generates feedback to encourage improvements to the identified problems and provides it to the user. The input is the identified problems, and the output is the generated feedback. As a specific operation, the server generates a feedback message and notifies the user.

[0466] Step 12:

[0467] The server uses artificial intelligence technology to generate code and create test cases. The input is requirements and specifications, and the output is the generated code and test cases. Specifically, the server uses a generative AI model to automatically generate program code and test cases.

[0468] (Application example 1)

[0469] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0470] In traditional customer service in brick-and-mortar stores, it can be difficult for store staff to accurately understand customer requests and questions and respond appropriately. It is also difficult to properly recognize customer emotions and respond accordingly. This can lead to lower customer satisfaction and have a negative impact on sales. Furthermore, recording customer conversations and extracting customer requirements is time-consuming and inefficient.

[0471] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0472] In this invention, the server includes means for automatically generating minutes and extracting requirements from the contents of the interview, means for automatically creating specifications, means for automatically creating a user interface and reflecting customer feedback, means for realizing automatic coding and test creation, means for converting the contents of the conversation into text in real time using speech recognition technology and extracting requirements using natural language processing technology, and means for recognizing user emotions using emotion recognition technology and automatically adjusting the user interface. This makes it possible to efficiently record the contents of conversations with customers, extract requirements, and furthermore, to respond appropriately according to the customer's emotions, which is expected to improve customer satisfaction and increase sales.

[0473] "Hearing content" refers to information and opinions obtained through dialogue and interviews with customers and users.

[0474] "Minutes generation" refers to the process of recording the contents of meetings and conversations and compiling them into a document.

[0475] "Requirements extraction" is the process of extracting necessary information and conditions from dialogue or documents.

[0476] A "specification" is a document that details the functions, performance, design, etc. of a system or product.

[0477] A "user interface" refers to the screens and methods of operation that allow a user to interact with a system or application.

[0478] "Customer feedback" refers to opinions and ratings provided by customers.

[0479] "Auto-coding" refers to the process of automatically generating program code.

[0480] "Test creation" refers to the task of creating test cases to verify the operation of a system or application.

[0481] "Speech recognition technology" refers to technology that converts speech into text.

[0482] "Natural language processing technology" refers to the technology of processing human language using a computer.

[0483] "Emotion recognition technology" is a technology that determines a user's emotions from their facial expressions and tone of voice.

[0484] "Real-time" means processing or reacting immediately, without delay.

[0485] "Text conversion" is the process of converting information such as audio and images into text data.

[0486] "Keywords" are important words or phrases in a document or conversation.

[0487] A "phrase" is a sequence of words that has meaning in a sentence or dialogue.

[0488] "Color" refers to the color of the user interface.

[0489] "Layout" refers to the arrangement and structure of a user interface.

[0490] A system for carrying out this invention has the following configuration: The server includes means for automatically generating minutes and extracting requirements from the contents of interviews, means for automatically creating specifications, means for automatically creating a user interface and reflecting customer feedback, means for realizing automatic coding and test creation, means for converting the contents of conversations into text in real time using speech recognition technology and extracting requirements using natural language processing technology, and means for recognizing user emotions using emotion recognition technology and automatically adjusting the user interface.

[0491] Hardware and software used

[0492] Hardware:

[0493] Smart glasses (with built-in camera and microphone)

[0494] Computer (processing speech recognition and emotion recognition)

[0495] software:

[0496] speech_recognition library: Performs speech recognition

[0497] transformers library: for natural language processing

[0498] emotion_recognition library: Performs emotion recognition

[0499] cv2(OpenCV): Acquiring and processing camera images

[0500] System Operation

[0501] The server first captures the conversation between the customer and the store clerk through the smart glasses' microphone. It then uses voice recognition technology to convert this voice data into text in real time. It then uses natural language processing technology to extract requirements from the text. This allows the server to clearly understand the customer's requests and questions.

[0502] Furthermore, the smart glasses' camera captures the customer's facial expressions and uses emotion recognition technology to determine their emotions. For example, if the customer is happy, the colors and layout of the user interface will automatically change to a bright and cheerful one. On the other hand, if the customer is angry, information to help resolve the problem will be highlighted.

[0503] Specific examples

[0504] For example, consider a scenario in which a customer asks a question about a new product and a sales associate responds. The speech recognition result is "Please tell me the features of this new smartphone." The requirement extraction result is "Features of the new smartphone." If the customer shows interest and has a happy expression, the system changes the user interface to a bright and fun theme.

[0505] Prompt Sentence Examples

[0506] A customer is asking about a new smartphone. The speech recognition result is "Please tell me the features of this new smartphone." The requirement extraction result is "Features of the new smartphone." The customer is interested and has a happy expression. Change the UI to a bright and fun theme.

[0507] In this way, the server can efficiently record the content of conversations with customers, extract requirements, and respond appropriately according to the customer's emotions, which is expected to improve customer satisfaction and increase sales.

[0508] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0509] Step 1:

[0510] The server captures the conversation between the customer and the store clerk through the microphone in the smart glasses. The input is voice data, and the output is text data converted using speech recognition technology. Specifically, the server uses the speech_recognition library to convert voice into text in real time.

[0511] Step 2:

[0512] The server analyzes the textual content of the conversation using natural language processing technology and extracts requirements. The input is text data, and the output is the extracted requirements. Specifically, it uses the transformers library to extract keywords and phrases from the text and list them as requirements.

[0513] Step 3:

[0514] The server acquires the customer's facial expression through the smart glasses' camera and uses emotion recognition technology to determine the customer's emotion. The input is image data, and the output is the recognized emotion. Specifically, the emotion_recognition library and cv2 (OpenCV) are used to analyze facial expressions from images and determine the emotion.

[0515] Step 4:

[0516] The server automatically adjusts the user interface based on the recognized emotion. The input is the recognized emotion, and the output is the adjusted user interface. Specifically, if the customer is happy, the color and layout of the user interface are changed to a bright and cheerful one, and if the customer is angry, information for resolving the problem is highlighted.

[0517] Step 5:

[0518] The server records the content of conversations with customers and the extracted requirements, and automatically creates specifications as needed. The input is text data and extracted requirements, and the output is specifications. Specifically, the information is automatically filled into a specification template based on the text data and requirements.

[0519] Step 6:

[0520] The server reflects customer feedback and continuously improves the user interface and service content. The input is customer feedback, and the output is an improved user interface and service content. Specifically, it analyzes the feedback data, identifies areas for improvement, and reflects them in the system.

[0521] Example 2

[0522] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0523] In conventional systems, requirements extraction from interviews and the creation of RFPs and specifications were often done manually, which was time-consuming and labor-intensive. Furthermore, automated coding and test creation did not take user emotions into consideration, resulting in results that did not meet user intent. Furthermore, automated UI creation and insufficient reflection of customer feedback made it difficult to improve the user experience.

[0524] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0525] In this invention, the server includes a means for automatically generating minutes and extracting requirements from the interview content, a means for automatically creating RFPs and specifications using templates based on the requirements, and a means for analyzing the user's facial expressions and automatically coding and creating tests based on their emotions. This automates the entire process from requirements extraction to document creation, code generation, and test creation, making it possible to provide more appropriate results that take the user's emotions into consideration.

[0526] "Hearing contents" refers to information including the requirements and wishes of the user regarding the project or system provided.

[0527] "Generating minutes" is the process of recording and documenting the contents of a meeting based on the hearing content.

[0528] "Requirements extraction" is the process of identifying and extracting the functions and conditions necessary for a project or system from the content of interviews.

[0529] A "template" is a document template with a predefined format and structure that is used when creating documents such as RFPs and specifications.

[0530] "RFP" stands for Request for Proposal, a request for proposal to request the provision of a project or service.

[0531] A "specification" is a document that lists the detailed requirements and specifications for a system or project.

[0532] "Facial expression analysis" is a technology that captures a user's facial expressions using a camera or other device and analyzes their emotions.

[0533] "Autocoding" is the process of automatically generating program code.

[0534] "Test creation" is the process of creating test cases to verify that the generated code works correctly.

[0535] "UI" stands for user interface and refers to the screen and operation methods that allow users to interact with the system.

[0536] "Customer feedback" refers to opinions and evaluations of systems and projects provided by users.

[0537] "AI technology" is a technology that uses artificial intelligence and is used to perform data analysis, automation, etc.

[0538] "Natural language processing technology" is a technology that allows computers to understand and analyze human language.

[0539] "Speech recognition technology" is a technology that converts speech into text.

[0540] This invention is a system that automatically generates minutes and extracts requirements from interview content, automatically creates RFPs and specifications using templates based on the requirements, and further analyzes the user's facial expressions to automatically code and create tests based on their emotions. A specific embodiment of this system is shown below.

[0541] Hardware and software used

[0542] Server: Performs data analysis, template embedding, code generation, and test case creation.

[0543] Terminal: Receives user input, captures facial expressions, and sends them to the server.

[0544] Camera: Used to capture the user's facial expressions.

[0545] Natural language processing technology (NLP): Used to analyze interview content and extract requirements.

[0546] Speech recognition technology: Used to convert the hearing content into text.

[0547] AI technology: Used for code generation and test case creation.

[0548] Templates: Used to automatically create RFPs and specifications.

[0549] Program processing

[0550] Receiving and analyzing the contents of the hearing

[0551] 1. The user enters the project requirements into the system.

[0552] 2. The terminal sends the input requirements to the server.

[0553] 3. The server converts the interview content into text using voice recognition technology and extracts requirements using natural language processing technology.

[0554] Automatic creation of RFPs and specifications

[0555] 1. The server automatically fills in each item in a pre-prepared template based on the extracted requirements.

[0556] 2. The server generates the final RFP and specifications based on the embedded template.

[0557] 3. The server sends the generated RFP and specifications to the terminal.

[0558] 4. The terminal displays the generated document to the user.

[0559] Automatic coding and test creation using an emotion engine

[0560] 1. The device captures the user's facial expression with a camera.

[0561] 2. The device analyzes the captured facial expression data to determine whether the user is showing confusion or hesitation.

[0562] 3. The device sends the analysis results to the server.

[0563] 4. The server receives the user's emotional data and determines that there is a problem with code generation or test case creation.

[0564] 5. The server identifies the problem area and regenerates code and creates test cases for that area.

[0565] 6. The server sends the modified code and test cases to the terminal.

[0566] 7. The terminal displays the modified code and test cases to the user.

[0567] Specific examples

[0568] For example, if a user enters "requirements for developing a new website," the server analyzes the requirements and automatically fills in the template's "project overview," "functional requirements," "technical requirements," etc. Finally, the completed RFP and specifications are provided to the user.

[0569] If the user shows a confused expression while generating code, the device captures that expression and sends it to the server. The server then uses this information to identify the problem area and regenerates the code or creates test cases. Finally, the corrected code and test cases are provided to the user.

[0570] Prompt Sentence Examples

[0571] "Automatically generate an RFP based on our new website development requirements."

[0572] "If the user looks confused, assume there is a problem with the code generation or test case creation and try again."

[0573] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0574] Step 1:

[0575] Users input project requirements into the system.

[0576] Input: Project requirements (e.g. new website development requirements)

[0577] Specific operation: The user enters the project requirements in text format into the input form on the terminal.

[0578] Step 2:

[0579] The terminal transmits the input requirements to the server.

[0580] Input: Project requirements entered by the user

[0581] Output: Project requirements data sent to the server

[0582] Specific operation: The terminal transmits the requirements input by the user to the server as a data packet.

[0583] Step 3:

[0584] The server converts the interview content into text using voice recognition technology and extracts requirements using natural language processing technology.

[0585] Input: Project requirements data

[0586] Output: Extracted requirements information

[0587] Specific operation: The server converts the received requirements data into text using speech recognition technology and extracts important requirements using natural language processing technology.

[0588] Step 4:

[0589] Based on the extracted requirements, the server automatically fills in each item in a pre-prepared template.

[0590] Input: Extracted requirements information, template

[0591] Output: Embedded template

[0592] Specific operation: The server automatically fills each item of the template with the extracted requirement information to complete the template.

[0593] Step 5:

[0594] The server generates the final RFP and specifications based on the embedded templates.

[0595] Input: Embedded template

[0596] Output: Generated RFP and specifications

[0597] Specific operation: The server automatically generates RFPs and specifications based on the embedded templates.

[0598] Step 6:

[0599] The server sends the generated RFP and specifications to the terminal.

[0600] Input: Generated RFP or specifications

[0601] Output: RFP and specifications sent to the terminal

[0602] Specific operation: The server sends the generated RFP and specifications to the terminal as a data packet.

[0603] Step 7:

[0604] The terminal displays the generated document to the user.

[0605] Input: RFP or specification sent from the server

[0606] Output: RFP or specification document displayed to the user

[0607] Specific operation: The terminal displays the received RFP and specifications to the user.

[0608] Step 8:

[0609] The device captures the user's facial expressions with a camera.

[0610] Input: User's facial expression

[0611] Output: Captured facial expression data

[0612] Specific operation: The device uses a camera to capture the user's facial expressions in real time.

[0613] Step 9:

[0614] The device analyzes the captured facial expression data to determine whether the user is showing confusion or hesitation.

[0615] Input: Captured facial expression data

[0616] Output: Analysis results (user's emotional state)

[0617] Specific operation: The device analyzes the captured facial expression data and determines the user's emotional state.

[0618] Step 10:

[0619] The terminal transmits the analysis results to the server.

[0620] Input: Analysis results (user's emotional state)

[0621] Output: Analysis results sent to the server

[0622] Specific operation: The terminal transmits the analysis result to the server as a data packet.

[0623] Step 11:

[0624] The server receives the user's emotional data and determines whether there is a problem with code generation or test case creation.

[0625] Input: Analysis results (user's emotional state)

[0626] Output: Identification of problem areas

[0627] Specific operation: Based on the received emotion data, the server determines whether there are any problems with code generation or test case creation.

[0628] Step 12:

[0629] The server identifies the problem areas and regenerates code and creates test cases for those areas.

[0630] Input: Problem location identification results

[0631] Output: Modified code and test cases

[0632] Specific operation: The server regenerates code and creates test cases for the identified problem areas.

[0633] Step 13:

[0634] The server sends the modified code and test cases to the terminal.

[0635] Input: Modified code and test cases

[0636] Output: Modified code and test cases sent to the terminal

[0637] Specific operation: The server sends the modified code and test cases to the terminal as data packets.

[0638] Step 14:

[0639] The terminal displays the modified code and test cases to the user.

[0640] Input: Modified code and test cases sent from the server

[0641] Output: The modified code and test cases that are displayed to the user

[0642] Specific operation: The terminal displays the received modified code and test cases to the user.

[0643] (Application example 2)

[0644] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0645] In conventional manufacturing process management systems, requirements extraction and RFP creation are done manually, which takes time and effort. Furthermore, if the user expresses confusion or hesitation, the system may not be able to capture that information, resulting in inappropriate code generation and test case creation. This reduces the efficiency of the manufacturing process and increases the likelihood of errors.

[0646] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0647] In this invention, the server includes means for automatically generating minutes and extracting requirements from the contents of interviews, means for automatically creating RFPs and specifications, means for automatically creating UIs and reflecting customer feedback, means for realizing automatic coding and test creation, means for detecting the user's facial expressions and correcting code generation and test case creation based on the user's emotions, and means for managing the manufacturing process based on the requirements. This enables efficient management of the manufacturing process and appropriate code generation and test case creation that reflects the user's emotions.

[0648] "Hearing content" refers to the content of conversations and interviews conducted to gather requirements and opinions from users and customers.

[0649] "Minutes generation" is the process of recording and documenting the contents of meetings and hearings.

[0650] "Requirements extraction" is the process of identifying and extracting the requirements necessary for a project or system from interview content and meeting minutes.

[0651] "RFP" stands for Request for Proposal. It is a document that lists the requirements and conditions of a project and requests proposals.

[0652] A "specification" is a document that details the requirements and design of a system or project.

[0653] "UI" stands for User Interface, and is the interface through which a user interacts with a system or application.

[0654] "Customer feedback" refers to opinions and evaluations provided by customers, and is information used to improve products and services.

[0655] "Autocoding" is the process of automatically generating program code.

[0656] "Test case creation" is the process of creating test scenarios to verify the operation of software.

[0657] "Emotion detection" is a technology that analyzes and identifies emotions from a user's facial expressions and voice.

[0658] "Manufacturing process management" is a management method for planning, executing, and monitoring production activities in the manufacturing industry.

[0659] A "template" is a model that defines the standard format and structure of a document or data.

[0660] A "camera" is a device for taking images or videos.

[0661] An "emotion detection engine" is software or hardware that analyzes and identifies emotions from image and audio data.

[0662] "AI technology" refers to artificial intelligence technology, which is a technology that mimics human intelligence using machine learning and data analysis.

[0663] "Natural language processing technology" is a technology that allows computers to understand and process human language.

[0664] A system for implementing the present invention includes the following major components:

[0665] 1. Automatic generation of meeting minutes and requirement extraction from interview content

[0666] The server converts the interview content into text using speech recognition technology and extracts requirements from the text using natural language processing technology. This automatically generates minutes of the interview conducted by the user and extracts the necessary requirements.

[0667] 2. Automatic creation of RFPs and specifications

[0668] The server automatically generates RFPs and specifications using templates based on the extracted requirements, allowing the necessary documents to be created quickly in the early stages of a project.

[0669] 3. Automatic UI creation and customer feedback

[0670] The server automatically creates a user interface (UI) based on the generated requirements and incorporates feedback from the customer, resulting in a highly usable UI being delivered quickly.

[0671] 4. Automated coding and test creation

[0672] The server uses AI technology to generate code and create test cases, which streamlines the development process and produces code with fewer errors.

[0673] 5. User Facial Expression Detection and Emotion-Based Code Modification

[0674] The server uses a camera and an emotion detection engine to analyze the user's facial expressions in real time, and if it detects an expression of confusion or hesitation, it identifies the problem area and regenerates it, making it possible to generate appropriate code and test cases that reflect the user's emotions.

[0675] 6. Manufacturing process management

[0676] The server manages the manufacturing process based on the requirements. It extracts the requirements, automatically generates RFPs and specifications using templates, and manages the progress of the manufacturing process. This enables efficient management of the manufacturing process.

[0677] Hardware and software used

[0678] Hardware:

[0679] Camera: Used to capture the user's facial expressions.

[0680] Computer: Used to run programs.

[0681] software:

[0682] OpenCV: Used to capture and display camera footage.

[0683] EmotionEngine: Used for emotion detection.

[0684] RFPGenerator: Used to automatically generate RFPs.

[0685] CodeGenerator: Used for code generation and problem identification.

[0686] Natural language processing technology: Used to convert interview content into text and extract requirements.

[0687] AI technology: Used for code generation and test case creation.

[0688] Specific examples

[0689] For example, if an engineer shows signs of confusion when designing a new manufacturing line in a factory, the system will identify the problem and regenerate the code. It will also automatically generate an RFP based on the requirements, helping to keep the project moving smoothly.

[0690] Prompt Sentence Examples

[0691] Please generate an RFP based on the following requirements for the design of a new manufacturing line:

[0692] Project name: New production line

[0693] Deadline:2023-12-31

[0694] Budget: 1,000,000

[0695] In this way, the system can efficiently manage the manufacturing process and reduce confusion for engineers.

[0696] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0697] Step 1:

[0698] The server converts the hearing content into text using speech recognition technology. As input, it receives the voice data of the hearing conducted by the user and converts it into text data using speech recognition technology. As output, it obtains the text of the hearing content.

[0699] Step 2:

[0700] The server extracts requirements from the textualized interview content using natural language processing technology. As input, it receives the text data obtained in step 1 and uses natural language processing technology to identify and extract requirements. As output, it obtains the extracted requirements.

[0701] Step 3:

[0702] The server automatically generates RFPs and specifications using templates based on the extracted requirements. As input, it receives the requirements data obtained in step 2 and embeds each item in the template. As output, it obtains the completed RFP and specifications.

[0703] Step 4:

[0704] The server automatically creates a user interface (UI) based on the generated requirements and incorporates customer feedback. It takes requirements data and customer feedback as input, generates the UI, and incorporates the feedback. The output is an updated UI.

[0705] Step 5:

[0706] The server uses AI technology to generate code and create test cases. It receives requirements data as input and generates code and test cases using AI technology. The generated code and test cases are obtained as output.

[0707] Step 6:

[0708] The server uses a camera and an emotion detection engine to analyze the user's facial expressions in real time, and if it detects an expression that indicates confusion or hesitation, it identifies the problem area and regenerates the code. As input, it receives the camera footage and the generated code, and analyzes the facial expression with the emotion detection engine. The output is the regenerated code.

[0709] Step 7:

[0710] The server manages the manufacturing process based on the requirements. As input, it receives the requirements data and generated RFPs and specifications, and manages the progress of the manufacturing process. As output, it obtains an efficiently managed manufacturing process.

[0711] Example 3

[0712] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0713] Conventional systems have the problem that it takes a lot of time and effort to design the user interface, incorporate customer feedback, generate code, and create test cases. Furthermore, it is difficult to improve the user experience because they are unable to recognize user emotions in real time and dynamically change the user interface based on that. To solve these issues, a system that automates and efficiently performs these processes is needed.

[0714] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0715] In this invention, the server includes means for automatically generating minutes and extracting requirements from the contents of the interview, means for automatically creating a requirements specification, means for automatically creating a user interface and reflecting customer feedback, means for realizing automatic coding and test case creation, and means for recognizing user emotions and changing the user interface based on those emotions. This not only enables efficient design of the user interface, reflection of customer feedback, code generation, and test case creation, but also enables dynamic changes to the user interface in response to user emotions.

[0716] "Hearing content" refers to information such as requirements, opinions, and feedback collected from users and customers.

[0717] "Generating minutes" is the process of recording and documenting the contents of meetings and discussions based on the interview content.

[0718] "Requirements elicitation" is the process of identifying and extracting the requirements necessary for a system or project from meeting minutes and interview content.

[0719] A "requirements specification" is a document that details the requirements of a system or project and serves as a guideline for development.

[0720] "User interface" refers to the screen and operation method that allows a user to interact with a system.

[0721] "Customer feedback" refers to information such as opinions, evaluations, and areas for improvement provided by users and customers.

[0722] "Auto-coding" is the process by which a system automatically generates program code.

[0723] A "test case" is a specific test scenario for verifying whether the generated code operates correctly.

[0724] "Emotion recognition" is a technology that analyzes and recognizes emotions from a user's voice and facial expressions.

[0725] "Dynamic user interface modification" is the process of modifying a user interface in real time in response to user sentiment and feedback.

[0726] This invention is a system that combines automatic creation of a user interface (UI), reflection of customer feedback, automatic coding and test case creation, and emotion recognition. Specific embodiments of this system are described below.

[0727] Hardware and software used

[0728] 1. Server

[0729] UI design generation engine: Automatically generate initial UI designs based on requirements.

[0730] Code generation AI model: Automatically generates program code based on requirements.

[0731] Test case generation engine: Automatically creates test cases based on the generated code.

[0732] Emotion recognition engine: Recognizes emotions from the user's voice and facial expressions.

[0733] 2. Terminal

[0734] Camera: Captures the user's facial expressions.

[0735] Microphone: Captures the user's voice.

[0736] Keyboard and mouse: Devices that allow the user to input feedback.

[0737] 3. Users

[0738] Provide feedback: Enter your thoughts and suggestions for improvements to the UI.

[0739] Emotional expression: Expressing emotions through facial expressions and vocalizations.

[0740] Program processing

[0741] 1. Automatic UI generation

[0742] The server automatically generates an initial UI design based on the requirements. For example, it sends instructions such as "Place the button in the top right corner and set the background color to white" to the UI design generation engine.

[0743] 2. Collect customer feedback

[0744] The user reviews the generated UI and provides feedback, for example, by typing "I want the button color to be changed to blue."

[0745] The terminal transmits the feedback input by the user to the server.

[0746] 3. UI refinement

[0747] The server receives feedback from the user and refines the UI. For example, it sends specific instructions, such as "change the button color to blue," to the UI design generation engine.

[0748] 4. Automated coding

[0749] The server generates code using AI technology based on the requirements. For example, it sends instructions such as "generate code for the user registration function" to the code-generation AI model.

[0750] 5. Automatic test case creation

[0751] The server automatically creates test cases based on the generated code. For example, it sends an instruction such as "Create a test case for the user registration function" to the test case generation engine.

[0752] 6. Emotion recognition

[0753] The device captures the user's voice and facial expressions through a camera and microphone. For example, if the user makes a surprised expression, the device sends that information to the server.

[0754] The server analyzes the captured data and recognizes the user's emotions.

[0755] 7. Emotion-Based UI Changes

[0756] The server automatically changes the UI based on the recognized emotion: for example, if the user makes a facial expression indicating joy, it changes the UI color and layout to be bright and cheerful.

[0757] Specific examples

[0758] Example prompt: "Generate code for the user registration function and create a test case."

[0759] The server receives the prompt and sends an instruction to the code generation AI model to "generate code for the user registration function."

[0760] The server analyzes the generated code and sends an instruction to the test case generation engine to "create a test case for the user registration function."

[0761] In this way, the system realizes a combination of functions that automatically create a UI, incorporate customer feedback, auto-code, create test cases, and recognize emotions through each processing step. This not only enables efficient user interface design, incorporate customer feedback, generate code, and create test cases, but also enables dynamic changes to the user interface in response to user emotions. The flow of the specific processing in Example 3 is explained using Figure 21.

[0762] Step 1:

[0763] Auto-generated UI

[0764] Input: Requirements (e.g., button placement, background color, number of text fields, etc.)

[0765] The server automatically generates an initial UI design based on the requirements using software called a "UI design generation engine."

[0766] Data processing: Analyzes requirements and sends instructions to the UI design generation engine.

[0767] Output: Initial UI design

[0768] Specific operation: The server sends instructions such as "Place the button in the top right corner and set the background color to white" to the UI design generation engine.

[0769] Step 2:

[0770] Collecting customer feedback

[0771] Input: Initial UI design

[0772] The user reviews the generated UI and provides feedback.

[0773] The terminal sends the feedback input by the user to the server, and uses hardware such as a keyboard and mouse.

[0774] Data processing: Feedback is sent to the server as text data.

[0775] Output: User feedback

[0776] Specific operation: The user types "I want the button color to be changed to blue," and the device sends that information to the server.

[0777] Step 3:

[0778] UI refinement

[0779] Input: User feedback

[0780] The server receives feedback from users and refines the UI using software called a "UI design generation engine."

[0781] Data processing: Analyze the feedback and send instructions to the UI design generation engine.

[0782] Output: Refined UI design

[0783] Specific operation: The server sends specific instructions, such as "change the button color to blue," to the UI design generation engine.

[0784] Step 4:

[0785] Auto-coding

[0786] Input: Requirements (e.g. user registration function)

[0787] The server generates code based on the requirements using AI technology. The software used is a "code generation AI model."

[0788] Data processing: Analyzing requirements and sending instructions to code-generating AI models.

[0789] Output: Generated code

[0790] Specific operation: The server sends instructions such as "Generate code for the user registration function" to the code generation AI model.

[0791] Step 5:

[0792] Automatic test case creation

[0793] Input: Generated code

[0794] The server automatically creates test cases based on the generated code using software called a "test case generation engine."

[0795] Data processing: Analyzes the generated code and sends instructions to the test case generation engine.

[0796] Output: Generated test cases

[0797] Specific operation: The server sends an instruction to the test case generation engine, such as "Create a test case for the user registration function."

[0798] Step 6:

[0799] emotion recognition

[0800] Input: User's voice and facial expressions

[0801] The device captures the user's voice and facial expressions through a camera and microphone. The hardware used is a "camera" and a "microphone."

[0802] The server analyzes the captured data and recognizes the user's emotions using software called an "emotion recognition engine."

[0803] Data processing: Analyzes the captured data and sends instructions to the emotion recognition engine.

[0804] Output: Recognized emotion data

[0805] Specific operation: The device performs processes such as "capturing the user's facial expressions and sending them to the server."

[0806] Step 7:

[0807] Emotion-based UI changes

[0808] Input: Recognized emotion data

[0809] The server automatically changes the UI based on the recognized emotion using software called a "UI design generation engine."

[0810] Data processing: Based on the information obtained from the emotion recognition engine, instructions are sent to the UI design generation engine.

[0811] Output: Modified UI design

[0812] Specific operation: The server performs processing such as "because the user has made an expression showing joy, change the background color of the UI to yellow."

[0813] (Application example 3)

[0814] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0815] Conventional systems do not automatically generate or update UIs that take user emotions into account, and as a result, the user experience is not sufficiently improved. Furthermore, there is a lack of means to effectively collect and analyze customer feedback and reflect it in UI improvements, making it difficult to increase user satisfaction.

[0816] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0817] In this invention, the server includes means for automatically generating minutes and extracting requirements from the contents of interviews, means for automatically creating requirements specifications and design documents, means for automatically creating a UI and incorporating customer feedback, means for automatically implementing coding and test creation, means for recognizing user emotions and automatically generating and updating a UI based on those emotions, and means for collecting customer feedback, analyzing the data, and identifying areas for improvement in the UI. This makes it possible to provide an optimal UI in real time that corresponds to user emotions and effectively incorporate customer feedback.

[0818] "Hearing content" refers to information such as requirements, opinions, and feedback collected from users and customers.

[0819] "Minutes generation" is the process of recording and documenting the contents of meetings and discussions.

[0820] "Requirements elicitation" is the process of identifying the requirements for a system or project from collected information.

[0821] A "requirements specification" is a document that details the requirements for a system or project.

[0822] A "design document" is a document that provides detailed information about the design of a system or project.

[0823] "UI" stands for user interface and refers to the screens and operating methods that allow users to interact with systems and applications.

[0824] "Customer feedback" refers to information such as opinions, evaluations, and areas for improvement provided by customers.

[0825] "Autocoding" is the process of automatically generating program code.

[0826] "Test creation" is the process of creating test cases to verify the operation of a system or application.

[0827] "User emotion" refers to an emotional state such as joy, surprise, anger, or dissatisfaction exhibited by a user.

[0828] "Emotion recognition" is a technology that identifies a user's emotions from their voice and facial expressions.

[0829] "Data analysis" is the process of analyzing collected data and extracting useful information and patterns.

[0830] A "generative AI model" is a model that uses artificial intelligence technology to generate new designs and content.

[0831] A "prompt sentence" is an input sentence used to give instructions to a generative AI model.

[0832] The following system configuration will be described as an embodiment of the present invention.

[0833] System Configuration

[0834] The system consists of the following major components:

[0835] 1. A method to automatically generate meeting minutes and extract requirements from interview content

[0836] 2. Means of automatically creating requirements specifications and design documents

[0837] 3. A way to automate UI creation and incorporate customer feedback

[0838] 4. Means of automated coding and test creation

[0839] 5. A method to recognize user emotions and automatically generate and update UI based on those emotions

[0840] 6. A means to collect customer feedback, analyze data, and identify areas for UI improvement

[0841] Program processing explanation

[0842] A means to automatically generate minutes and extract requirements from interview content

[0843] The server converts the interview content into text using speech recognition technology and extracts requirements from the text using natural language processing technology. Specifically, it collects voice data and converts it into text using speech recognition software (e.g., Google Speech-to-Text API). It then extracts requirements using natural language processing technology (e.g., NLTK or SpaCy).

[0844] A means of automatically creating requirements specifications and design documents

[0845] The server automatically generates requirements specifications and design documents based on the extracted requirements. It uses a generative AI model (e.g., OpenAI's GPT-4) to generate detailed documents based on a prompt.

[0846] A way to automate UI creation and incorporate customer feedback

[0847] The server automatically generates a UI using a generative AI model, collects customer feedback, and updates the UI using a feedback collection tool (e.g., Google Forms API) and analyzes it with a data analysis tool (e.g., Python's Pandas).

[0848] A means to automate coding and test creation

[0849] The server uses artificial intelligence technology to generate code and create test cases. It uses a generative AI model to generate code and test cases based on prompts.

[0850] A method to recognize user emotions and automatically generate and update UI based on those emotions

[0851] The device uses the smartphone's camera and microphone to capture the user's facial expressions and voice, and recognizes their emotions using emotion recognition AI (e.g., Microsoft Azure Cognitive Services' Emotion API). Based on the recognized emotions, the device sends prompt sentences to a generative AI model to generate an appropriate UI design.

[0852] A means to collect customer feedback, analyze data, and identify areas for UI improvement

[0853] The server collects customer feedback and analyzes it using data analysis tools. Based on the analysis results, it identifies areas for improvement in the UI and reflects them in the next UI generation.

[0854] Specific examples

[0855] For example, if a user smiles while browsing a product, the device will send a prompt to the generative AI model saying, "Generate a bright and cheerful UI," which will generate a UI with a colorful and positive design.

[0856] Prompt Sentence Examples

[0857] "If the user smiles, generate a bright and cheerful UI."

[0858] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0859] Step 1:

[0860] The device captures the user's facial expressions and voice. Specifically, it uses the smartphone's camera and microphone to collect video and audio data of the user's face. The input is the user's real-time facial expressions and voice, and the output is the captured video and audio data.

[0861] Step 2:

[0862] The device sends the captured video and audio data to an emotion recognition AI. Specifically, it uses the Emotion API from Microsoft Azure Cognitive Services to recognize the user's emotions. The input is the captured video and audio data, and the output is the recognized emotion data.

[0863] Step 3:

[0864] The server generates a prompt sentence based on the recognized emotion data. Specifically, if the emotion data indicates "happiness" or "surprise," the server generates the prompt sentence "Please generate a bright and fun UI." The input is the recognized emotion data, and the output is the generated prompt sentence.

[0865] Step 4:

[0866] The server sends the generated prompt sentence to a generative AI model. Specifically, it uses OpenAI's GPT-4 to generate a UI design based on the prompt sentence. The input is the generated prompt sentence, and the output is the generated UI design.

[0867] Step 5:

[0868] The server sends the generated UI design to the device. Specifically, the generated UI design is applied to the smartphone app and displayed to the user. The input is the generated UI design, and the output is the UI displayed to the user.

[0869] Step 6:

[0870] Users provide feedback on the displayed UI. Specifically, a feedback collection tool (e.g., Google Forms API) is used to collect user opinions and ratings. The input is the user feedback, and the output is the collected feedback data.

[0871] Step 7:

[0872] The server analyzes the collected feedback data using a data analysis tool. Specifically, it uses Python's Pandas to analyze the feedback data and identify areas for UI improvement. The input is the collected feedback data, and the output is the identified areas for UI improvement.

[0873] Step 8:

[0874] The server reflects the identified UI improvements in the next UI generation. Specifically, it sends a new prompt to the generative AI model to generate an improved UI design. The input is the identified UI improvements, and the output is the improved UI design.

[0875] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0876] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0877] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0878] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0879] [Second embodiment]

[0880] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0881] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0882] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0883] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0884] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0885] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0886] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0887] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0888] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0889] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0890] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0891] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0892] "Example 1"

[0893] In one embodiment of the present invention, the system automatically generates meeting minutes and extracts requirements from the contents of interviews. Specifically, the system converts the interview contents into text using speech recognition technology and extracts requirements from the text using natural language processing technology. For example, the system converts the audio data during the interview into text in real time, and extracts keywords and phrases from the text to create requirements.

[0894] "Example 2"

[0895] The system also automatically creates RFPs and specifications. Specifically, it generates RFPs and specifications using templates based on the extracted requirements. For example, it automatically fills in each item in the template based on the requirements to generate the RFP and specifications.

[0896] "Example 3"

[0897] Furthermore, the system can automatically create a UI and incorporate customer feedback. Specifically, the system automatically generates a UI based on requirements and then updates the UI based on customer feedback. For example, the system can generate an initial UI design based on requirements and then refine the UI based on customer feedback.

[0898] "Example 4"

[0899] The system also enables automatic coding and test creation. Specifically, it uses AI technology to generate code and create test cases. For example, AI generates code based on requirements and simultaneously creates test cases. This allows developers to significantly reduce the effort required for coding and test creation.

[0900] The processing flow of each embodiment will be described below.

[0901] "Example 1"

[0902] Step 1: The system converts the audio data being heard into text in real time using voice recognition technology.

[0903] Step 2: The system then uses natural language processing techniques to extract keywords and phrases from the text, which automatically extracts requirements.

[0904] Step 3: Finally, minutes are generated based on the extracted requirements. This automatically generates minutes and extracts requirements from the interview content.

[0905] "Example 2"

[0906] Step 1: The system selects an RFP or specification template based on the extracted requirements. Step 2: Next, each item in the selected template is automatically filled in based on the requirements. Step 3: Finally, the filled-in template is output as an RFP or specification. This allows the RFP or specification to be automatically created.

[0907] "Example 3"

[0908] Step 1: The system automatically generates a UI based on the requirements, using AI technology.

[0909] Step 2: Next, the system receives feedback from the customer.

[0910] Step 3: Finally, update the UI based on the feedback received, which automates the creation of the UI and incorporates customer feedback.

[0911] "Example 4"

[0912] Step 1: The system generates code based on the requirements. This is done using AI technology.

[0913] Step 2: At the same time, the system generates test cases, also using AI technology.

[0914] Step 3: Finally, the generated code and test cases are output, which allows for automated coding and test creation.

[0915] Example 1

[0916] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0917] In traditional meetings and hearings, minutes and requirements extraction were often done manually, which was time-consuming and labor-intensive. Manual minutes creation could lead to information leaks and misunderstandings, hindering efficient project management. Furthermore, the lack of automation in requirements extraction and minutes creation made it difficult to organize information in the early stages of a project.

[0918] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0919] In this invention, the server includes means for acquiring voice data, means for transmitting the acquired voice data to the server, means for converting the voice data into text using voice recognition technology, means for analyzing the text data and extracting requirements using natural language processing technology, means for automatically generating minutes based on the extracted requirements, and means for providing the generated minutes to a user. This makes it possible to automatically convert the contents of meetings and hearings into text, extract requirements, and generate minutes. This eliminates the need for manual minutes creation, prevents information leaks and misunderstandings, and enables efficient project management.

[0920] "Audio data" refers to data in which audio information from meetings, hearings, etc. is recorded in digital format.

[0921] A "server" is a computer system that processes and stores data over a network.

[0922] "Voice recognition technology" is a technology that analyzes voice data and converts it into text data.

[0923] "Text data" is data expressed as character information.

[0924] "Natural language processing technology" is a technology that analyzes text data, understands its meaning, and extracts information.

[0925] "Requirements" are the conditions and specifications required for a project or system.

[0926] "Minutes" are documents that record the contents of meetings or hearings.

[0927] "Automatic generation" refers to the system automatically creating data or documents without manual intervention.

[0928] "User" refers to a person or organization that uses the system.

[0929] "RFP" stands for Request for Proposal, a document that requests proposals for projects or services.

[0930] A "specification" is a document that lists the detailed specifications of a system or project.

[0931] "UI" stands for user interface, and refers to the screen and operation methods that allow users to interact with the system.

[0932] "Customer feedback" refers to opinions and requests provided by customers.

[0933] "Auto-coding" is a technique for automatically generating program code.

[0934] "Test creation" refers to creating test cases to check the operation of a system or program.

[0935] "Artificial intelligence technology" is a technology that allows machines to imitate human intelligence and learn and reason.

[0936] The present invention is a system that automatically converts the contents of a meeting or hearing into text, extracts requirements, and generates minutes. A specific embodiment of this system will be described below.

[0937] Hardware and Software Configuration

[0938] 1. Acquiring audio data

[0939] Users acquire audio data from meetings and hearings using devices such as laptops and smartphones. The devices are equipped with microphones, and audio data is recorded by launching a recording application.

[0940] 2. Sending audio data

[0941] The device sends the acquired audio data to the server. Specifically, the device application uploads the audio data to the server as an HTTP request. At this time, the audio data is sent in an appropriate format, such as WAV or MP3.

[0942] 3. Converting audio data into text

[0943] The server processes the received voice data in real time. Specifically, it converts the voice data into text data using voice recognition technology. For example, it uses the Google Cloud Speech-to-Text API. The server sends the voice data to the API and stores the returned text data.

[0944] 4. Text Data Analysis

[0945] The server analyzes the text data using natural language processing (NLP) techniques. Specifically, it uses the Python libraries spaCy and NLTK to extract keywords and important phrases from the text. The server passes the text data to the analysis module and obtains the analysis results.

[0946] 5. Requirements Extraction

[0947] The server extracts requirements from the analysis results. Specifically, it lists important requirements from the conversation based on the extracted keywords and phrases. For example, requirements such as "project start date," "budget," and "person in charge" are extracted.

[0948] 6. Generate minutes

[0949] The server automatically generates minutes based on the extracted requirements. Specifically, it organizes the main points of the conversation and decisions made, and creates text-based minutes. The server uses templates to format the minutes.

[0950] 7. Providing Results

[0951] The server provides the generated minutes to the user. Specifically, it generates the minutes in PDF or Word format and sends a download link to the user's device. The user can click the link to download the minutes and check their contents.

[0952] Specific examples

[0953] For example, consider a scenario where a user is hearing requirements for a new project. The user uses their laptop to record the conversation, and the audio data is sent to a server. The server converts the audio into text using the Google Cloud Speech-to-Text API, and then analyzes the text using spaCy. As a result of the analysis, requirements such as "project start date," "budget," and "person in charge" are extracted. Finally, minutes containing these requirements are automatically generated and provided to the user.

[0954] Prompt Sentence Examples

[0955] Examples of prompts to be input to a generative AI model include:

[0956] "Please transcribe the following audio and extract the key requirements. The audio contains requirements for a new project."

[0957] Using this prompt, the generative AI model can properly process the voice data and extract the necessary information.

[0958] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0959] Step 1: Acquire audio data

[0960] Users use devices such as laptops and smartphones to capture audio data from meetings and hearings. Specifically, they start up the device's recording application and press the record button to begin capturing audio data. The input is the audio from the meeting or hearing, and the output is digital audio data.

[0961] Step 2: Sending audio data

[0962] The device sends the acquired audio data to the server. Specifically, the device application uploads the audio data to the server as an HTTP request. The audio data is sent in an appropriate format, such as WAV or MP3. The input is digital audio data, and the output is the audio data sent to the server.

[0963] Step 3: Converting audio data to text

[0964] The server processes the received voice data in real time. Specifically, it uses voice recognition technology to convert the voice data into text data. For example, it uses the Google Cloud Speech-to-Text API. The server sends the voice data to the API and saves the returned text data. The input is the voice data sent to the server, and the output is text data.

[0965] Step 4: Analyzing the text data

[0966] The server analyzes the text data using natural language processing (NLP) techniques. Specifically, it uses the Python libraries spaCy and NLTK to extract keywords and important phrases from the text. The server passes the text data to the analysis module and obtains the analysis results. The input is the text data, and the output is the keywords and important phrases that are the analysis results.

[0967] Step 5: Eliciting requirements

[0968] The server extracts requirements from the analysis results. Specifically, it lists important requirements from the conversation based on the extracted keywords and phrases. For example, requirements such as "project start date," "budget," and "person in charge" are extracted. The input is the keywords and important phrases from the analysis results, and the output is the extracted requirements.

[0969] Step 6: Generate the transcript

[0970] The server automatically generates minutes based on the extracted requirements. Specifically, it organizes the main points of the conversation and decisions and creates text-based minutes. The server uses a template to format the minutes. The input is the extracted requirements, and the output is the generated minutes.

[0971] Step 7: Delivering results

[0972] The server provides the generated minutes to the user. Specifically, it generates the minutes in PDF or Word format and sends a download link to the user's device. The user can click the link to download the minutes and check their contents. The input is the generated minutes, and the output is the minutes provided to the user.

[0973] (Application example 1)

[0974] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0975] In autonomous vehicles, smooth communication between passengers and vehicle systems is required, and passenger instructions and requests must be accurately reflected. However, with current systems, it is difficult to convert passenger voice instructions into text in real time, extract requirements, and reflect them in the vehicle system. In addition, since minutes are generated and requirements are extracted manually, this is inefficient and lacks accuracy. Technology is needed to solve these problems.

[0976] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0977] In this invention, the server includes means for automatically generating minutes and extracting requirements from the contents of the interview, means for automatically creating RFPs and specifications, means for automatically creating a UI and reflecting customer feedback, means for realizing automatic coding and test creation, means for converting the contents of the interview into text using speech recognition technology and extracting requirements from the text using natural language processing technology, means for converting conversations inside the autonomous vehicle into text in real time, generating minutes, and extracting requirements, and means for reflecting the extracted requirements in the vehicle system. This makes it possible to accurately reflect passenger instructions and requests in real time.

[0978] "Hearing content" refers to audio information collected from users and customers.

[0979] "Minutes generation" is the process of recording the contents of a meeting or conversation as a document based on the contents of the hearing.

[0980] "Requirements extraction" is the process of extracting important keywords and phrases from the interview content and organizing them as system or project requirements.

[0981] "RFP" stands for Request for Proposal, a document requesting proposals for the provision of projects or services.

[0982] A "specification" is a document that describes the detailed requirements and functions of a system or project.

[0983] "UI" stands for user interface and refers to the screens and methods of operation that allow users to interact with systems and applications.

[0984] "Customer feedback" refers to information such as opinions, requests, and evaluations provided by customers.

[0985] "Autocoding" is the process of automatically generating program code.

[0986] "Test creation" is the process of creating test cases to verify the operation of a system or application.

[0987] "Speech recognition technology" is a technology that converts speech into text.

[0988] "Natural language processing technology" is a technology for analyzing text data and understanding its meaning and structure.

[0989] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[0990] "Real-time" means processing or reacting immediately, without delay.

[0991] A "vehicle system" is a system for controlling the operation and functions of an autonomous vehicle.

[0992] The following system configuration is used as an embodiment of this invention. The server includes means for automatically generating minutes and extracting requirements from the contents of interviews, means for automatically creating RFPs and specifications, means for automatically creating UIs and reflecting customer feedback, means for realizing automatic coding and test creation, means for converting the contents of interviews into text using speech recognition technology and extracting requirements from the text using natural language processing technology, means for converting conversations inside an autonomous vehicle into text in real time, generating minutes, and extracting requirements, and means for reflecting the extracted requirements in the vehicle system.

[0993] Program processing explanation

[0994] The server captures voice data using the smartphone's microphone and converts it to text using the Google Speech Recognition API. The converted text is then analyzed using natural language processing models in the Transformers library to extract important keywords and phrases, ensuring passengers' instructions and requests are accurately reflected in real time.

[0995] Specifically, the system converts the interview content into text using speech recognition technology and then extracts requirements from the text using natural language processing technology. For example, if a passenger says, "Turn right at the next intersection," the system extracts the requirements "next intersection" and "turn right" and reflects them in the autonomous vehicle's system.

[0996] Hardware and software used

[0997] Hardware: Smartphone microphone

[0998] Software: Python, Google Speech Recognition API, Transformers library

[0999] Specific examples

[1000] For example, if a passenger says, "Turn right at the next intersection," the system extracts the requirements "next intersection" and "turn right" and reflects them in the autonomous vehicle's system.

[1001] Prompt Sentence Examples

[1002] Passenger: "Turn right at the next intersection."

[1003] System: Extracted the requirements "next intersection" and "turn right."

[1004] In this way, passenger instructions can be recognized in real time and reflected in the operation of the autonomous vehicle.

[1005] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1006] Step 1:

[1007] The server acquires voice data using the smartphone's microphone. The user's voice is input, and voice data is collected through the microphone. This voice data is then sent to the server in real time.

[1008] Step 2:

[1009] The server converts the acquired voice data into text using the Google Speech Recognition API. The input is voice data and the output is text data. Using voice recognition technology, the voice data is analyzed and converted into the corresponding text.

[1010] Step 3:

[1011] The server analyzes the converted text data using a natural language processing model from the Transformers library. The input is text data, and the output is extracted keywords and phrases. Natural language processing techniques are used to extract important keywords and phrases from the text data.

[1012] Step 4:

[1013] The server organizes the extracted keywords and phrases as requirements and reflects them in the autonomous vehicle system. The input is the extracted keywords and phrases, and the output is the requirements reflected in the vehicle system. The requirements are organized and data processing is performed to apply them to the vehicle system.

[1014] Step 5:

[1015] The server confirms that the requirements have been accurately reflected and provides feedback to the user. The input is the requirements reflected in the vehicle system, and the output is a feedback message to the user. A process is performed to generate the feedback message and notify the user.

[1016] Example 2

[1017] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1018] In the conventional system, tasks such as generating minutes from interviews, extracting requirements, and creating RFPs and specifications based on the requirements were all done manually, which was time-consuming and labor-intensive. There was also a high possibility of human error occurring in the extraction of requirements and document creation. Furthermore, there was a lack of efficient means for creating UIs, incorporating customer feedback, and automated coding and test creation. To solve these issues, a more efficient and accurate system is needed.

[1019] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1020] In this invention, the server includes a means for automatically generating minutes and extracting requirements from the interview content, a means for receiving requirements documents and extracting requirements using a generative AI model, and a means for automatically creating RFPs and specifications using templates based on the extracted requirements. This automates the process from requirements extraction to creating RFPs and specifications, significantly reducing time and labor. It also reduces human error and enables more accurate document creation.

[1021] "Hearing contents" refers to information including the requirements and wishes regarding the project or system provided by the user.

[1022] "Generating minutes" is the process of recording and documenting the contents of a meeting based on the hearing content.

[1023] "Requirements extraction" is the process of extracting necessary information from interview content and requirements documents and organizing it as structured data.

[1024] A "generative AI model" is a model that uses artificial intelligence technology to generate text and extract requirements.

[1025] A "prompt sentence" is an input sentence that gives specific instructions to a generative AI model.

[1026] A "template" is a predefined document template used when creating RFPs and specifications.

[1027] "RFP" stands for Request for Proposal, a document used to request proposals for projects or services.

[1028] A "specification" is a document that lists the detailed requirements and specifications for a system or project.

[1029] "UI" stands for user interface and refers to the screen and operation methods that users use to interact with a system.

[1030] "Customer feedback" refers to opinions and requests provided by customers, and is information that can be used to improve and adjust the system.

[1031] "Autocoding" is the process of automatically generating program code.

[1032] "Test creation" is the process of creating test cases to verify the operation of a system or program.

[1033] This invention relates to a system that automatically generates minutes and extracts requirements from interview content, receives requirements documents, extracts requirements using a generative AI model, and automatically creates RFPs and specifications using templates based on the extracted requirements.

[1034] The server receives the requirements document provided by the user and extracts requirements using a generative AI model. Specifically, the server performs the following steps:

[1035] First, the user uploads a requirements document through the system interface. The requirements document can be provided in the form of a text file, PDF, Word, etc. The server receives the uploaded requirements document and analyzes its contents.

[1036] Next, the server generates a prompt based on the requirements document. A prompt is an input sentence that provides specific instructions to the generative AI model. For example, the following prompt might be generated:

[1037] Generate an RFP based on the requirements below.

[1038] Project Objective: Development of a new customer management system

[1039] Goal: Efficiently manage and analyze customer data

[1040] Functional requirements: Ability to register, search, update, and delete customer information

[1041] Non-functional requirements: high security, scalability

[1042] The server inputs this prompt into a generative AI model to extract requirements. The generative AI model, such as OpenAI's GPT-4, extracts the necessary information from the requirements document based on the prompt and saves it as structured data.

[1043] The server then selects a pre-prepared RFP or specification template and automatically fills in each item in the template with the extracted requirements. For example, the extracted "project objectives" and "goals" are filled in the "Project Overview" section of the template.

[1044] Finally, the server generates the final RFP and specifications based on the requirements embedded in the template. The generated documents are output in PDF or Word format and made available for download by users. Users can download and review the generated RFP and specifications from the system interface.

[1045] This system automates the process from requirements extraction to the creation of RFPs and specifications, significantly reducing time and effort, and also reducing human error, resulting in more accurate document creation.

[1046] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1047] Step 1:

[1048] A user uploads a requirements document.

[1049] The user logs in to the system's web interface and clicks the "Upload requirements document" button. A file selection dialog appears, and the user selects and uploads the requirements document. The input is the requirements document, and the output is the requirements document saved on the server.

[1050] Step 2:

[1051] The server receives the requirements document and parses the contents.

[1052] The server receives the uploaded requirements document and analyzes its contents. Specifically, it reads the text of the requirements document and identifies each section. The input is the uploaded requirements document, and the output is the parsed text data of the requirements document.

[1053] Step 3:

[1054] The server generates a prompt.

[1055] The server generates a prompt sentence based on the content of the parsed requirements document. For example, if the requirements document states "Project Objective: Develop a new customer management system," the prompt sentence will contain "Project Objective: Develop a new customer management system." The input is the text data of the parsed requirements document, and the output is the generated prompt sentence.

[1056] Step 4:

[1057] The server uses a generative AI model to extract requirements.

[1058] The server inputs the generated prompt sentence into a generative AI model to extract requirements. Examples of generative AI models that can be used include OpenAI's GPT-4. The generative AI model extracts necessary information from the requirements document based on the prompt sentence and saves it as structured data. The input is the prompt sentence, and the output is structured data of the extracted requirements.

[1059] Step 5:

[1060] The server selects a template and fills it with the extracted requirements.

[1061] The server selects a pre-prepared RFP or specification template. Then, it automatically embeds the extracted requirements into each section of the template. For example, the extracted "Project Objectives" and "Goals" are embedded in the "Project Overview" section. The input is structured data of the extracted requirements, and the output is a template with the embedded requirements.

[1062] Step 6:

[1063] The server generates the final RFP and specifications and provides them to the user.

[1064] The server generates the final RFP or specification based on the requirements embedded in the template. The generated document is output in PDF or Word format and made available for download by the user. The user can download and review the generated RFP or specification from the system interface. The input is the template with the embedded requirements, and the output is the generated RFP or specification.

[1065] (Application example 2)

[1066] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1067] In the conventional system, processes such as generating minutes from interviews, extracting requirements, creating RFPs and specifications, automatically creating UI, incorporating customer feedback, and automatically coding and creating tests were all done manually, which was time-consuming and labor-intensive. Additionally, creating specifications for machinery used in factories was also done manually, which was inefficient and prone to errors. To solve these issues, a system was needed that could automate these processes and perform them efficiently and accurately.

[1068] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1069] In this invention, the server includes a means for automatically generating minutes and extracting requirements from interview content, a means for automatically creating RFPs and specifications, a means for automatically creating UIs and incorporating customer feedback, a means for realizing automatic coding and test creation, and a means for automatically generating specifications for machines used in factories. This automates these processes, making them efficient and accurate.

[1070] "Hearing content" refers to oral information collected through meetings, interviews, etc.

[1071] "Generating minutes" means creating a document that records the contents of a meeting based on the contents of the hearing.

[1072] "Requirements elicitation" is the process of identifying and extracting necessary requirements from interviews and other sources of information.

[1073] "RFP" stands for Request for Proposal, a document that solicits proposals for a specific project or service.

[1074] A "specification" is a document that details the functions, performance, design, etc. of a product or system.

[1075] "UI" stands for user interface, the means by which a user interacts with a system or application.

[1076] "Customer feedback" refers to information such as opinions, requests, and evaluations provided by customers.

[1077] "Auto-coding" means automatically generating program code.

[1078] "Test creation" means creating test cases to verify the operation of software or systems.

[1079] "Machinery used in a factory" refers to equipment and facilities used for production and processing in the manufacturing industry.

[1080] "Automatic specification generation" means automatically creating specifications based on specific requirements and conditions.

[1081] A system for carrying out this invention is configured as follows: The server includes means for automatically generating minutes and extracting requirements from interview content, means for automatically creating RFPs and specifications, means for automatically creating UIs and incorporating customer feedback, means for realizing automatic coding and test creation, and means for automatically generating specifications for machines used in factories.

[1082] The server converts the interview content into text using speech recognition technology and extracts requirements from the text using natural language processing technology. This makes it possible to automatically generate meeting minutes and extract requirements from the interview content. Furthermore, it uses a generative AI model to generate prompt sentences based on the requirements and automatically create RFPs and specifications.

[1083] Specifically, the server performs the following process. First, the user inputs the interview content by voice, and the server converts it into text using speech recognition technology. Next, it uses natural language processing technology to extract requirements from the text. The extracted requirements are input into a generative AI model, which generates prompt text. Based on this prompt text, an RFP and specifications are automatically created.

[1084] The server also includes a means for automatically generating specifications for machines used in the factory. When a user inputs requirements for machines in the factory, the server analyzes them and automatically generates specifications using an appropriate template. For example, when requirements for the installation of a new manufacturing line are input, the server generates a prompt sentence such as the following:

[1085] Example prompt sentence:

[1086] Generate a specification based on the following requirements:

[1087] 1. Installation of a new production line

[1088] 2. Machine layout and connection

[1089] 3. Implementation of safety measures

[1090] 4. Regular maintenance schedule

[1091] Template:

[1092] 1. Overview

[1093] 2. Detailed requirements

[1094] 3. Implementation Plan

[1095] 4. Maintenance Plan

[1096] Generated specification:

[1097] Based on this prompt, the server uses a generative AI model to automatically generate specifications, enabling efficient and accurate creation of specifications for machines used in factories.

[1098] The hardware used includes factory robots (e.g., ABB's IRB series) and servers. The software used includes OpenAI GPT-3 models and Python programs. The combination of these technologies allows the invention to achieve its objectives.

[1099] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1100] Step 1:

[1101] The user inputs what he or she has heard by voice.

[1102] Input: The user inputs what they heard by voice.

[1103] How it works: A user uses a microphone to dictate a meeting or interview.

[1104] Output: The audio data is sent to the server.

[1105] Step 2:

[1106] The server uses voice recognition technology to convert the voice data into text.

[1107] Input: Audio data

[1108] How it works: The server uses speech recognition software (e.g., Google Speech-to-Text API) to convert the audio data into text.

[1109] Output: Text data

[1110] Step 3:

[1111] The server uses natural language processing technology to extract requirements from the text data.

[1112] Input: Text data

[1113] How it works: The server uses a natural language processing library (e.g., spaCy) to extract requirements from text data, specifically by extracting keywords and analyzing context.

[1114] Output: Requirement list

[1115] Step 4:

[1116] The server generates a prompt using a generative AI model.

[1117] Input: Requirement List

[1118] How it works: The server feeds a generative AI model (e.g., OpenAI GPT-3) with a list of requirements and generates a prompt.

[1119] Output: prompt statement

[1120] Step 5:

[1121] The server automatically generates RFPs and specifications based on the prompt text.

[1122] Input: prompt statement

[1123] How it works: The server uses a generative AI model to generate RFPs and specifications based on the prompt text, specifically by embedding requirements in templates to create documents.

[1124] Output: RFP and specifications

[1125] Step 6:

[1126] The server automatically generates specifications for the machines used in the factory.

[1127] Input: Factory machinery requirements

[1128] How it works: The server analyzes the requirements for machines in a factory and automatically generates specifications using appropriate templates.

[1129] Output: Machine specifications

[1130] Step 7:

[1131] The server provides the generated RFP and specifications to the user.

[1132] Input: Generated RFP or specification

[1133] What happens: The server sends the generated document to the user, either via email or by making it available for download within the system.

[1134] Output: User-accessible RFP and specifications

[1135] Example 3

[1136] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1137] In the traditional system development process, each phase, from requirements definition to design, coding, and testing, requires a lot of manual work, which is time-consuming and labor-intensive. Also, incorporating customer feedback requires manual corrections, which is inefficient. Furthermore, there is little automation in code generation and test case creation, placing a heavy burden on developers. To solve these problems, automation and efficiency in each phase are required.

[1138] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1139] In this invention, the server includes a means for automatically generating minutes and extracting requirements from interview content, a means for automatically creating requirements specifications and design documents, a means for automatically generating a user interface based on requirements and incorporating customer feedback, and a means for generating code and creating test cases using a generative AI model based on requirements, thereby automating and streamlining each phase of the system development process.

[1140] "Hearing content" refers to information such as requirements, wishes, and opinions collected from customers and stakeholders during system development.

[1141] "Generating minutes" is the process of recording and documenting the contents of a meeting based on the hearing content.

[1142] "Requirements extraction" is the process of identifying and extracting the requirements necessary for system development from interview content and meeting minutes.

[1143] A "requirements specification" is a document that describes in detail the functions, performance, and constraints of a system or software.

[1144] A "design document" is a document that describes in detail the structure and operation of a system or software.

[1145] "User interface" refers to the screens and methods of operation that allow a user to interact with a system or software.

[1146] "Customer feedback" refers to information such as opinions, evaluations, and areas for improvement provided by customers.

[1147] A "generative AI model" is a model that uses artificial intelligence technology to automatically perform specific tasks.

[1148] "Code generation" is the process of automatically creating program code based on requirements and designs.

[1149] "Test case generation" is the process of automatically creating test scenarios to verify whether the generated code works correctly.

[1150] The present invention is a system for automating and streamlining each phase of a system development process. Specific embodiments of this system will be described below.

[1151] Generate meeting minutes and extract requirements from interview content

[1152] The server uses speech recognition technology to convert the contents of the hearing into text. For example, speech recognition software is used to convert what is said during the meeting into text in real time. It then uses natural language processing technology to extract requirements from the text. Specifically, it uses a text analysis algorithm to identify important keywords and phrases and extract them as requirements.

[1153] Automatic creation of requirements specifications and design documents

[1154] The server automatically creates requirements specifications and design documents based on the extracted requirements, for example, by using a template-based document generation tool to document the requirements in an appropriate format, thereby eliminating the need to manually create the documents.

[1155] Automatically generate user interfaces and incorporate customer feedback

[1156] The server automatically generates a user interface (UI) based on the requirements. Specifically, an initial UI design is created using a UI design tool (e.g., Figma). The user then reviews the generated UI design and provides feedback. The server analyzes the user's feedback and automatically applies the necessary changes to update the UI design.

[1157] Examples:

[1158] Prompt: "If the user gives feedback that they would like to change the position of the buttons on the login screen, move the buttons to the center."

[1159] Code generation and test case creation

[1160] The server generates code using a generative AI model (e.g., OpenAI Codex) based on the requirements. The generated code satisfies the system's functional requirements. At the same time, the server generates test cases using a test automation tool (e.g., Selenium). This automatically creates test scenarios to verify that the generated code works correctly.

[1161] Examples:

[1162] Prompt: "Generate the code for the user authentication function and create a test case for user authentication at the same time."

[1163] Reviewing the code and test cases

[1164] The user checks the generated code and test cases and makes any necessary modifications. The terminal then sends the modified code and test cases to the server, completing the final system.

[1165] In this way, this invention can automate and streamline each phase of the system development process. Specific software used includes voice recognition software, natural language processing algorithms, UI design tools (e.g., Figma), generative AI models (e.g., OpenAI Codex), and test automation tools (e.g., Selenium). The flow of specific processing in Example 3 will be described with reference to FIG. 15.

[1166] Step 1:

[1167] The user inputs UI and function requirements into the system. Specifically, the user describes the requirements in detail using forms and text boxes. The terminal then sends the input requirements to the server.

[1168] Input: User-entered requirements (e.g., "Please add username and password input fields to the login screen")

[1169] Output: Requirement data sent to the server

[1170] Step 2:

[1171] The server automatically generates an initial UI design using a UI design tool based on the received requirements. Specifically, it uses a UI design tool (e.g., Figma) to create a UI design based on the requirements. The server then sends the generated UI design to the device.

[1172] Input: Requirement data

[1173] Output: Initial UI design data

[1174] Step 3:

[1175] The user checks the generated UI design and provides feedback, specifically indicating which parts of the UI they would like to change using comments or markup. The device then sends the user's feedback to the server.

[1176] Input: Initial UI design data

[1177] Output: User feedback data (e.g. "Please move the button to the center")

[1178] Step 4:

[1179] The server analyzes the user feedback and automatically applies the necessary changes. Specifically, it analyzes the feedback and updates the design using a UI design tool. The server then sends the updated UI design back to the device.

[1180] Input: User feedback data

[1181] Output: Updated UI design data

[1182] Step 5:

[1183] The server generates code using a generative AI model based on the requirements. Specifically, it uses a generative AI model (e.g., OpenAI Codex) to create program code that meets the requirements. The server then sends the generated code to the device.

[1184] Input: Requirement data

[1185] Output: Generated code data

[1186] Step 6:

[1187] The server generates test cases based on the generated code using a test automation tool. Specifically, it uses a test automation tool (e.g., Selenium) to create test scenarios to verify the code's behavior. The server then sends the generated test cases to the terminal.

[1188] Input: Generated code data

[1189] Output: Generated test case data

[1190] Step 7:

[1191] The user checks the generated code and test cases and makes any necessary modifications. The terminal then sends the modified code and test cases to the server, completing the final system.

[1192] Input: Generated code data, generated test case data

[1193] Output: Modified code data, modified test case data

[1194] (Application example 3)

[1195] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1196] With the previous system, designing and updating the user interface (UI) required a great deal of time and effort, making it difficult to quickly reflect user feedback. Furthermore, adding new features and creating test cases for them had to be done manually, reducing development efficiency. This made it difficult to improve the user experience and quickly add new features, potentially resulting in a loss of competitiveness.

[1197] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1198] In this invention, the server includes means for automatically generating minutes and extracting requirements from interview content, means for automatically creating requirements specifications and design documents, means for automatically creating a UI and incorporating customer feedback, means for automatically coding and creating tests, means for optimizing the user interface in real time, and means for automatically generating code and test cases for new functions. This makes it possible to quickly incorporate user feedback and optimize the UI in real time. Furthermore, the addition of new functions and the creation of their test cases are also automated, significantly improving development efficiency.

[1199] "Hearing content" refers to the content of conversations and interviews conducted to gather requirements and opinions from users and customers.

[1200] "Minutes generation" is the process of automatically creating a record of a meeting based on the contents of the hearing.

[1201] "Requirements extraction" is the process of identifying the functions and specifications required for a system or application from meeting minutes and interview content.

[1202] A "requirements specification" is a document that details the functions and performance that a system or application must meet.

[1203] A "design document" is a technical document that describes in detail the structure and operation of a system or application.

[1204] "UI" stands for user interface and refers to the screens and methods of operation that allow users to interact with systems and applications.

[1205] "Customer feedback" refers to information such as opinions, evaluations, and areas for improvement provided by users and customers.

[1206] "Auto-coding" is the process of using AI technology to automatically generate program code instead of writing it manually.

[1207] "Test creation" is the process of designing and creating test cases to verify the operation of a system or application.

[1208] "User interface optimization" is the process of improving and adjusting the UI to improve user operability and convenience.

[1209] "New function code" is program code for implementing new functions that are added to a system or application.

[1210] A "test case" describes specific test scenarios and conditions for verifying the operation of a system or application.

[1211] A system for carrying out this invention includes the following main components: The server includes means for automatically generating minutes and extracting requirements from interview content, means for automatically creating requirement specifications and design documents, means for automatically creating a UI and incorporating customer feedback, means for realizing automatic coding and test creation, means for optimizing the user interface in real time, and means for automatically generating code and test cases for new functions.

[1212] Hardware and software used

[1213] Hardware: Cloud-based servers, user devices (smartphones, tablets, PCs, etc.)

[1214] Software: Python, AI model (e.g., GPT-3, BERT, etc.), database (e.g., MySQL, MongoDB, etc.)

[1215] Data processing and calculation

[1216] Processing of hearing contents

[1217] The server uses speech recognition technology to convert the interview content into text and natural language processing technology to extract requirements from the text, automatically generating minutes and clarifying the requirements.

[1218] Automatic creation of requirements specifications and design documents

[1219] The server automatically generates requirements specifications and design documents based on the extracted requirements, thereby streamlining the development process.

[1220] Automatic UI generation and feedback reflection

[1221] The server automatically generates an initial UI design based on requirements and collects real-time user feedback to update the UI, resulting in a better user experience.

[1222] Automated coding and test creation

[1223] The server uses AI technology to automatically generate code and test cases for new features, significantly reducing the amount of work required by developers.

[1224] User interface optimization

[1225] The server optimizes the user interface in real time based on user operation data and feedback, thereby improving user convenience.

[1226] Specific examples

[1227] For example, if a user provides feedback saying "I would like a wishlist feature," the server would do the following:

[1228] 1. Collect feedback and feed it into the AI ​​model.

[1229] 2. AI model generates new UI design and adds wishlist feature.

[1230] 3. Automatically generate code and test cases for new features.

[1231] Prompt Sentence Examples

[1232] User feedback: "I want a wishlist feature"

[1233] Requirement: {"feature": "wishlist", "description": "A feature that allows users to add products to a wishlist"}

[1234] By inputting this prompt into the AI ​​model, new UI designs, code, and test cases are automatically generated.

[1235] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1236] Step 1:

[1237] The user provides feedback.

[1238] The user enters feedback, such as "I want a wishlist function," into the feedback form of the application. The input feedback is sent to the server.

[1239] Step 2:

[1240] The server collects the feedback and inputs it into the AI ​​model.

[1241] The server receives the feedback sent by the user and inputs it into the AI ​​model. Specifically, the feedback content is passed to the AI ​​model in text format. The input data is in the form of "User feedback: 'I want a wishlist function'".

[1242] Step 3:

[1243] The server uses the AI ​​model to generate a new UI design.

[1244] The server inputs the feedback into the AI ​​model and generates a new UI design. The AI ​​model analyzes the feedback and adds or modifies appropriate UI elements. The output data is a specification for the new UI design.

[1245] Step 4:

[1246] The server automatically generates code for the new feature.

[1247] The server uses the AI ​​model to generate code for new functions. Specifically, it identifies the required functions based on the feedback and generates program code to realize those functions. The output data is the program code for the new functions.

[1248] Step 5:

[1249] The server automatically generates test cases for new features.

[1250] The server uses the AI ​​model to generate test cases for new functions. Specifically, it creates test scenarios based on the generated program code to verify whether the code works correctly. The output data is a test case specification.

[1251] Step 6:

[1252] The server deploys the generated UI design and code.

[1253] The server deploys the generated UI design and program code into the application, integrating new UI elements and functionality into the existing application and making them available to users. The output data is the updated application.

[1254] Step 7:

[1255] Users will experience the new UI and features.

[1256] Users use the updated application to experience the new UI and added features, such as adding products to a wishlist. The output data is the user's operation log and new feedback.

[1257] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1258] "Example 1"

[1259] One embodiment of the present invention is a system that incorporates an emotion engine. This system recognizes emotions from the user's voice and facial expressions, and automatically creates a UI based on those emotions and incorporates customer feedback. Specifically, if the user makes an expression that shows joy or surprise, the system captures that information and automatically changes the UI color and layout to be brighter and more fun. Also, if the user makes an expression that shows anger or dissatisfaction, the system captures that information, identifies the problem area, and automatically generates feedback to encourage improvement in that area.

[1260] "Example 2"

[1261] Another embodiment of the present invention is automatic coding and test creation using an emotion engine. Specifically, if a user makes an expression that indicates confusion or hesitation, the system captures that information and determines that there is a problem with the code generation or test case creation. The system then identifies the problem area and redoes the code generation or test case creation for that area. This enables more appropriate code generation and test case creation that reflects the user's emotions.

[1262] "Example 3"

[1263] One embodiment of the present invention is a system that incorporates an emotion engine. This system recognizes emotions from the user's voice and facial expressions, and automatically creates a UI based on those emotions and incorporates customer feedback. Specifically, if the user makes an expression that shows joy or surprise, the system captures that information and automatically changes the UI color and layout to be brighter and more fun. Also, if the user makes an expression that shows anger or dissatisfaction, the system captures that information, identifies the problem area, and automatically generates feedback to encourage improvement in that area.

[1264] The processing flow of each embodiment will be described below.

[1265] "Example 1"

[1266] Step 1: Capture the user's voice and facial expressions through the camera and microphone.

[1267] Step 2: The acquired voice and facial expression data is input into the emotion engine to recognize the user's emotions.

[1268] Step 3: Automatically change UI color and layout based on the recognized emotion.

[1269] Step 4: If the user makes a facial expression that indicates anger or dissatisfaction, the system will identify the problem area and automatically generate feedback to encourage improvement in that area.

[1270] "Example 2"

[1271] Step 1: Capture the user's voice and facial expressions through the camera and microphone.

[1272] Step 2: The acquired voice and facial expression data is input into the emotion engine to recognize the user's emotions.

[1273] Step 3: If the user looks confused or unsure, it is determined that there is a problem with the code generation or test case creation.

[1274] Step 4: Identify the problem areas and regenerate code and create test cases for those areas.

[1275] Example 1

[1276] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1277] In traditional meetings and interviews, creating minutes and extracting requirements is often done manually, which takes time and effort. Furthermore, it is difficult to automatically create a user interface that reflects user emotions or automatically generate customer feedback, making improving the user experience a challenge. Furthermore, there is a demand for more efficient automated coding and test creation.

[1278] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1279] In this invention, the server includes means for collecting voice data and converting it into text in real time, means for analyzing the text data and extracting requirements, means for recognizing user emotions and automatically changing the user interface, means for automatically generating customer feedback, and means for realizing automatic coding and test creation. This enables the automatic generation of minutes and efficient extraction of requirements, enabling the automatic creation of a user interface that reflects user emotions and the rapid provision of customer feedback. Furthermore, automatic coding and test creation using artificial intelligence technology can improve the efficiency of the development process.

[1280] "Voice data" refers to data in which a user's speech or voice is recorded in digital format.

[1281] "Text data" refers to data obtained by converting voice data into character information.

[1282] "Speech recognition technology" is a technology that analyzes voice data and converts it into corresponding text data.

[1283] "Natural language processing technology" is a technology that analyzes text data and understands its meaning and structure.

[1284] "Emotion recognition technology" is a technology that analyzes and recognizes emotions from a user's facial expressions and voice.

[1285] "User interface" refers to the screen and operation method that allows a user to interact with a system.

[1286] "Customer feedback" refers to improvements and evaluations based on users' usage experiences and opinions.

[1287] "Auto-coding" is a technique for automatically generating program code.

[1288] "Test creation" is the technique of generating test cases to verify the behavior of a program.

[1289] "Artificial intelligence technology" is a technology that allows machines to imitate human intelligence and perform learning and reasoning.

[1290] This invention is a system that includes means for collecting voice data and converting it into text in real time, means for analyzing the text data and extracting requirements, means for recognizing user emotions and automatically changing the user interface, means for automatically generating customer feedback, and means for realizing automatic coding and test creation.

[1291] Collecting and converting audio data into text

[1292] The device captures the user's voice data in real time through a microphone, which is then sent over the network to a server, which then converts the received voice data into text using voice recognition technology such as the Google Cloud Speech-to-Text API.

[1293] Example: When a user says "Let's schedule our next meeting for next Tuesday" during a meeting, the device captures the audio and sends it to the server, which uses a speech recognition API to convert the audio into text: "Let's schedule our next meeting for next Tuesday."

[1294] Text data analysis and requirement extraction

[1295] The server analyzes the text data using natural language processing technologies such as Python's NLTK library and SpaCy, extracts important keywords and phrases from the analysis results, and records them as requirements.

[1296] Example: The server analyzes the text data "Please set the next meeting for next Tuesday" and extracts the keywords "next meeting," "next Tuesday," and "set."

[1297] Emotion Recognition and Automatic User Interface Creation

[1298] The device captures the user's facial expressions and voice in real time through a camera and microphone. This data is then sent to a server via the network. The server then analyzes the received emotional data using emotion recognition technologies such as Microsoft Azure's Emotion API and OpenCV. Based on the analysis results, the color and layout of the user interface are automatically changed.

[1299] Example: If a user makes a surprised expression, the device captures that expression and sends it to the server, which uses the Emotion API to recognize that the user is surprised and changes the UI background color to bright yellow.

[1300] Automated customer feedback generation

[1301] The server analyzes the user's facial expressions, which indicate anger or dissatisfaction, to identify the problem area, and automatically generates and provides feedback to the user to encourage improvement of the identified problem.

[1302] Example: If a user makes a facial expression that indicates dissatisfaction, the server analyzes that information and generates feedback such as, "The user has expressed dissatisfaction. Please improve the following areas."

[1303] Automated coding and test creation

[1304] The server uses artificial intelligence technology to generate code and create test cases, streamlining the development process.

[1305] Example prompt: "Transcribe audio data from a meeting in real time and extract key requirements. Also, describe a system that recognizes user emotions and automatically changes the UI color and layout."

[1306] This system enables the automatic generation of meeting minutes and efficient extraction of requirements, automatically creating user interfaces that reflect user emotions, and providing rapid customer feedback. Furthermore, the system also streamlines the development process by using artificial intelligence technology for automatic coding and test creation.

[1307] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1308] Step 1:

[1309] A user speaks during a meeting or hearing. The device captures the user's voice data in real time through a microphone. The input is the user's voice, and the output is digital voice data. A specific example of this is when the user says, "Please schedule the next meeting for next Tuesday."

[1310] Step 2:

[1311] The terminal transmits the captured audio data to the server via the network. The input is digital audio data, and the output is the audio data transmitted to the server. Specifically, the terminal divides the audio data into packets and transmits them to the server.

[1312] Step 3:

[1313] The server uses the Google Cloud Speech-to-Text API to convert the received voice data into text data. The input is the voice data sent to the server, and the output is text data. Specifically, the server calls the speech recognition API and generates the text "Please schedule the next meeting for next Tuesday."

[1314] Step 4:

[1315] The server uses Python's NLTK library and SpaCy to analyze text data. The input is text data, and the output is the analysis results. Specifically, the server tokenizes the text data and extracts important keywords and phrases.

[1316] Step 5:

[1317] The server extracts important keywords and phrases from the analysis results and records them as requirements. The input is the analysis results, and the output is the keywords and phrases recorded as requirements. Specifically, the server extracts the keywords "next meeting," "next Tuesday," and "settings" and records them in the database.

[1318] Step 6:

[1319] The device captures the user's facial expressions and voice in real time through a camera and microphone. The input is the user's facial expressions and voice, and the output is digital emotional data. Specifically, if the user makes a surprised expression, the device will capture that expression.

[1320] Step 7:

[1321] The device transmits the captured emotion data to the server via the network. The input is the emotion data in digital format, and the output is the emotion data transmitted to the server. Specifically, the device divides the emotion data into packets and transmits them to the server.

[1322] Step 8:

[1323] The server analyzes the received emotion data using Microsoft Azure's Emotion API and OpenCV. The input is the emotion data sent to the server, and the output is the analysis result. Specifically, the server calls the Emotion API to recognize that the user is surprised.

[1324] Step 9:

[1325] The server automatically changes the color and layout of the user interface based on the analysis results. The input is the analysis results, and the output is the modified user interface. Specifically, the server changes the background color of the UI to bright yellow.

[1326] Step 10:

[1327] The server analyzes the user's facial expressions, which indicate anger or dissatisfaction, and identifies the problem area. The input is the analysis result, and the output is the identified problem. Specifically, the server generates feedback such as, "The user has expressed dissatisfaction. Please improve the following areas."

[1328] Step 11:

[1329] The server automatically generates feedback to encourage improvements to the identified problems and provides it to the user. The input is the identified problems, and the output is the generated feedback. As a specific operation, the server generates a feedback message and notifies the user.

[1330] Step 12:

[1331] The server uses artificial intelligence technology to generate code and create test cases. The input is requirements and specifications, and the output is the generated code and test cases. Specifically, the server uses a generative AI model to automatically generate program code and test cases.

[1332] (Application example 1)

[1333] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1334] In traditional customer service in brick-and-mortar stores, it can be difficult for store staff to accurately understand customer requests and questions and respond appropriately. It is also difficult to properly recognize customer emotions and respond accordingly. This can lead to lower customer satisfaction and have a negative impact on sales. Furthermore, recording customer conversations and extracting customer requirements is time-consuming and inefficient.

[1335] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1336] In this invention, the server includes means for automatically generating minutes and extracting requirements from the contents of the interview, means for automatically creating specifications, means for automatically creating a user interface and reflecting customer feedback, means for realizing automatic coding and test creation, means for converting the contents of the conversation into text in real time using speech recognition technology and extracting requirements using natural language processing technology, and means for recognizing user emotions using emotion recognition technology and automatically adjusting the user interface. This makes it possible to efficiently record the contents of conversations with customers, extract requirements, and furthermore, to respond appropriately according to the customer's emotions, which is expected to improve customer satisfaction and increase sales.

[1337] "Hearing content" refers to information and opinions obtained through dialogue and interviews with customers and users.

[1338] "Minutes generation" refers to the process of recording the contents of meetings and conversations and compiling them into a document.

[1339] "Requirements extraction" is the process of extracting necessary information and conditions from dialogue or documents.

[1340] A "specification" is a document that details the functions, performance, design, etc. of a system or product.

[1341] A "user interface" refers to the screens and methods of operation that allow a user to interact with a system or application.

[1342] "Customer feedback" refers to opinions and ratings provided by customers.

[1343] "Auto-coding" refers to the process of automatically generating program code.

[1344] "Test creation" refers to the task of creating test cases to verify the operation of a system or application.

[1345] "Speech recognition technology" refers to technology that converts speech into text.

[1346] "Natural language processing technology" refers to the technology of processing human language using a computer.

[1347] "Emotion recognition technology" is a technology that determines a user's emotions from their facial expressions and tone of voice.

[1348] "Real-time" means processing or reacting immediately, without delay.

[1349] "Text conversion" is the process of converting information such as audio and images into text data.

[1350] "Keywords" are important words or phrases in a document or conversation.

[1351] A "phrase" is a sequence of words that has meaning in a sentence or dialogue.

[1352] "Color" refers to the color of the user interface.

[1353] "Layout" refers to the arrangement and structure of a user interface.

[1354] A system for carrying out this invention has the following configuration: The server includes means for automatically generating minutes and extracting requirements from the contents of interviews, means for automatically creating specifications, means for automatically creating a user interface and reflecting customer feedback, means for realizing automatic coding and test creation, means for converting the contents of conversations into text in real time using speech recognition technology and extracting requirements using natural language processing technology, and means for recognizing user emotions using emotion recognition technology and automatically adjusting the user interface.

[1355] Hardware and software used

[1356] Hardware:

[1357] Smart glasses (with built-in camera and microphone)

[1358] Computer (processing speech recognition and emotion recognition)

[1359] software:

[1360] speech_recognition library: Performs speech recognition

[1361] transformers library: for natural language processing

[1362] emotion_recognition library: Performs emotion recognition

[1363] cv2(OpenCV): Acquiring and processing camera images

[1364] System Operation

[1365] The server first captures the conversation between the customer and the store clerk through the microphone in the smart glasses. It then uses voice recognition technology to convert this voice data into text in real time. It then uses natural language processing technology to extract requirements from the text. This allows the server to clearly understand the customer's requests and questions.

[1366] Furthermore, the smart glasses' camera captures the customer's facial expressions and uses emotion recognition technology to determine their emotions. For example, if the customer is happy, the colors and layout of the user interface will automatically change to a bright and cheerful one. On the other hand, if the customer is angry, information to help resolve the problem will be highlighted.

[1367] Specific examples

[1368] For example, consider a scenario in which a customer asks a question about a new product and a sales associate responds. The speech recognition result is "Please tell me the features of this new smartphone." The requirement extraction result is "Features of the new smartphone." If the customer shows interest and has a happy expression, the system changes the user interface to a bright and fun theme.

[1369] Prompt Sentence Examples

[1370] A customer is asking about a new smartphone. The speech recognition result is "Please tell me the features of this new smartphone." The requirement extraction result is "Features of the new smartphone." The customer is interested and has a happy expression. Change the UI to a bright and fun theme.

[1371] In this way, the server can efficiently record the content of conversations with customers, extract requirements, and respond appropriately according to the customer's emotions, which is expected to improve customer satisfaction and increase sales.

[1372] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1373] Step 1:

[1374] The server captures the conversation between the customer and the store clerk through the microphone in the smart glasses. The input is voice data, and the output is text data converted using speech recognition technology. Specifically, the server uses the speech_recognition library to convert voice into text in real time.

[1375] Step 2:

[1376] The server analyzes the textual content of the conversation using natural language processing technology and extracts requirements. The input is text data, and the output is the extracted requirements. Specifically, it uses the transformers library to extract keywords and phrases from the text and list them as requirements.

[1377] Step 3:

[1378] The server acquires the customer's facial expression through the smart glasses' camera and uses emotion recognition technology to determine the customer's emotion. The input is image data, and the output is the recognized emotion. Specifically, the emotion_recognition library and cv2 (OpenCV) are used to analyze facial expressions from images and determine the emotion.

[1379] Step 4:

[1380] The server automatically adjusts the user interface based on the recognized emotion. The input is the recognized emotion, and the output is the adjusted user interface. Specifically, if the customer is happy, the color and layout of the user interface are changed to a bright and cheerful one, and if the customer is angry, information for resolving the problem is highlighted.

[1381] Step 5:

[1382] The server records the content of conversations with customers and the extracted requirements, and automatically creates specifications as needed. The input is text data and extracted requirements, and the output is specifications. Specifically, the information is automatically filled into a specification template based on the text data and requirements.

[1383] Step 6:

[1384] The server reflects customer feedback and continuously improves the user interface and service content. The input is customer feedback, and the output is an improved user interface and service content. Specifically, it analyzes the feedback data, identifies areas for improvement, and reflects them in the system.

[1385] Example 2

[1386] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1387] In conventional systems, requirements extraction from interviews and the creation of RFPs and specifications were often done manually, which was time-consuming and labor-intensive. Furthermore, automated coding and test creation did not take user emotions into consideration, resulting in results that did not meet user intent. Furthermore, automated UI creation and insufficient reflection of customer feedback made it difficult to improve the user experience.

[1388] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1389] In this invention, the server includes a means for automatically generating minutes and extracting requirements from the interview content, a means for automatically creating RFPs and specifications using templates based on the requirements, and a means for analyzing the user's facial expressions and automatically coding and creating tests based on their emotions. This automates the entire process from requirements extraction to document creation, code generation, and test creation, making it possible to provide more appropriate results that take the user's emotions into consideration.

[1390] "Hearing contents" refers to information including the requirements and wishes of the user regarding the project or system provided.

[1391] "Generating minutes" is the process of recording and documenting the contents of a meeting based on the hearing content.

[1392] "Requirements extraction" is the process of identifying and extracting the functions and conditions necessary for a project or system from the content of interviews.

[1393] A "template" is a document template with a predefined format and structure that is used when creating documents such as RFPs and specifications.

[1394] "RFP" stands for Request for Proposal, a request for proposal to request the provision of a project or service.

[1395] A "specification" is a document that lists the detailed requirements and specifications for a system or project.

[1396] "Facial expression analysis" is a technology that captures a user's facial expressions using a camera or other device and analyzes their emotions.

[1397] "Autocoding" is the process of automatically generating program code.

[1398] "Test creation" is the process of creating test cases to verify that the generated code works correctly.

[1399] "UI" stands for user interface and refers to the screen and operation methods that allow users to interact with the system.

[1400] "Customer feedback" refers to opinions and evaluations of systems and projects provided by users.

[1401] "AI technology" is a technology that uses artificial intelligence and is used to perform data analysis, automation, etc.

[1402] "Natural language processing technology" is a technology that allows computers to understand and analyze human language.

[1403] "Speech recognition technology" is a technology that converts speech into text.

[1404] This invention is a system that automatically generates minutes and extracts requirements from interview content, automatically creates RFPs and specifications using templates based on the requirements, and further analyzes the user's facial expressions to automatically code and create tests based on their emotions. A specific embodiment of this system is shown below.

[1405] Hardware and software used

[1406] Server: Performs data analysis, template embedding, code generation, and test case creation.

[1407] Terminal: Receives user input, captures facial expressions, and sends them to the server.

[1408] Camera: Used to capture the user's facial expressions.

[1409] Natural language processing technology (NLP): Used to analyze interview content and extract requirements.

[1410] Speech recognition technology: Used to convert the hearing content into text.

[1411] AI technology: Used for code generation and test case creation.

[1412] Templates: Used to automatically create RFPs and specifications.

[1413] Program processing

[1414] Receiving and analyzing the contents of the hearing

[1415] 1. The user enters the project requirements into the system.

[1416] 2. The terminal sends the input requirements to the server.

[1417] 3. The server converts the interview content into text using voice recognition technology and extracts requirements using natural language processing technology.

[1418] Automatic creation of RFPs and specifications

[1419] 1. The server automatically fills in each item in a pre-prepared template based on the extracted requirements.

[1420] 2. The server generates the final RFP and specifications based on the embedded template.

[1421] 3. The server sends the generated RFP and specifications to the terminal.

[1422] 4. The terminal displays the generated document to the user.

[1423] Automatic coding and test creation using an emotion engine

[1424] 1. The device captures the user's facial expression with a camera.

[1425] 2. The device analyzes the captured facial expression data to determine whether the user is showing confusion or hesitation.

[1426] 3. The device sends the analysis results to the server.

[1427] 4. The server receives the user's emotional data and determines that there is a problem with code generation or test case creation.

[1428] 5. The server identifies the problem area and regenerates code and creates test cases for that area.

[1429] 6. The server sends the modified code and test cases to the terminal.

[1430] 7. The terminal displays the modified code and test cases to the user.

[1431] Specific examples

[1432] For example, if a user enters "requirements for developing a new website," the server analyzes the requirements and automatically fills in the template's "project overview," "functional requirements," "technical requirements," etc. Finally, the completed RFP and specifications are provided to the user.

[1433] If the user shows a confused expression while generating code, the device captures that expression and sends it to the server. The server then uses this information to identify the problem area and regenerates the code or creates test cases. Finally, the corrected code and test cases are provided to the user.

[1434] Prompt Sentence Examples

[1435] "Automatically generate an RFP based on our new website development requirements."

[1436] "If the user looks confused, assume there is a problem with the code generation or test case creation and try again."

[1437] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1438] Step 1:

[1439] Users input project requirements into the system.

[1440] Input: Project requirements (e.g. new website development requirements)

[1441] Specific operation: The user enters the project requirements in text format into the input form on the terminal.

[1442] Step 2:

[1443] The terminal transmits the input requirements to the server.

[1444] Input: Project requirements entered by the user

[1445] Output: Project requirements data sent to the server

[1446] Specific operation: The terminal transmits the requirements input by the user to the server as a data packet.

[1447] Step 3:

[1448] The server converts the interview content into text using voice recognition technology and extracts requirements using natural language processing technology.

[1449] Input: Project requirements data

[1450] Output: Extracted requirements information

[1451] Specific operation: The server converts the received requirements data into text using speech recognition technology and extracts important requirements using natural language processing technology.

[1452] Step 4:

[1453] Based on the extracted requirements, the server automatically fills in each item in a pre-prepared template.

[1454] Input: Extracted requirements information, template

[1455] Output: Embedded template

[1456] Specific operation: The server automatically fills each item of the template with the extracted requirement information to complete the template.

[1457] Step 5:

[1458] The server generates the final RFP and specifications based on the embedded templates.

[1459] Input: Embedded template

[1460] Output: Generated RFP and specifications

[1461] Specific operation: The server automatically generates RFPs and specifications based on the embedded templates.

[1462] Step 6:

[1463] The server sends the generated RFP and specifications to the terminal.

[1464] Input: Generated RFP or specifications

[1465] Output: RFP and specifications sent to the terminal

[1466] Specific operation: The server sends the generated RFP and specifications to the terminal as a data packet.

[1467] Step 7:

[1468] The terminal displays the generated document to the user.

[1469] Input: RFP or specification sent from the server

[1470] Output: RFP or specification document displayed to the user

[1471] Specific operation: The terminal displays the received RFP and specifications to the user.

[1472] Step 8:

[1473] The device captures the user's facial expressions with a camera.

[1474] Input: User's facial expression

[1475] Output: Captured facial expression data

[1476] Specific operation: The device uses a camera to capture the user's facial expressions in real time.

[1477] Step 9:

[1478] The device analyzes the captured facial expression data to determine whether the user is showing confusion or hesitation.

[1479] Input: Captured facial expression data

[1480] Output: Analysis results (user's emotional state)

[1481] Specific operation: The device analyzes the captured facial expression data and determines the user's emotional state.

[1482] Step 10:

[1483] The terminal transmits the analysis results to the server.

[1484] Input: Analysis results (user's emotional state)

[1485] Output: Analysis results sent to the server

[1486] Specific operation: The terminal transmits the analysis result to the server as a data packet.

[1487] Step 11:

[1488] The server receives the user's emotional data and determines whether there is a problem with code generation or test case creation.

[1489] Input: Analysis results (user's emotional state)

[1490] Output: Identification of problem areas

[1491] Specific operation: Based on the received emotion data, the server determines whether there are any problems with code generation or test case creation.

[1492] Step 12:

[1493] The server identifies the problem areas and regenerates code and creates test cases for those areas.

[1494] Input: Problem location identification results

[1495] Output: Modified code and test cases

[1496] Specific operation: The server regenerates code and creates test cases for the identified problem areas.

[1497] Step 13:

[1498] The server sends the modified code and test cases to the terminal.

[1499] Input: Modified code and test cases

[1500] Output: Modified code and test cases sent to the terminal

[1501] Specific operation: The server sends the modified code and test cases to the terminal as data packets.

[1502] Step 14:

[1503] The terminal displays the modified code and test cases to the user.

[1504] Input: Modified code and test cases sent from the server

[1505] Output: The modified code and test cases that are displayed to the user

[1506] Specific operation: The terminal displays the received modified code and test cases to the user.

[1507] (Application example 2)

[1508] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1509] In conventional manufacturing process management systems, requirements extraction and RFP creation are done manually, which takes time and effort. Furthermore, if the user expresses confusion or hesitation, the system may not be able to capture that information, resulting in inappropriate code generation and test case creation. This reduces the efficiency of the manufacturing process and increases the likelihood of errors.

[1510] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1511] In this invention, the server includes means for automatically generating minutes and extracting requirements from the contents of interviews, means for automatically creating RFPs and specifications, means for automatically creating UIs and reflecting customer feedback, means for realizing automatic coding and test creation, means for detecting the user's facial expressions and correcting code generation and test case creation based on the user's emotions, and means for managing the manufacturing process based on the requirements. This enables efficient management of the manufacturing process and appropriate code generation and test case creation that reflects the user's emotions.

[1512] "Hearing content" refers to the content of conversations and interviews conducted to gather requirements and opinions from users and customers.

[1513] "Minutes generation" is the process of recording and documenting the contents of meetings and hearings.

[1514] "Requirements extraction" is the process of identifying and extracting the requirements necessary for a project or system from interview content and meeting minutes.

[1515] "RFP" stands for Request for Proposal. It is a document that lists the requirements and conditions of a project and requests proposals.

[1516] A "specification" is a document that details the requirements and design of a system or project.

[1517] "UI" stands for User Interface, and is the interface through which a user interacts with a system or application.

[1518] "Customer feedback" refers to opinions and evaluations provided by customers, and is information used to improve products and services.

[1519] "Autocoding" is the process of automatically generating program code.

[1520] "Test case creation" is the process of creating test scenarios to verify the operation of software.

[1521] "Emotion detection" is a technology that analyzes and identifies emotions from a user's facial expressions and voice.

[1522] "Manufacturing process management" is a management method for planning, executing, and monitoring production activities in the manufacturing industry.

[1523] A "template" is a model that defines the standard format and structure of a document or data.

[1524] A "camera" is a device for taking images or videos.

[1525] An "emotion detection engine" is software or hardware that analyzes and identifies emotions from image and audio data.

[1526] "AI technology" refers to artificial intelligence technology, which uses machine learning and data analysis to mimic human intelligence.

[1527] "Natural language processing technology" is a technology that allows computers to understand and process human language.

[1528] A system for implementing the present invention includes the following major components:

[1529] 1. Automatic generation of meeting minutes and requirement extraction from interview content

[1530] The server converts the interview content into text using speech recognition technology and extracts requirements from the text using natural language processing technology. This automatically generates minutes of the interview conducted by the user and extracts the necessary requirements.

[1531] 2. Automatic creation of RFPs and specifications

[1532] The server automatically generates RFPs and specifications using templates based on the extracted requirements, allowing the necessary documents to be created quickly in the early stages of a project.

[1533] 3. Automatic UI creation and customer feedback

[1534] The server automatically creates a user interface (UI) based on the generated requirements and incorporates feedback from the customer, resulting in a highly usable UI being delivered quickly.

[1535] 4. Automated coding and test creation

[1536] The server uses AI technology to generate code and create test cases, which streamlines the development process and produces code with fewer errors.

[1537] 5. User Facial Expression Detection and Emotion-Based Code Modification

[1538] The server uses a camera and an emotion detection engine to analyze the user's facial expressions in real time, and if it detects an expression of confusion or hesitation, it identifies the problem area and regenerates it, making it possible to generate appropriate code and test cases that reflect the user's emotions.

[1539] 6. Manufacturing process management

[1540] The server manages the manufacturing process based on the requirements. It extracts the requirements, automatically generates RFPs and specifications using templates, and manages the progress of the manufacturing process. This enables efficient management of the manufacturing process.

[1541] Hardware and software used

[1542] Hardware:

[1543] Camera: Used to capture the user's facial expressions.

[1544] Computer: Used to run programs.

[1545] software:

[1546] OpenCV: Used to capture and display camera footage.

[1547] EmotionEngine: Used for emotion detection.

[1548] RFPGenerator: Used to automatically generate RFPs.

[1549] CodeGenerator: Used for code generation and problem identification.

[1550] Natural language processing technology: Used to convert interview content into text and extract requirements.

[1551] AI technology: Used for code generation and test case creation.

[1552] Specific examples

[1553] For example, if an engineer shows signs of confusion when designing a new manufacturing line in a factory, the system will identify the problem and regenerate the code. It will also automatically generate an RFP based on the requirements, helping to keep the project moving smoothly.

[1554] Prompt Sentence Examples

[1555] Please generate an RFP based on the following requirements for the design of a new manufacturing line:

[1556] Project name: New production line

[1557] Deadline:2023-12-31

[1558] Budget: 1,000,000

[1559] In this way, the system can efficiently manage the manufacturing process and reduce confusion for engineers.

[1560] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1561] Step 1:

[1562] The server converts the hearing content into text using speech recognition technology. As input, it receives the voice data of the hearing conducted by the user and converts it into text data using speech recognition technology. As output, it obtains the text of the hearing content.

[1563] Step 2:

[1564] The server extracts requirements from the textualized interview content using natural language processing technology. As input, it receives the text data obtained in step 1 and uses natural language processing technology to identify and extract requirements. As output, it obtains the extracted requirements.

[1565] Step 3:

[1566] The server automatically generates RFPs and specifications using templates based on the extracted requirements. As input, it receives the requirements data obtained in step 2 and embeds each item in the template. As output, it obtains the completed RFP and specifications.

[1567] Step 4:

[1568] The server automatically creates a user interface (UI) based on the generated requirements and incorporates customer feedback. It takes requirements data and customer feedback as input, generates the UI, and incorporates the feedback. The output is an updated UI.

[1569] Step 5:

[1570] The server uses AI technology to generate code and create test cases. It receives requirements data as input and generates code and test cases using AI technology. The generated code and test cases are obtained as output.

[1571] Step 6:

[1572] The server uses a camera and an emotion detection engine to analyze the user's facial expressions in real time, and if it detects an expression that indicates confusion or hesitation, it identifies the problem area and regenerates the code. As input, it receives the camera footage and the generated code, and analyzes the facial expression with the emotion detection engine. The output is the regenerated code.

[1573] Step 7:

[1574] The server manages the manufacturing process based on the requirements. As input, it receives the requirements data and generated RFPs and specifications, and manages the progress of the manufacturing process. As output, it obtains an efficiently managed manufacturing process.

[1575] Example 3

[1576] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1577] Conventional systems have the problem that it takes a lot of time and effort to design the user interface, incorporate customer feedback, generate code, and create test cases. Furthermore, it is difficult to improve the user experience because they are unable to recognize user emotions in real time and dynamically change the user interface based on that. To solve these issues, a system that automates and efficiently performs these processes is needed.

[1578] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1579] In this invention, the server includes means for automatically generating minutes and extracting requirements from the contents of the interview, means for automatically creating a requirements specification, means for automatically creating a user interface and reflecting customer feedback, means for realizing automatic coding and test case creation, and means for recognizing user emotions and changing the user interface based on those emotions. This not only enables efficient design of the user interface, reflection of customer feedback, code generation, and test case creation, but also enables dynamic changes to the user interface in response to user emotions.

[1580] "Hearing content" refers to information such as requirements, opinions, and feedback collected from users and customers.

[1581] "Generating minutes" is the process of recording and documenting the contents of meetings and discussions based on the interview content.

[1582] "Requirements elicitation" is the process of identifying and extracting the requirements necessary for a system or project from meeting minutes and interview content.

[1583] A "requirements specification" is a document that details the requirements of a system or project and serves as a guideline for development.

[1584] "User interface" refers to the screen and operation method that allows a user to interact with a system.

[1585] "Customer feedback" refers to information such as opinions, evaluations, and areas for improvement provided by users and customers.

[1586] "Auto-coding" is the process by which a system automatically generates program code.

[1587] A "test case" is a specific test scenario for verifying whether the generated code operates correctly.

[1588] "Emotion recognition" is a technology that analyzes and recognizes emotions from a user's voice and facial expressions.

[1589] "Dynamic user interface modification" is the process of modifying a user interface in real time in response to user sentiment and feedback.

[1590] This invention is a system that combines automatic creation of a user interface (UI), reflection of customer feedback, automatic coding and test case creation, and emotion recognition. Specific embodiments of this system are described below.

[1591] Hardware and software used

[1592] 1. Server

[1593] UI design generation engine: Automatically generate initial UI designs based on requirements.

[1594] Code generation AI model: Automatically generates program code based on requirements.

[1595] Test case generation engine: Automatically creates test cases based on the generated code.

[1596] Emotion recognition engine: Recognizes emotions from the user's voice and facial expressions.

[1597] 2. Terminal

[1598] Camera: Captures the user's facial expressions.

[1599] Microphone: Captures the user's voice.

[1600] Keyboard and mouse: Devices that allow the user to input feedback.

[1601] 3. Users

[1602] Provide feedback: Enter your thoughts and suggestions for improvements to the UI.

[1603] Emotional expression: Expressing emotions through facial expressions and vocalizations.

[1604] Program processing

[1605] 1. Automatic UI generation

[1606] The server automatically generates an initial UI design based on the requirements. For example, it sends instructions such as "Place the button in the top right corner and set the background color to white" to the UI design generation engine.

[1607] 2. Collect customer feedback

[1608] The user reviews the generated UI and provides feedback, for example, by typing "I want the button color to be changed to blue."

[1609] The terminal transmits the feedback input by the user to the server.

[1610] 3. UI refinement

[1611] The server receives feedback from the user and refines the UI. For example, it sends specific instructions, such as "change the button color to blue," to the UI design generation engine.

[1612] 4. Automated coding

[1613] The server generates code using AI technology based on the requirements. For example, it sends instructions such as "generate code for the user registration function" to the code-generation AI model.

[1614] 5. Automatic test case creation

[1615] The server automatically creates test cases based on the generated code. For example, it sends an instruction such as "Create a test case for the user registration function" to the test case generation engine.

[1616] 6. Emotion recognition

[1617] The device captures the user's voice and facial expressions through a camera and microphone. For example, if the user makes a surprised expression, the device sends that information to the server.

[1618] The server analyzes the captured data and recognizes the user's emotions.

[1619] 7. Emotion-Based UI Changes

[1620] The server automatically changes the UI based on the recognized emotion: for example, if the user makes a facial expression indicating joy, it changes the UI color and layout to be bright and cheerful.

[1621] Specific examples

[1622] Example prompt: "Generate code for the user registration function and create a test case."

[1623] The server receives the prompt and sends an instruction to the code generation AI model to "generate code for the user registration function."

[1624] The server analyzes the generated code and sends an instruction to the test case generation engine to "create a test case for the user registration function."

[1625] In this way, the system realizes a combination of functions that automatically create a UI, incorporate customer feedback, auto-code, create test cases, and recognize emotions through each processing step. This not only enables efficient user interface design, incorporate customer feedback, generate code, and create test cases, but also enables dynamic changes to the user interface in response to user emotions. The flow of the specific processing in Example 3 is explained using Figure 21.

[1626] Step 1:

[1627] Auto-generated UI

[1628] Input: Requirements (e.g., button placement, background color, number of text fields, etc.)

[1629] The server automatically generates an initial UI design based on the requirements using software called a "UI design generation engine."

[1630] Data processing: Analyzes requirements and sends instructions to the UI design generation engine.

[1631] Output: Initial UI design

[1632] Specific operation: The server sends instructions such as "Place the button in the top right corner and set the background color to white" to the UI design generation engine.

[1633] Step 2:

[1634] Collecting customer feedback

[1635] Input: Initial UI design

[1636] The user reviews the generated UI and provides feedback.

[1637] The terminal sends the feedback input by the user to the server, and the hardware used is a keyboard and a mouse.

[1638] Data processing: Feedback is sent to the server as text data.

[1639] Output: User feedback

[1640] Specific operation: The user inputs "I want the button color to be changed to blue," and the device sends that information to the server.

[1641] Step 3:

[1642] UI refinement

[1643] Input: User feedback

[1644] The server receives feedback from users and refines the UI using software called a "UI design generation engine."

[1645] Data processing: Analyze the feedback and send instructions to the UI design generation engine.

[1646] Output: Refined UI design

[1647] Specific operation: The server sends specific instructions, such as "change the button color to blue," to the UI design generation engine.

[1648] Step 4:

[1649] Auto-coding

[1650] Input: Requirements (e.g. user registration function)

[1651] The server uses AI technology to generate code based on the requirements. The software used is a "code generation AI model."

[1652] Data processing: Analyzing requirements and sending instructions to code-generating AI models.

[1653] Output: Generated code

[1654] Specific operation: The server sends instructions such as "Generate code for the user registration function" to the code generation AI model.

[1655] Step 5:

[1656] Automatic test case creation

[1657] Input: Generated code

[1658] The server automatically creates test cases based on the generated code using software called a "test case generation engine."

[1659] Data processing: Analyzes the generated code and sends instructions to the test case generation engine.

[1660] Output: Generated test cases

[1661] Specific operation: The server sends an instruction to the test case generation engine, such as "Create a test case for the user registration function."

[1662] Step 6:

[1663] emotion recognition

[1664] Input: User's voice and facial expressions

[1665] The device captures the user's voice and facial expressions through a camera and microphone. The hardware used is a "camera" and a "microphone."

[1666] The server analyzes the captured data and recognizes the user's emotions using software called an "emotion recognition engine."

[1667] Data processing: Analyzes the captured data and sends instructions to the emotion recognition engine.

[1668] Output: Recognized emotion data

[1669] Specific operation: The device performs processes such as "capturing the user's facial expressions and sending them to the server."

[1670] Step 7:

[1671] Emotion-based UI changes

[1672] Input: Recognized emotion data

[1673] The server automatically changes the UI based on the recognized emotion using software called a "UI design generation engine."

[1674] Data processing: Based on the information obtained from the emotion recognition engine, instructions are sent to the UI design generation engine.

[1675] Output: Modified UI design

[1676] Specific operation: The server performs processing such as "because the user has made an expression showing joy, change the background color of the UI to yellow."

[1677] (Application example 3)

[1678] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1679] Conventional systems do not automatically generate or update UIs that take user emotions into account, and as a result, the user experience is not sufficiently improved. Furthermore, there is a lack of means to effectively collect and analyze customer feedback and reflect it in UI improvements, making it difficult to increase user satisfaction.

[1680] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1681] In this invention, the server includes means for automatically generating minutes and extracting requirements from the contents of interviews, means for automatically creating requirements specifications and design documents, means for automatically creating a UI and incorporating customer feedback, means for automatically implementing coding and test creation, means for recognizing user emotions and automatically generating and updating a UI based on those emotions, and means for collecting customer feedback, analyzing the data, and identifying areas for improvement in the UI. This makes it possible to provide an optimal UI in real time that corresponds to user emotions and effectively incorporate customer feedback.

[1682] "Hearing content" refers to information such as requirements, opinions, and feedback collected from users and customers.

[1683] "Minutes generation" is the process of recording and documenting the contents of meetings and discussions.

[1684] "Requirements elicitation" is the process of identifying the requirements for a system or project from collected information.

[1685] A "requirements specification" is a document that details the requirements for a system or project.

[1686] A "design document" is a document that provides detailed information about the design of a system or project.

[1687] "UI" stands for user interface and refers to the screens and operating methods that allow users to interact with systems and applications.

[1688] "Customer feedback" refers to information such as opinions, evaluations, and areas for improvement provided by customers.

[1689] "Autocoding" is the process of automatically generating program code.

[1690] "Test creation" is the process of creating test cases to verify the operation of a system or application.

[1691] "User emotion" refers to an emotional state such as joy, surprise, anger, or dissatisfaction exhibited by a user.

[1692] "Emotion recognition" is a technology that identifies a user's emotions from their voice and facial expressions.

[1693] "Data analysis" is the process of analyzing collected data and extracting useful information and patterns.

[1694] A "generative AI model" is a model that uses artificial intelligence technology to generate new designs and content.

[1695] A "prompt sentence" is an input sentence used to give instructions to a generative AI model.

[1696] The following system configuration will be described as an embodiment of the present invention.

[1697] System Configuration

[1698] The system consists of the following major components:

[1699] 1. A method to automatically generate meeting minutes and extract requirements from interview content

[1700] 2. Means of automatically creating requirements specifications and design documents

[1701] 3. A way to automate UI creation and incorporate customer feedback

[1702] 4. Means of automated coding and test creation

[1703] 5. A method to recognize user emotions and automatically generate and update UI based on those emotions

[1704] 6. A means to collect customer feedback, analyze data, and identify areas for UI improvement

[1705] Program processing explanation

[1706] A means to automatically generate minutes and extract requirements from interview content

[1707] The server converts the interview content into text using speech recognition technology and extracts requirements from the text using natural language processing technology. Specifically, it collects voice data and converts it into text using speech recognition software (e.g., Google Speech-to-Text API). It then extracts requirements using natural language processing technology (e.g., NLTK or SpaCy).

[1708] A means of automatically creating requirements specifications and design documents

[1709] The server automatically generates requirements specifications and design documents based on the extracted requirements. It uses a generative AI model (e.g., OpenAI's GPT-4) to generate detailed documents based on a prompt.

[1710] A way to automate UI creation and incorporate customer feedback

[1711] The server automatically generates a UI using a generative AI model, collects customer feedback, and updates the UI using a feedback collection tool (e.g., Google Forms API) and analyzes it with a data analysis tool (e.g., Python's Pandas).

[1712] A means to automate coding and test creation

[1713] The server uses artificial intelligence technology to generate code and create test cases. It uses a generative AI model to generate code and test cases based on prompts.

[1714] A method to recognize user emotions and automatically generate and update UI based on those emotions

[1715] The device uses the smartphone's camera and microphone to capture the user's facial expressions and voice, and recognizes their emotions using emotion recognition AI (e.g., Microsoft Azure Cognitive Services' Emotion API). Based on the recognized emotions, the device sends prompt sentences to a generative AI model to generate an appropriate UI design.

[1716] A means to collect customer feedback, analyze data, and identify areas for UI improvement

[1717] The server collects customer feedback and analyzes it using data analysis tools. Based on the analysis results, it identifies areas for improvement in the UI and reflects them in the next UI generation.

[1718] Specific examples

[1719] For example, if a user smiles while browsing a product, the device will send a prompt to the generative AI model saying, "Generate a bright and cheerful UI," which will generate a UI with a colorful and positive design.

[1720] Prompt Sentence Examples

[1721] "If the user smiles, generate a bright and cheerful UI."

[1722] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1723] Step 1:

[1724] The device captures the user's facial expressions and voice. Specifically, it uses the smartphone's camera and microphone to collect video and audio data of the user's face. The input is the user's real-time facial expressions and voice, and the output is the captured video and audio data.

[1725] Step 2:

[1726] The device sends the captured video and audio data to an emotion recognition AI. Specifically, it uses the Emotion API from Microsoft Azure Cognitive Services to recognize the user's emotions. The input is the captured video and audio data, and the output is the recognized emotion data.

[1727] Step 3:

[1728] The server generates a prompt sentence based on the recognized emotion data. Specifically, if the emotion data indicates "happiness" or "surprise," the server generates the prompt sentence "Please generate a bright and fun UI." The input is the recognized emotion data, and the output is the generated prompt sentence.

[1729] Step 4:

[1730] The server sends the generated prompt sentence to a generative AI model. Specifically, it uses OpenAI's GPT-4 to generate a UI design based on the prompt sentence. The input is the generated prompt sentence, and the output is the generated UI design.

[1731] Step 5:

[1732] The server sends the generated UI design to the device. Specifically, the generated UI design is applied to the smartphone app and displayed to the user. The input is the generated UI design, and the output is the UI displayed to the user.

[1733] Step 6:

[1734] Users provide feedback on the displayed UI. Specifically, a feedback collection tool (e.g., Google Forms API) is used to collect user opinions and ratings. The input is the user feedback, and the output is the collected feedback data.

[1735] Step 7:

[1736] The server analyzes the collected feedback data using a data analysis tool. Specifically, it uses Python's Pandas to analyze the feedback data and identify areas for UI improvement. The input is the collected feedback data, and the output is the identified areas for UI improvement.

[1737] Step 8:

[1738] The server reflects the identified UI improvements in the next UI generation. Specifically, it sends a new prompt to the generative AI model to generate an improved UI design. The input is the identified UI improvements, and the output is the improved UI design.

[1739] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1740] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1741] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1742] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1743] [Third embodiment]

[1744] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1745] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1746] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1747] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1748] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1749] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1750] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1751] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1752] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1753] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1754] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1755] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1756] "Example 1"

[1757] In one embodiment of the present invention, the system automatically generates meeting minutes and extracts requirements from the contents of interviews. Specifically, the system converts the interview contents into text using speech recognition technology and extracts requirements from the text using natural language processing technology. For example, the system converts the audio data during the interview into text in real time, and extracts keywords and phrases from the text to create requirements.

[1758] "Example 2"

[1759] The system also automatically creates RFPs and specifications. Specifically, it generates RFPs and specifications using templates based on the extracted requirements. For example, it automatically fills in each item in the template based on the requirements to generate the RFP and specifications.

[1760] "Example 3"

[1761] Furthermore, the system can automatically create a UI and incorporate customer feedback. Specifically, the system automatically generates a UI based on requirements and then updates the UI based on customer feedback. For example, the system can generate an initial UI design based on requirements and then refine the UI based on customer feedback.

[1762] "Example 4"

[1763] The system also enables automatic coding and test creation. Specifically, it uses AI technology to generate code and create test cases. For example, AI generates code based on requirements and simultaneously creates test cases. This allows developers to significantly reduce the effort required for coding and test creation.

[1764] The processing flow of each embodiment will be described below.

[1765] "Example 1"

[1766] Step 1: The system converts the audio data being heard into text in real time using voice recognition technology.

[1767] Step 2: The system then uses natural language processing techniques to extract keywords and phrases from the text, which automatically extracts requirements.

[1768] Step 3: Finally, minutes are generated based on the extracted requirements. This automatically generates minutes and extracts requirements from the interview content.

[1769] "Example 2"

[1770] Step 1: The system selects an RFP or specification template based on the extracted requirements. Step 2: Next, each item in the selected template is automatically filled in based on the requirements. Step 3: Finally, the filled-in template is output as an RFP or specification. This allows the RFP or specification to be automatically created.

[1771] "Example 3"

[1772] Step 1: The system automatically generates a UI based on the requirements, using AI technology.

[1773] Step 2: Next, the system receives feedback from the customer.

[1774] Step 3: Finally, update the UI based on the feedback received, which automates the creation of the UI and incorporates customer feedback.

[1775] "Example 4"

[1776] Step 1: The system generates code based on the requirements. This is done using AI technology.

[1777] Step 2: At the same time, the system generates test cases, also using AI technology.

[1778] Step 3: Finally, the generated code and test cases are output, which allows for automated coding and test creation.

[1779] Example 1

[1780] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1781] In traditional meetings and hearings, minutes and requirements extraction were often done manually, which was time-consuming and labor-intensive. Manual minutes creation could lead to information leaks and misunderstandings, hindering efficient project management. Furthermore, the lack of automation in requirements extraction and minutes creation made it difficult to organize information in the early stages of a project.

[1782] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1783] In this invention, the server includes means for acquiring voice data, means for transmitting the acquired voice data to the server, means for converting the voice data into text using voice recognition technology, means for analyzing the text data and extracting requirements using natural language processing technology, means for automatically generating minutes based on the extracted requirements, and means for providing the generated minutes to a user. This makes it possible to automatically convert the contents of meetings and hearings into text, extract requirements, and generate minutes. This eliminates the need for manual minutes creation, prevents information leaks and misunderstandings, and enables efficient project management.

[1784] "Audio data" refers to data in which audio information from meetings, hearings, etc. is recorded in digital format.

[1785] A "server" is a computer system that processes and stores data over a network.

[1786] "Voice recognition technology" is a technology that analyzes voice data and converts it into text data.

[1787] "Text data" is data expressed as character information.

[1788] "Natural language processing technology" is a technology that analyzes text data, understands its meaning, and extracts information.

[1789] "Requirements" are the conditions and specifications required for a project or system.

[1790] "Minutes" are documents that record the contents of meetings or hearings.

[1791] "Automatic generation" refers to the system automatically creating data or documents without manual intervention.

[1792] "User" refers to a person or organization that uses the system.

[1793] "RFP" stands for Request for Proposal, a document that requests proposals for projects or services.

[1794] A "specification" is a document that lists the detailed specifications of a system or project.

[1795] "UI" stands for user interface, and refers to the screen and operation methods that allow users to interact with the system.

[1796] "Customer feedback" refers to opinions and requests provided by customers.

[1797] "Auto-coding" is a technique for automatically generating program code.

[1798] "Test creation" refers to creating test cases to check the operation of a system or program.

[1799] "Artificial intelligence technology" is a technology that allows machines to imitate human intelligence and learn and reason.

[1800] The present invention is a system that automatically converts the contents of a meeting or hearing into text, extracts requirements, and generates minutes. A specific embodiment of this system will be described below.

[1801] Hardware and Software Configuration

[1802] 1. Acquiring audio data

[1803] Users acquire audio data from meetings and hearings using devices such as laptops and smartphones. The devices are equipped with microphones, and audio data is recorded by launching a recording application.

[1804] 2. Sending audio data

[1805] The device sends the acquired audio data to the server. Specifically, the device application uploads the audio data to the server as an HTTP request. At this time, the audio data is sent in an appropriate format, such as WAV or MP3.

[1806] 3. Converting audio data into text

[1807] The server processes the received voice data in real time. Specifically, it converts the voice data into text data using voice recognition technology. For example, it uses the Google Cloud Speech-to-Text API. The server sends the voice data to the API and stores the returned text data.

[1808] 4. Text Data Analysis

[1809] The server analyzes the text data using natural language processing (NLP) techniques. Specifically, it uses the Python libraries spaCy and NLTK to extract keywords and important phrases from the text. The server passes the text data to the analysis module and obtains the analysis results.

[1810] 5. Requirements Extraction

[1811] The server extracts requirements from the analysis results. Specifically, it lists important requirements from the conversation based on the extracted keywords and phrases. For example, requirements such as "project start date," "budget," and "person in charge" are extracted.

[1812] 6. Generate minutes

[1813] The server automatically generates minutes based on the extracted requirements. Specifically, it organizes the main points of the conversation and decisions made, and creates text-based minutes. The server uses templates to format the minutes.

[1814] 7. Providing Results

[1815] The server provides the generated minutes to the user. Specifically, it generates the minutes in PDF or Word format and sends a download link to the user's device. The user can click the link to download the minutes and check their contents.

[1816] Specific examples

[1817] For example, consider a scenario where a user is hearing requirements for a new project. The user uses their laptop to record the conversation, and the audio data is sent to a server. The server converts the audio into text using the Google Cloud Speech-to-Text API, and then analyzes the text using spaCy. As a result of the analysis, requirements such as "project start date," "budget," and "person in charge" are extracted. Finally, minutes containing these requirements are automatically generated and provided to the user.

[1818] Prompt Sentence Examples

[1819] Examples of prompts to be input to a generative AI model include:

[1820] "Please transcribe the following audio and extract the key requirements. The audio contains requirements for a new project."

[1821] Using this prompt, the generative AI model can properly process the voice data and extract the necessary information.

[1822] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1823] Step 1: Acquire audio data

[1824] Users use devices such as laptops and smartphones to capture audio data from meetings and hearings. Specifically, they start up the device's recording application and press the record button to begin capturing audio data. The input is the audio from the meeting or hearing, and the output is digital audio data.

[1825] Step 2: Sending audio data

[1826] The device sends the acquired audio data to the server. Specifically, the device application uploads the audio data to the server as an HTTP request. The audio data is sent in an appropriate format, such as WAV or MP3. The input is digital audio data, and the output is the audio data sent to the server.

[1827] Step 3: Converting audio data to text

[1828] The server processes the received voice data in real time. Specifically, it uses voice recognition technology to convert the voice data into text data. For example, it uses the Google Cloud Speech-to-Text API. The server sends the voice data to the API and saves the returned text data. The input is the voice data sent to the server, and the output is text data.

[1829] Step 4: Analyzing the text data

[1830] The server analyzes the text data using natural language processing (NLP) techniques. Specifically, it uses the Python libraries spaCy and NLTK to extract keywords and important phrases from the text. The server passes the text data to the analysis module and obtains the analysis results. The input is the text data, and the output is the keywords and important phrases that are the analysis results.

[1831] Step 5: Eliciting requirements

[1832] The server extracts requirements from the analysis results. Specifically, it lists important requirements from the conversation based on the extracted keywords and phrases. For example, requirements such as "project start date," "budget," and "person in charge" are extracted. The input is the keywords and important phrases from the analysis results, and the output is the extracted requirements.

[1833] Step 6: Generate the transcript

[1834] The server automatically generates minutes based on the extracted requirements. Specifically, it organizes the main points of the conversation and decisions and creates text-based minutes. The server uses a template to format the minutes. The input is the extracted requirements, and the output is the generated minutes.

[1835] Step 7: Delivering results

[1836] The server provides the generated minutes to the user. Specifically, it generates the minutes in PDF or Word format and sends a download link to the user's device. The user can click the link to download the minutes and check their contents. The input is the generated minutes, and the output is the minutes provided to the user.

[1837] (Application example 1)

[1838] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1839] In autonomous vehicles, smooth communication between passengers and vehicle systems is required, and passenger instructions and requests must be accurately reflected. However, with current systems, it is difficult to convert passenger voice instructions into text in real time, extract requirements, and reflect them in the vehicle system. In addition, since minutes are generated and requirements are extracted manually, this is inefficient and lacks accuracy. Technology is needed to solve these problems.

[1840] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1841] In this invention, the server includes means for automatically generating minutes and extracting requirements from the contents of the interview, means for automatically creating RFPs and specifications, means for automatically creating a UI and reflecting customer feedback, means for realizing automatic coding and test creation, means for converting the contents of the interview into text using speech recognition technology and extracting requirements from the text using natural language processing technology, means for converting conversations inside the autonomous vehicle into text in real time, generating minutes, and extracting requirements, and means for reflecting the extracted requirements in the vehicle system. This makes it possible to accurately reflect passenger instructions and requests in real time.

[1842] "Hearing content" refers to audio information collected from users and customers.

[1843] "Minutes generation" is the process of recording the contents of a meeting or conversation as a document based on the contents of the hearing.

[1844] "Requirements extraction" is the process of extracting important keywords and phrases from the interview content and organizing them as system or project requirements.

[1845] "RFP" stands for Request for Proposal, a document requesting proposals for the provision of projects or services.

[1846] A "specification" is a document that describes the detailed requirements and functions of a system or project.

[1847] "UI" stands for user interface and refers to the screens and methods of operation that allow users to interact with systems and applications.

[1848] "Customer feedback" refers to information such as opinions, requests, and evaluations provided by customers.

[1849] "Autocoding" is the process of automatically generating program code.

[1850] "Test creation" is the process of creating test cases to verify the operation of a system or application.

[1851] "Speech recognition technology" is a technology that converts speech into text.

[1852] "Natural language processing technology" is a technology for analyzing text data and understanding its meaning and structure.

[1853] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[1854] "Real-time" means processing or reacting immediately, without delay.

[1855] A "vehicle system" is a system for controlling the operation and functions of an autonomous vehicle.

[1856] The following system configuration is used as an embodiment of this invention. The server includes means for automatically generating minutes and extracting requirements from the contents of interviews, means for automatically creating RFPs and specifications, means for automatically creating UIs and reflecting customer feedback, means for realizing automatic coding and test creation, means for converting the contents of interviews into text using speech recognition technology and extracting requirements from the text using natural language processing technology, means for converting conversations inside an autonomous vehicle into text in real time, generating minutes, and extracting requirements, and means for reflecting the extracted requirements in the vehicle system.

[1857] Program processing explanation

[1858] The server captures voice data using the smartphone's microphone and converts it to text using the Google Speech Recognition API. The converted text is then analyzed using natural language processing models in the Transformers library to extract important keywords and phrases, ensuring passengers' instructions and requests are accurately reflected in real time.

[1859] Specifically, the system converts the interview content into text using speech recognition technology and then extracts requirements from the text using natural language processing technology. For example, if a passenger says, "Turn right at the next intersection," the system extracts the requirements "next intersection" and "turn right" and reflects them in the autonomous vehicle's system.

[1860] Hardware and software used

[1861] Hardware: Smartphone microphone

[1862] Software: Python, Google Speech Recognition API, Transformers library

[1863] Specific examples

[1864] For example, if a passenger says, "Turn right at the next intersection," the system extracts the requirements "next intersection" and "turn right" and reflects them in the autonomous vehicle's system.

[1865] Prompt Sentence Examples

[1866] Passenger: "Turn right at the next intersection."

[1867] System: Extracted the requirements "next intersection" and "turn right."

[1868] In this way, passenger instructions can be recognized in real time and reflected in the operation of the autonomous vehicle.

[1869] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1870] Step 1:

[1871] The server acquires voice data using the smartphone's microphone. The user's voice is input, and voice data is collected through the microphone. This voice data is then sent to the server in real time.

[1872] Step 2:

[1873] The server converts the acquired voice data into text using the Google Speech Recognition API. The input is voice data and the output is text data. Using voice recognition technology, the voice data is analyzed and converted into the corresponding text.

[1874] Step 3:

[1875] The server analyzes the converted text data using a natural language processing model from the Transformers library. The input is text data, and the output is extracted keywords and phrases. Natural language processing techniques are used to extract important keywords and phrases from the text data.

[1876] Step 4:

[1877] The server organizes the extracted keywords and phrases as requirements and reflects them in the autonomous vehicle system. The input is the extracted keywords and phrases, and the output is the requirements reflected in the vehicle system. The requirements are organized and data processing is performed to apply them to the vehicle system.

[1878] Step 5:

[1879] The server confirms that the requirements have been accurately reflected and provides feedback to the user. The input is the requirements reflected in the vehicle system, and the output is a feedback message to the user. A process is performed to generate the feedback message and notify the user.

[1880] Example 2

[1881] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1882] In the conventional system, tasks such as generating minutes from interviews, extracting requirements, and creating RFPs and specifications based on the requirements were all done manually, which was time-consuming and labor-intensive. There was also a high possibility of human error occurring in the extraction of requirements and document creation. Furthermore, there was a lack of efficient means for creating UIs, incorporating customer feedback, and automated coding and test creation. To solve these issues, a more efficient and accurate system is needed.

[1883] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1884] In this invention, the server includes a means for automatically generating minutes and extracting requirements from the interview content, a means for receiving requirements documents and extracting requirements using a generative AI model, and a means for automatically creating RFPs and specifications using templates based on the extracted requirements. This automates the process from requirements extraction to creating RFPs and specifications, significantly reducing time and labor. It also reduces human error and enables more accurate document creation.

[1885] "Hearing contents" refers to information including the requirements and wishes of the user regarding the project or system provided.

[1886] "Generating minutes" is the process of recording and documenting the contents of a meeting based on the hearing content.

[1887] "Requirements extraction" is the process of extracting necessary information from interview content and requirements documents and organizing it as structured data.

[1888] A "generative AI model" is a model that uses artificial intelligence technology to generate text and extract requirements.

[1889] A "prompt sentence" is an input sentence that gives specific instructions to a generative AI model.

[1890] A "template" is a predefined document template used when creating RFPs and specifications.

[1891] "RFP" stands for Request for Proposal, a document used to request proposals for projects or services.

[1892] A "specification" is a document that lists the detailed requirements and specifications for a system or project.

[1893] "UI" stands for user interface and refers to the screen and operation methods that users use to interact with a system.

[1894] "Customer feedback" refers to opinions and requests provided by customers, and is information that can be used to improve and adjust the system.

[1895] "Autocoding" is the process of automatically generating program code.

[1896] "Test creation" is the process of creating test cases to verify the operation of a system or program.

[1897] This invention relates to a system that automatically generates minutes and extracts requirements from interview content, receives requirements documents, extracts requirements using a generative AI model, and automatically creates RFPs and specifications using templates based on the extracted requirements.

[1898] The server receives the requirements document provided by the user and extracts requirements using a generative AI model. Specifically, the server performs the following steps:

[1899] First, the user uploads a requirements document through the system interface. The requirements document can be provided in the form of a text file, PDF, Word, etc. The server receives the uploaded requirements document and analyzes its contents.

[1900] Next, the server generates a prompt based on the requirements document. A prompt is an input sentence that provides specific instructions to the generative AI model. For example, the following prompt might be generated:

[1901] Generate an RFP based on the requirements below.

[1902] Project Objective: Development of a new customer management system

[1903] Goal: Efficiently manage and analyze customer data

[1904] Functional requirements: Ability to register, search, update, and delete customer information

[1905] Non-functional requirements: high security, scalability

[1906] The server inputs this prompt into a generative AI model to extract requirements. The generative AI model, such as OpenAI's GPT-4, extracts the necessary information from the requirements document based on the prompt and saves it as structured data.

[1907] The server then selects a pre-prepared RFP or specification template and automatically fills in each item in the template with the extracted requirements. For example, the extracted "project objectives" and "goals" are filled in the "Project Overview" section of the template.

[1908] Finally, the server generates the final RFP and specifications based on the requirements embedded in the template. The generated documents are output in PDF or Word format and made available for download by users. Users can download and review the generated RFP and specifications from the system interface.

[1909] This system automates the process from requirements extraction to the creation of RFPs and specifications, significantly reducing time and effort, and also reducing human error, resulting in more accurate document creation.

[1910] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1911] Step 1:

[1912] A user uploads a requirements document.

[1913] The user logs in to the system's web interface and clicks the "Upload requirements document" button. A file selection dialog appears, and the user selects and uploads the requirements document. The input is the requirements document, and the output is the requirements document saved on the server.

[1914] Step 2:

[1915] The server receives the requirements document and parses the contents.

[1916] The server receives the uploaded requirements document and analyzes its contents. Specifically, it reads the text of the requirements document and identifies each section. The input is the uploaded requirements document, and the output is the parsed text data of the requirements document.

[1917] Step 3:

[1918] The server generates a prompt.

[1919] The server generates a prompt sentence based on the content of the parsed requirements document. For example, if the requirements document states "Project Objective: Develop a new customer management system," the prompt sentence will contain "Project Objective: Develop a new customer management system." The input is the text data of the parsed requirements document, and the output is the generated prompt sentence.

[1920] Step 4:

[1921] The server uses a generative AI model to extract requirements.

[1922] The server inputs the generated prompt sentence into a generative AI model to extract requirements. Examples of generative AI models that can be used include OpenAI's GPT-4. The generative AI model extracts necessary information from the requirements document based on the prompt sentence and saves it as structured data. The input is the prompt sentence, and the output is structured data of the extracted requirements.

[1923] Step 5:

[1924] The server selects a template and fills it with the extracted requirements.

[1925] The server selects a pre-prepared RFP or specification template. Then, it automatically embeds the extracted requirements into each section of the template. For example, the extracted "Project Objectives" and "Goals" are embedded in the "Project Overview" section. The input is structured data of the extracted requirements, and the output is a template with the embedded requirements.

[1926] Step 6:

[1927] The server generates the final RFP and specifications and provides them to the user.

[1928] The server generates the final RFP or specification based on the requirements embedded in the template. The generated document is output in PDF or Word format and made available for download by the user. The user can download and review the generated RFP or specification from the system interface. The input is the template with the embedded requirements, and the output is the generated RFP or specification.

[1929] (Application example 2)

[1930] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1931] In the conventional system, processes such as generating minutes from interviews, extracting requirements, creating RFPs and specifications, automatically creating UI, incorporating customer feedback, and automatically coding and creating tests were all done manually, which was time-consuming and labor-intensive. Additionally, creating specifications for machinery used in factories was also done manually, which was inefficient and prone to errors. To solve these issues, a system was needed that could automate these processes and perform them efficiently and accurately.

[1932] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1933] In this invention, the server includes a means for automatically generating minutes and extracting requirements from interview content, a means for automatically creating RFPs and specifications, a means for automatically creating UIs and incorporating customer feedback, a means for realizing automatic coding and test creation, and a means for automatically generating specifications for machines used in factories. This automates these processes, making them efficient and accurate.

[1934] "Hearing content" refers to oral information collected through meetings, interviews, etc.

[1935] "Generating minutes" means creating a document that records the contents of a meeting based on the contents of the hearing.

[1936] "Requirements elicitation" is the process of identifying and extracting necessary requirements from interviews and other sources of information.

[1937] "RFP" stands for Request for Proposal, a document that solicits proposals for a specific project or service.

[1938] A "specification" is a document that details the functions, performance, design, etc. of a product or system.

[1939] "UI" stands for user interface, the means by which a user interacts with a system or application.

[1940] "Customer feedback" refers to information such as opinions, requests, and evaluations provided by customers.

[1941] "Auto-coding" means automatically generating program code.

[1942] "Test creation" means creating test cases to verify the operation of software or systems.

[1943] "Machinery used in a factory" refers to equipment and facilities used for production and processing in the manufacturing industry.

[1944] "Automatic specification generation" means automatically creating specifications based on specific requirements and conditions.

[1945] A system for carrying out this invention is configured as follows: The server includes means for automatically generating minutes and extracting requirements from interview content, means for automatically creating RFPs and specifications, means for automatically creating UIs and incorporating customer feedback, means for realizing automatic coding and test creation, and means for automatically generating specifications for machines used in factories.

[1946] The server converts the interview content into text using speech recognition technology and extracts requirements from the text using natural language processing technology. This makes it possible to automatically generate meeting minutes and extract requirements from the interview content. Furthermore, it uses a generative AI model to generate prompt sentences based on the requirements and automatically create RFPs and specifications.

[1947] Specifically, the server performs the following process. First, the user inputs the interview content by voice, and the server converts it into text using speech recognition technology. Next, it uses natural language processing technology to extract requirements from the text. The extracted requirements are input into a generative AI model, which generates prompt text. Based on this prompt text, an RFP and specifications are automatically created.

[1948] The server also includes a means for automatically generating specifications for machines used in the factory. When a user inputs requirements for machines in the factory, the server analyzes them and automatically generates specifications using an appropriate template. For example, when requirements for the installation of a new manufacturing line are input, the server generates a prompt sentence such as the following:

[1949] Example prompt sentence:

[1950] Generate a specification based on the following requirements:

[1951] 1. Installation of a new production line

[1952] 2. Machine layout and connection

[1953] 3. Implementation of safety measures

[1954] 4. Regular maintenance schedule

[1955] Template:

[1956] 1. Overview

[1957] 2. Detailed requirements

[1958] 3. Implementation Plan

[1959] 4. Maintenance Plan

[1960] Generated specification:

[1961] Based on this prompt, the server uses a generative AI model to automatically generate specifications, enabling efficient and accurate creation of specifications for machines used in factories.

[1962] The hardware used includes factory robots (e.g., ABB's IRB series) and servers. The software used includes OpenAI GPT-3 models and Python programs. The combination of these technologies allows the invention to achieve its objectives.

[1963] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1964] Step 1:

[1965] The user inputs what he or she has heard by voice.

[1966] Input: The user inputs what they heard by voice.

[1967] How it works: A user uses a microphone to dictate a meeting or interview.

[1968] Output: The audio data is sent to the server.

[1969] Step 2:

[1970] The server uses voice recognition technology to convert the voice data into text.

[1971] Input: Audio data

[1972] How it works: The server uses speech recognition software (e.g., Google Speech-to-Text API) to convert the audio data into text.

[1973] Output: Text data

[1974] Step 3:

[1975] The server uses natural language processing technology to extract requirements from the text data.

[1976] Input: Text data

[1977] How it works: The server uses a natural language processing library (...

Claims

1. A means for acquiring audio data; A means for converting voice data into text using voice recognition technology; A means for analyzing text data using natural language processing technology and extracting requirements; A means for automatically generating minutes based on the extracted requirements; A means for providing the generated minutes to a user; A system including:

2. a means for obtaining a user provided requirements document; A means for extracting requirements contained in the requirements document using a generative AI model; A means for embedding the extracted requirements into each item of a pre-prepared document template; A means for generating final documentation based on the template with embedded requirements; A system including:

3. A means for obtaining requirements for a UI provided by a user; A means for generating an initial UI design using a UI design tool based on the obtained requirements; means for analyzing user feedback on the generated initial UI design and applying necessary changes; A means for generating program code that satisfies requirements using a generative AI model; A means for generating test cases based on the generated program code using a test automation tool to verify the operation of the generated program code; A system including:

Images